The Attention Mechanism of LLMs

DataChad taps into the attention mechanism offered by LLMs, such as GPT-3, using the OpenAI API. This attention mechanism enables the model to weigh the importance of different words or tokens in a text sequence, capturing contextual relationships and semantic nuances. By leveraging LLMs, DataChad benefits from their ability to generate rich and accurate representations of textual data.


Factors to Consider as You Build a LangChain & Large Language Model-based app (k. arg, chunks, etc.)

The DataChad project is built upon the fusion of two critical natural language processing (NLP) technologies, leveraging the attention mechanism of Language Model-based Models (LLMs) like GPT-4 through the OpenAI API and employing vector similarity for efficient embedding comparison when querying the vector database. This combination allows for robust analysis and retrieval of information from textual data. Let's delve into the details, focusing on the querying parameters of the vector database within DataChad.


Streamlit

DataChad is implemented as a Streamlit app, a way to build demo apps in Python quickly. It takes the pain away from caring about how to implement a UI and how to host the app properly and lets you focus on the backend work.


Vector Similarity for Embedding Comparison

When querying the vector database, DataChad employs vector similarity to compare document embeddings. This technique measures the geometric similarity between embeddings, allowing for the efficient retrieval of similar documents. Vector similarity provides a simple yet effective method for identifying related content in large-scale datasets.


chunk_size

chunk_size in LangChain-based apps determines the size at which the text is divided into smaller chunks before being embedded. This parameter ensures the efficient processing of large documents and controls the granularity of the resulting embeddings. The DataChad default is 1000.


Parameters for Querying the Vector Database and the LLM

DataChad's querying process involves several important parameters that influence the retrieval and analysis of document embeddings. What are those parameters?


fetch_k

fetch_k in LangChain-based apps specifies the number of documents to pull from the vector database. This parameter determines the scope of the search and influences the relevance of the retrieved documents. The DataChad default is 20.


max_tokens

The max_tokens parameter limits the documents returned from the vector store based on tokens before building the context to query the LLM. This parameter ensures that DataChad does not run into the LLM's prompt limit (4069 for gpt-3.5-turbo). The DataChad default is 3375.


temperature

LLM temperature controls the randomness of the LLM output. A temperature of 0 means the response is deterministic: it always returns the same completion (making it significantly less prone to hallucination). A temperature of greater than zero results in increasing variation in the completion. The DataChad default is 0.7.By carefully tuning these parameters, DataChad optimizes the trade-off between computational efficiency and the quality of results obtained from both the vector database and LLM-based querying. By ticking the Advanced Options checkbox in the app, experienced users can further modify these parameters.


Issue 1: Running into errors related to the prompt length

Solution: Decrease one or many of k, chunk_size, and max_tokens.


How to Solve Most Common Issues When Building With LangChain

The previous section discussed the importance of selecting appropriate parameters for querying the vector database and the language model within the DataChad project. However, despite the default values having been carefully chosen and tested, it is not uncommon to encounter challenges or the desire for further improvement in the overall query experience. In this section, we will address some common issues you may face as you build your app and provide suggested solutions that can help overcome these challenges.


Issue 3: The answers are not relevant enough

Solution: Increase chunk_size, or if this leads to running into issue 1, then increase k and fetch_k while decreasing chunk_size


k

The k in LangChain-based apps represents the most similar embeddings selected to build the context for the LLM prompt in the langchain. This parameter affects the contextual understanding and response generation of the LLM when querying the OpenAI API. The DataChad default is 10.


Practical Guide: Building an All-In-One Chat with Anything App

The code is split in three parts. First, we build out the Streamlit app defined in app.py. The second part, utils.py, contains all processing functionality and API calls. Final part is constants.py, where all project-specific paths, names, and descriptions are defined.


app.py

1import streamlit as st 2from streamlit_chat import message 3 4from constants import ( 5 ACTIVELOOP_HELP, 6 APP_NAME, 7 AUTHENTICATION_HELP, 8 CHUNK_SIZE, 9 DEFAULT_DATA_SOURCE, 10 ENABLE_ADVANCED_OPTIONS, 11 FETCH_K, 12 MAX_TOKENS, 13 OPENAI_HELP, 14 PAGE_ICON, 15 REPO_URL, 16 TEMPERATURE, 17 USAGE_HELP, 18 K, 19) 20from utils import ( 21 advanced_options_form, 22 authenticate, 23 delete_uploaded_file, 24 generate_response, 25 logger, 26 save_uploaded_file, 27 update_chain, 28) 29 30# Page options and header 31st.set_option('client.showErrorDetails', True) 32st.set_page_config( 33 page_title=APP_NAME, page_icon=PAGE_ICON, initial_sidebar_state='expanded' 34) 35st.markdown( 36 f'
{APP_NAME} {PAGE_ICON}
 I know all about your data!
', 37 unsafe_allow_html=True, 38) 39 40# Initialise session state variables 41# Chat and Data Source 42if 'past' not in st.session_state: 43 st.session_state['past'] = [] 44if 'usage' not in st.session_state: 45 st.session_state['usage'] = {} 46if 'chat_history' not in st.session_state: 47 st.session_state['chat_history'] = [] 48if 'generated' not in st.session_state: 49 st.session_state['generated'] = [] 50if 'data_source' not in st.session_state: 51 st.session_state['data_source'] = DEFAULT_DATA_SOURCE 52if 'uploaded_file' not in st.session_state: 53 st.session_state['uploaded_file'] = None 54# Authentication and Credentials 55if 'auth_ok' not in st.session_state: 56 st.session_state['auth_ok'] = False 57if 'openai_api_key' not in st.session_state: 58 st.session_state['openai_api_key'] = None 59if 'activeloop_token' not in st.session_state: 60 st.session_state['activeloop_token'] = None 61if 'activeloop_org_name' not in st.session_state: 62 st.session_state['activeloop_org_name'] = None 63# Advanced Options 64if 'k' not in st.session_state: 65 st.session_state['k'] = K 66if 'fetch_k' not in st.session_state: 67 st.session_state['fetch_k'] = FETCH_K 68if 'chunk_size' not in st.session_state: 69 st.session_state['chunk_size'] = CHUNK_SIZE 70if 'temperature' not in st.session_state: 71 st.session_state['temperature'] = TEMPERATURE 72if 'max_tokens' not in st.session_state: 73 st.session_state['max_tokens'] = MAX_TOKENS 74 75# Sidebar with Authentication 76# Only start App if authentication is OK 77with st.sidebar: 78 st.title('Authentication', help=AUTHENTICATION_HELP) 79 with st.form('authentication'): 80 openai_api_key = st.text_input( 81 'OpenAI API Key', 82 type='password', 83 help=OPENAI_HELP, 84 placeholder='This field is mandatory', 85 ) 86 activeloop_token = st.text_input( 87 'ActiveLoop Token', 88 type='password', 89 help=ACTIVELOOP_HELP, 90 placeholder='Optional, using ours if empty', 91 ) 92 activeloop_org_name = st.text_input( 93 'ActiveLoop Organisation Name', 94 type='password', 95 help=ACTIVELOOP_HELP, 96 placeholder='Optional, using ours if empty', 97 ) 98 submitted = st.form_submit_button('Submit') 99 if submitted: 100 authenticate(openai_api_key, activeloop_token, activeloop_org_name) 101 102 st.info(f'Learn how it works [here]({REPO_URL})') 103 if not st.session_state['auth_ok']: 104 st.stop() 105 106 # Clear button to reset all chat communication 107 clear_button = st.button('Clear Conversation', key='clear') 108 109 # Advanced Options 110 if ENABLE_ADVANCED_OPTIONS: 111 advanced_options_form() 112 113# the chain can only be initialized after authentication is OK 114if 'chain' not in st.session_state: 115 update_chain() 116 117if clear_button: 118 # resets all chat history related caches 119 st.session_state['past'] = [] 120 st.session_state['generated'] = [] 121 st.session_state['chat_history'] = [] 122 123# file upload and data source inputs 124uploaded_file = st.file_uploader('Upload a file') 125data_source = st.text_input( 126 'Enter any data source', 127 placeholder='Any path or url pointing to a file or directory of files', 128) 129 130# generate new chain for new data source / uploaded file 131# make sure to do this only once per input / on change 132if data_source and data_source != st.session_state['data_source']: 133 logger.info(f'Data source provided: '{data_source}'') 134 st.session_state['data_source'] = data_source 135 update_chain() 136 137if uploaded_file and uploaded_file != st.session_state['uploaded_file']: 138 logger.info(f'Uploaded file: '{uploaded_file.name}'') 139 st.session_state['uploaded_file'] = uploaded_file 140 data_source = save_uploaded_file(uploaded_file) 141 st.session_state['data_source'] = data_source 142 update_chain() 143 delete_uploaded_file(uploaded_file) 144 145# container for chat history 146response_container = st.container() 147# container for text box 148container = st.container() 149 150# As streamlit reruns the whole script on each change 151# it is necessary to repopulate the chat containers 152with container: 153 with st.form(key='prompt_input', clear_on_submit=True): 154 user_input = st.text_area('You:', key='input', height=100) 155 submit_button = st.form_submit_button(label='Send') 156 157 if submit_button and user_input: 158 output = generate_response(user_input) 159 st.session_state['past'].append(user_input) 160 st.session_state['generated'].append(output) 161 162if st.session_state['generated']: 163 with response_container: 164 for i in range(len(st.session_state['generated'])): 165 message(st.session_state['past'][i], is_user=True, key=str(i) + '_user') 166 message(st.session_state['generated'][i], key=str(i)) 167 168# Usage sidebar with total used tokens and costs 169# We put this at the end to be able to show usage starting with the first response 170with st.sidebar: 171 if st.session_state['usage']: 172 st.divider() 173 st.title('Usage', help=USAGE_HELP) 174 col1, col2 = st.columns(2) 175 col1.metric('Total Tokens', st.session_state['usage']['total_tokens']) 176 col2.metric('Total Costs in $', st.session_state['usage']['total_cost']) 177 178Copy


utils.py

Consider contributing: {REPO_URL}!' 194 error_msg = f'Failed to load {st.session_state['data_source']} with Error:\n{e}' 195 st.error(error_msg, icon=PAGE_ICON) 196 logger.info(error_msg) 197 st.stop() 198 199def load_git(data_source: str, chunk_size: int = CHUNK_SIZE) -> List[Document]: 200 # We need to try both common main branches 201 # Thank you github for the 'master' to 'main' switch 202 repo_name = data_source.split('/')[-1].split('.')[0] 203 repo_path = str(DATA_PATH / repo_name) 204 text_splitter = RecursiveCharacterTextSplitter( 205 chunk_size=chunk_size, chunk_overlap=0 206 ) 207 branches = ['main', 'master'] 208 for branch in branches: 209 if os.path.exists(repo_path): 210 data_source = None 211 try: 212 docs = GitLoader(repo_path, data_source, branch).load_and_split( 213 text_splitter 214 ) 215 break 216 except Exception as e: 217 logger.info(f'Error loading git: {e}') 218 if os.path.exists(repo_path): 219 # cleanup repo afterwards 220 shutil.rmtree(repo_path) 221 try: 222 return docs 223 except Exception as e: 224 handle_load_error() 225 226def load_any_data_source( 227 data_source: str, chunk_size: int = CHUNK_SIZE 228) -> List[Document]: 229 # Ugly thing that decides how to load data 230 # It aint much, but it's honest work 231 is_text = data_source.endswith('.txt') 232 is_web = data_source.startswith('http') 233 is_pdf = data_source.endswith('.pdf') 234 is_csv = data_source.endswith('csv') 235 is_html = data_source.endswith('.html') 236 is_git = data_source.endswith('.git') 237 is_notebook = data_source.endswith('.ipynb') 238 is_doc = data_source.endswith('.doc') 239 is_py = data_source.endswith('.py') 240 is_dir = os.path.isdir(data_source) 241 is_file = os.path.isfile(data_source) 242 243 loader = None 244 if is_dir: 245 loader = DirectoryLoader(data_source, recursive=True, silent_errors=True) 246 elif is_git: 247 return load_git(data_source, chunk_size) 248 elif is_web: 249 if is_pdf: 250 loader = OnlinePDFLoader(data_source) 251 else: 252 loader = WebBaseLoader(data_source) 253 elif is_file: 254 if is_text: 255 loader = TextLoader(data_source) 256 elif is_notebook: 257 loader = NotebookLoader(data_source) 258 elif is_pdf: 259 loader = UnstructuredPDFLoader(data_source) 260 elif is_html: 261 loader = UnstructuredHTMLLoader(data_source) 262 elif is_doc: 263 loader = UnstructuredWordDocumentLoader(data_source) 264 elif is_csv: 265 loader = CSVLoader(data_source, encoding='utf-8') 266 elif is_py: 267 loader = PythonLoader(data_source) 268 else: 269 loader = UnstructuredFileLoader(data_source) 270 try: 271 # Chunk size is a major trade-off parameter to control result accuracy over computaion 272 text_splitter = RecursiveCharacterTextSplitter( 273 chunk_size=chunk_size, chunk_overlap=0 274 ) 275 docs = loader.load_and_split(text_splitter) 276 logger.info(f'Loaded: {len(docs)} document chucks') 277 return docs 278 except Exception as e: 279 handle_load_error(e if loader else None) 280 281def clean_data_source_string(data_source_string: str) -> str: 282 # replace all non-word characters with dashes 283 # to get a string that can be used to create a new dataset 284 dashed_string = re.sub(r'\W+', '-', data_source_string) 285 cleaned_string = re.sub(r'--+', '- ', dashed_string).strip('-') 286 return cleaned_string 287 288def setup_vector_store(data_source: str, chunk_size: int = CHUNK_SIZE) -> VectorStore: 289 # either load existing vector store or upload a new one to the hub 290 embeddings = OpenAIEmbeddings( 291 disallowed_special=(), openai_api_key=st.session_state['openai_api_key'] 292 ) 293 data_source_name = clean_data_source_string(data_source) 294 dataset_path = f'hub://{st.session_state['activeloop_org_name']}/{data_source_name}-{chunk_size}' 295 if deeplake.exists(dataset_path, token=st.session_state['activeloop_token']): 296 with st.spinner('Loading vector store...'): 297 logger.info(f'Dataset '{dataset_path}' exists -> loading') 298 vector_store = DeepLake( 299 dataset_path=dataset_path, 300 read_only=True, 301 embedding_function=embeddings, 302 token=st.session_state['activeloop_token'], 303 ) 304 else: 305 with st.spinner('Reading, embedding and uploading data to hub...'): 306 logger.info(f'Dataset '{dataset_path}' does not exist -> uploading') 307 docs = load_any_data_source(data_source, chunk_size) 308 vector_store = DeepLake.from_documents( 309 docs, 310 embeddings, 311 dataset_path=dataset_path, 312 token=st.session_state['activeloop_token'], 313 ) 314 return vector_store 315 316def build_chain( 317 data_source: str, 318 k: int = K, 319 fetch_k: int = FETCH_K, 320 chunk_size: int = CHUNK_SIZE, 321 temperature: float = TEMPERATURE, 322 max_tokens: int = MAX_TOKENS, 323) -> ConversationalRetrievalChain: 324 # create the langchain that will be called to generate responses 325 vector_store = setup_vector_store(data_source, chunk_size) 326 retriever = vector_store.as_retriever() 327 # Search params 'fetch_k' and 'k' define how many documents are pulled from the hub 328 # and selected after the document matching to build the context 329 # that is fed to the model together with your prompt 330 search_kwargs = { 331 'maximal_marginal_relevance': True, 332 'distance_metric': 'cos', 333 'fetch_k': fetch_k, 334 'k': k, 335 } 336 retriever.search_kwargs.update(search_kwargs) 337 model = ChatOpenAI( 338 model_name=MODEL, 339 temperature=temperature, 340 openai_api_key=st.session_state['openai_api_key'], 341 ) 342 chain = ConversationalRetrievalChain.from_llm( 343 model, 344 retriever=retriever, 345 chain_type='stuff', 346 verbose=True, 347 # we limit the maximum number of used tokens 348 # to prevent running into the models token limit of 4096 349 max_tokens_limit=max_tokens, 350 ) 351 logger.info(f'Data source '{data_source}' is ready to go!') 352 return chain 353 354def update_chain() -> None: 355 # Build chain with parameters from session state and store it back 356 # Also delete chat history to not confuse the bot with old context 357 try: 358 st.session_state['chain'] = build_chain( 359 data_source=st.session_state['data_source'], 360 k=st.session_state['k'], 361 fetch_k=st.session_state['fetch_k'], 362 chunk_size=st.session_state['chunk_size'], 363 temperature=st.session_state['temperature'], 364 max_tokens=st.session_state['max_tokens'], 365 ) 366 st.session_state['chat_history'] = [] 367 except Exception as e: 368 msg = f'Failed to build chain for data source {st.session_state['data_source']} with error: {e}' 369 logger.error(msg) 370 st.error(msg, icon=PAGE_ICON) 371 372def update_usage(cb: OpenAICallbackHandler) -> None: 373 # Accumulate API call usage via callbacks 374 logger.info(f'Usage: {cb}') 375 callback_properties = [ 376 'total_tokens', 377 'prompt_tokens', 378 'completion_tokens', 379 'total_cost', 380 ] 381 for prop in callback_properties: 382 value = getattr(cb, prop, 0) 383 st.session_state['usage'].setdefault(prop, 0) 384 st.session_state['usage'][prop] += value 385 386def generate_response(prompt: str) -> str: 387 # call the chain to generate responses and add them to the chat history 388 with st.spinner('Generating response'), get_openai_callback() as cb: 389 response = st.session_state['chain']( 390 {'question': prompt, 'chat_history': st.session_state['chat_history']} 391 ) 392 update_usage(cb) 393 logger.info(f'Response: '{response}'') 394 st.session_state['chat_history'].append((prompt, response['answer'])) 395 return response['answer'] 396 397Copy


Issue 2: The answers contain hallucinations or do not match the true data content

Solution: Decrease the temperature. Set it to 0 for the most conservative answers that are unlikely to deviate from the sources.


utils.py

1import logging 2import os 3import re 4import shutil 5import sys 6from typing import List 7 8import deeplake 9import openai 10import streamlit as st 11from dotenv import load_dotenv 12from langchain.callbacks import OpenAICallbackHandler, get_openai_callback 13from langchain.chains import ConversationalRetrievalChain 14from langchain.chat_models import ChatOpenAI 15from langchain.document_loaders import ( 16 CSVLoader, 17 DirectoryLoader, 18 GitLoader, 19 NotebookLoader, 20 OnlinePDFLoader, 21 PythonLoader, 22 TextLoader, 23 UnstructuredFileLoader, 24 UnstructuredHTMLLoader, 25 UnstructuredPDFLoader, 26 UnstructuredWordDocumentLoader, 27 WebBaseLoader, 28) 29from langchain.embeddings.openai import OpenAIEmbeddings 30from langchain.schema import Document 31from langchain.text_splitter import RecursiveCharacterTextSplitter 32from langchain.vectorstores import DeepLake, VectorStore 33from streamlit.runtime.uploaded_file_manager import UploadedFile 34 35from constants import ( 36 APP_NAME, 37 CHUNK_SIZE, 38 DATA_PATH, 39 FETCH_K, 40 MAX_TOKENS, 41 MODEL, 42 PAGE_ICON, 43 REPO_URL, 44 TEMPERATURE, 45 K, 46) 47 48# loads environment variables 49load_dotenv() 50 51logger = logging.getLogger(APP_NAME) 52 53def configure_logger(debug: int = 0) -> None: 54 # boilerplate code to enable logging in the streamlit app console 55 log_level = logging.DEBUG if debug == 1 else logging.INFO 56 logger.setLevel(log_level) 57 58 stream_handler = logging.StreamHandler(stream=sys.stdout) 59 stream_handler.setLevel(log_level) 60 61 formatter = logging.Formatter('%(message)s') 62 63 stream_handler.setFormatter(formatter) 64 65 logger.addHandler(stream_handler) 66 logger.propagate = False 67 68configure_logger(0) 69 70def authenticate( 71 openai_api_key: str, activeloop_token: str, activeloop_org_name: str 72) -> None: 73 # Validate all credentials are set and correct 74 # Check for env variables to enable local dev and deployments with shared credentials 75 openai_api_key = ( 76 openai_api_key 77 or os.environ.get('OPENAI_API_KEY') 78 or st.secrets.get('OPENAI_API_KEY') 79 ) 80 activeloop_token = ( 81 activeloop_token 82 or os.environ.get('ACTIVELOOP_TOKEN') 83 or st.secrets.get('ACTIVELOOP_TOKEN') 84 ) 85 activeloop_org_name = ( 86 activeloop_org_name 87 or os.environ.get('ACTIVELOOP_ORG_NAME') 88 or st.secrets.get('ACTIVELOOP_ORG_NAME') 89 ) 90 if not (openai_api_key and activeloop_token and activeloop_org_name): 91 st.session_state['auth_ok'] = False 92 st.error('Credentials neither set nor stored', icon=PAGE_ICON) 93 return 94 try: 95 # Try to access openai and deeplake 96 with st.spinner('Authentifying...'): 97 openai.api_key = openai_api_key 98 openai.Model.list() 99 deeplake.exists( 100 f'hub://{activeloop_org_name}/DataChad-Authentication-Check', 101 token=activeloop_token, 102 ) 103 except Exception as e: 104 logger.error(f'Authentication failed with {e}') 105 st.session_state['auth_ok'] = False 106 st.error('Authentication failed', icon=PAGE_ICON) 107 return 108 # store credentials in the session state 109 st.session_state['auth_ok'] = True 110 st.session_state['openai_api_key'] = openai_api_key 111 st.session_state['activeloop_token'] = activeloop_token 112 st.session_state['activeloop_org_name'] = activeloop_org_name 113 logger.info('Authentification successful!') 114 115def advanced_options_form() -> None: 116 # Input Form that takes advanced options and rebuilds chain with them 117 advanced_options = st.checkbox( 118 'Advanced Options', help='Caution! This may break things!' 119 ) 120 if advanced_options: 121 with st.form('advanced_options'): 122 temperature = st.slider( 123 'temperature', 124 min_value=0.0, 125 max_value=1.0, 126 value=TEMPERATURE, 127 help='Controls the randomness of the language model output', 128 ) 129 col1, col2 = st.columns(2) 130 fetch_k = col1.number_input( 131 'k_fetch', 132 min_value=1, 133 max_value=1000, 134 value=FETCH_K, 135 help='The number of documents to pull from the vector database', 136 ) 137 k = col2.number_input( 138 'k', 139 min_value=1, 140 max_value=100, 141 value=K, 142 help='The number of most similar documents to build the context from', 143 ) 144 chunk_size = col1.number_input( 145 'chunk_size', 146 min_value=1, 147 max_value=100000, 148 value=CHUNK_SIZE, 149 help=( 150 'The size at which the text is divided into smaller chunks ' 151 'before being embedded.\n\nChanging this parameter makes re-embedding ' 152 'and re-uploading the data to the database necessary ' 153 ), 154 ) 155 max_tokens = col2.number_input( 156 'max_tokens', 157 min_value=1, 158 max_value=4069, 159 value=MAX_TOKENS, 160 help='Limits the documents returned from database based on number of tokens', 161 ) 162 applied = st.form_submit_button('Apply') 163 if applied: 164 st.session_state['k'] = k 165 st.session_state['fetch_k'] = fetch_k 166 st.session_state['chunk_size'] = chunk_size 167 st.session_state['temperature'] = temperature 168 st.session_state['max_tokens'] = max_tokens 169 update_chain() 170 171def save_uploaded_file(uploaded_file: UploadedFile) -> str: 172 # streamlit uploaded files need to be stored locally 173 # before embedded and uploaded to the hub 174 if not os.path.exists(DATA_PATH): 175 os.makedirs(DATA_PATH) 176 file_path = str(DATA_PATH / uploaded_file.name) 177 uploaded_file.seek(0) 178 file_bytes = uploaded_file.read() 179 file = open(file_path, 'wb') 180 file.write(file_bytes) 181 file.close() 182 logger.info(f'Saved: {file_path}') 183 return file_path 184 185def delete_uploaded_file(uploaded_file: UploadedFile) -> None: 186 # cleanup locally stored files 187 file_path = DATA_PATH / uploaded_file.name 188 if os.path.exists(DATA_PATH): 189 os.remove(file_path) 190 logger.info(f'Removed: {file_path}') 191 192def handle_load_error(e: str = None) -> None: 193 e = e or 'No Loader found for your data source


constants.py

1from pathlib import Path 2 3APP_NAME = 'DataChad' 4MODEL = 'gpt-3.5-turbo' 5PAGE_ICON = '🤖' 6 7K = 10 8FETCH_K = 20 9CHUNK_SIZE = 1000 10TEMPERATURE = 0.7 11MAX_TOKENS = 3357 12ENABLE_ADVANCED_OPTIONS = True 13 14DATA_PATH = Path.cwd() / 'data' 15DEFAULT_DATA_SOURCE = 'git@github.com:gustavz/DataChad.git' 16 17REPO_URL = 'https://github.com/gustavz/DataChad' 18 19AUTHENTICATION_HELP = f''' 20Your credentials are only stored in your session state.\n 21The keys are neither exposed nor made visible or stored permanently in any way.\n 22Feel free to check out [the code base]({REPO_URL}) to validate how things work. 23''' 24 25USAGE_HELP = f''' 26These are the accumulated OpenAI API usage metrics.\n 27The app uses '{MODEL}' for chat and 'text-embedding-ada-002' for embeddings.\n 28Learn more about OpenAI's pricing [here](https://openai.com/pricing#language-models) 29''' 30 31OPENAI_HELP = ''' 32You can sign-up for OpenAI's API [here](https://openai.com/blog/openai-api).\n 33Once you are logged in, you find the API keys [here](https://platform.openai.com/account/api-keys) 34''' 35 36ACTIVELOOP_HELP = ''' 37You can create an Activeloop account (including 200GB of free database storage) [here](https://www.activeloop.ai/).\n 38Once you are logged in, you find the API token [here](https://app.activeloop.ai/profile/gustavz/apitoken).\n 39The organisation name is your username, or you can create new organisations [here](https://app.activeloop.ai/organization/new/create) 40''' 41 42Copy


Concluding Remarks: Build your Chat with Data Tool, or Use DataChad

DataChad elevates conversing with CSVs, PDFs, JSONs, GitHub repositories, local paths or web URLs to a completely new level. If you've read this far, consider giving DataChad a try.By harnessing the power of embeddings, Deep Lake's vector database for all AI data, large language models (LLMs), and LangChain, DataChad enables users to query any data source easily. DataChad seamlessly transforms any data into text documents, embeds them using OpenAI embeddings, and stores the embeddings as a vector dataset in Activeloop's Deep Lake Cloud. And creates a LangChain, which serves as the context for generating precise responses to user queries. Whether the task at hand is understanding a complex project or seeking quick answers from a single data source, DataChad allows users to pose natural language questions and receive relevant answers in seconds.


How can I deploy a ChatGPT for my Data fully locally?

Local Mode


What is SalesCopilot?

SalesCopilot is a sales call assistant that transcribes audio in real-time and connects the user to a chatbot with full knowledge of the transcript, powered by GPT-3.5 or GPT-4. This live chat allows for highly relevant assistance to be provided within seconds upon the user's request.Additionally, SalesCopilot is able to detect potential objections from the customer (e.g. 'It's too expensive' or 'The product doesn't work for us') and provide well-informed recommendations to the salesperson on how best to handle them. Relying solely on the LLM to come up with these recommendations has some flaws - ChatGPT isn't fine tuned to be a great salesperson, and it may give recommendations that don't align with your personal approach. Integrating it with Deep Lake and a custom knowledge base is the perfect solution - let's dive into how it works!


What Did and Didn't Work while Building an Open-Source Conversational Intelligence Assistant

Before we look at the exact solution we eventually decided on, let's take a glance at the approaches that didn't work, and what we learned from them:


Didn't Work: Naively Splitting the Custom Knowledge Base

To leverage our custom knowledge base, our initial approach was to split the knowledge base into chunks of equal length using LangChain's built-in text splitters. Then we took the detected customer objection, embedded it, and searched the database for those chuhnks that were most relevant. This allowed us to pass relevant excerpts from our knowledge base to the LLM every time we wanted a response, which improved the quality of the responses and made the prompts to the LLM shorter and more efficient. However, our 'naive' approach to splitting the custom knowledge base had a major flaw.To illustrate the issue we faced, let's look at an example. Say we have the following text:1Objection: 'There's no money.' 2It could be that your prospect's business simply isn't big enough or generating enough cash right now to afford a product like yours. Track their growth and see how you can help your prospect get to a place where your offering would fit into their business. 3 4Objection: 'We don't have any budget left this year.' 5A variation of the 'no money' objection, what your prospect's telling you here is that they're having cash flow issues. But if there's a pressing problem, it needs to get solved eventually. Either help your prospect secure a budget from executives to buy now or arrange a follow-up call for when they expect funding to return. 6 7Objection: 'We need to use that budget somewhere else.' 8Prospects sometimes try to earmark resources for other uses. It's your job to make your product/service a priority that deserves budget allocation now. Share case studies of similar companies that have saved money, increased efficiency, or had a massive ROI with you. 9CopyIf we naively split this text, we might end up with individual sections that look like this:1A variation of the 'no money' objection, what your prospect's telling you here is that they're having cash flow issues. But if there's a pressing problem, it needs to get solved eventually. Either help your prospect secure a budget from executives to buy now or arrange a follow-up call for when they expect funding to return. 2 3Objection: 'We need to use that budget somewhere else.' 4CopyHere, we see that the advice does not match the objection. When we try to retrieve the most relevant chunk for the objection 'We need to use that budget somewhere else', this will likely be our top result, which isn't what we want. When we pass it to the LLM, it might be confusing.What we really need to do is split the text in a more sophisticated way, that maintains semantic boundaries between each chunk. This will improve retrieval performance and keep the LLM responses higher quality.


Didn't Work: No Custom Knowledge Base

This was the first solution we tried - instead of using a custom knowledge base, we could completely rely on the LLM.Unfortunately, we ran into some issues: GPT-4 is awesome, but way too slow: To get the highest quality responses without a custom knowledge base, using GPT-4 is the best choice. However, the time it takes for the API to return a response mean that by the time the user gets the advice they need, it's too late. This means we have to use GPT-3.5, which comes with a noticeable drop in quality. Response quality is inconsistent: Relying solely on the LLM, sometimes we get great responses that are exactly what we're looking for. However, without any additional info, guidelines, or domain-specific information, we also get bad responses that aren't very on-topic. Cramming information into the prompt is not token-efficient: Using the OpenAI API, cost is a consideration, as you pay for the amount of tokens in your prompt + the completion. To ground the LLM and keep the responses high-quality, we could fill the prompt with tons of relevant information, how we'd like it to respond in different situations, etc. This solution isn't ideal, because it means every time we query the LLM we have to pass all that information, and all those tokens, to the API. The costs can add up, and GPT-3.5 can get confused if you give it too much info at once. That didn't work - the main issue is that we need a way to efficiently ground the LLM's response. The next thing we tried was to use a custom knowledge base combined with a vector database to pass the LLM relevant info for each individual customer objection.


Creating, Loading, and Querying Our Database for AI

We're going to define a class that handles the database creation, database loading, and database querying.1import os 2import re 3from langchain.embeddings import OpenAIEmbeddings 4from langchain.vectorstores import DeepLake 5 6class DeepLakeLoader: 7 def __init__(self, source_data_path): 8 self.source_data_path = source_data_path 9 self.file_name = os.path.basename(source_data_path) # What we'll name our database 10 self.data = self.split_data() 11 if self.check_if_db_exists(): 12 self.db = self.load_db() 13 else: 14 self.db = self.create_db() 15CopyThere's a few things happening here. First, the data is being processed by a method called split_data:1def split_data(self): 2 ''' 3 Preprocess the data by splitting it into passages. 4 5 If using a different data source, this function will need to be modified. 6 7 Returns: 8 split_data (list): List of passages. 9 ''' 10 with open(self.source_data_path, 'r') as f: 11 content = f.read() 12 split_data = re.split(r'(?=\d+\. )', content) 13 if split_data[0] == '': 14 split_data.pop(0) 15 split_data = [entry for entry in split_data if len(entry) >= 30] 16 return split_data 17CopySince we know the structure of our knowledge base, we use this method to split it into individual entries, each representing an example of a customer objection. When we run our similarity search using the detected customer objection, this will improve the results, as outlined above.After preprocessing the data, we check if we've already created a database for this data. One of the great things about Deep Lake is that it provides us with persistent storage, so we only need to create the database once. If you restart the app, the database doesn't disappear!Creating and loading the database is super easy:1def load_db(self): 2 ''' 3 Load the database if it already exists. 4 5 Returns: 6 DeepLake: DeepLake object. 7 ''' 8 return DeepLake(dataset_path=f'deeplake/{self.file_name}', embedding_function=OpenAIEmbeddings(), read_only=True) 9 10def create_db(self): 11 ''' 12 Create the database if it does not already exist. 13 14 Databases are stored in the deeplake directory. 15 16 Returns: 17 DeepLake: DeepLake object. 18 ''' 19 return DeepLake.from_texts(self.data, OpenAIEmbeddings(), dataset_path=f'deeplake/{self.file_name}') 20CopyJust like that, our knowledge base becomes a vector database that we can now query.1def query_db(self, query): 2 ''' 3 Query the database for passages that are similar to the query. 4 5 Args: 6 query (str): Query string. 7 8 Returns: 9 content (list): List of passages that are similar to the query. 10 ''' 11 results = self.db.similarity_search(query, k=3) 12 content = [] 13 for result in results: 14 content.append(result.page_content) 15 return content 16CopyWe don't want the metadata to be passed to the LLM, so we take the results of our similarity search and pull just the content from them. And that's it! We now have our custom knowledge base stored in a Deep Lake vector database and ready to be queried!


Integrating SalesCopilot with Deep Lake

By using Deep Lake as a vector database, we can quickly and easily retrieve only the most relevant info to provide to the LLM. We can also persist the vector database, so we don't have to re-create it every time we load the app. The knowledge base we're using here is this list of common customer objections. Before we get into the code, here's a rough overview of how it works:First, we take our knowledge base and embed it, storing the embeddings in a Deep Lake vector database. Then, when we detect an objection in the transcript, we embed the objection and use it to search our database, retrieving the most similar guidelines. We then pass those guidelines along with the objection to the LLM and send the result to the user.


Did Work: Intelligent Splitting

In our example text, there is a set structure to each individual objection and its recommended response. Rather than split the text based on size, why don't we split the text based on its structure? We want each chunk to begin with the objection, and end before the 'Objection' of the next chunk. Here's how we could do it:1text = ''' 2Objection: 'There's no money.' 3It could be that your prospect's business simply isn't big enough or generating enough cash right now to afford a product like yours. Track their growth and see how you can help your prospect get to a place where your offering would fit into their business. 4 5Objection: 'We don't have any budget left this year.' 6A variation of the 'no money' objection, what your prospect's telling you here is that they're having cash flow issues. But if there's a pressing problem, it needs to get solved eventually. Either help your prospect secure a budget from executives to buy now or arrange a follow-up call for when they expect funding to return. 7 8Objection: 'We need to use that budget somewhere else.' 9Prospects sometimes try to earmark resources for other uses. It's your job to make your product/service a priority that deserves budget allocation now. Share case studies of similar companies that have saved money, increased efficiency, or had a massive ROI with you. 10''' 11 12# Split the text into a list using the keyword 'Objection: ' 13objections_list = text.split('Objection: ')[1:] # We ignore the first split as it is empty 14 15# Now, prepend 'Objection: ' to each item as splitting removed it 16objections_list = ['Objection: ' + objection for objection in objections_list] 17CopyThis gave us the best results. Nailing the way we split and embed our knowledge base means more relevant documents are retrieved and the LLM gets the best possible context to generate a response from. Now let's see how we integrated this solution with Deep Lake and SalesCopilot!


Leveraging Custom Knowledge Bases with Deep Lake

Integrating SalesCopilot with Deep Lake allows for a significant enhancement of its capabilities, with immediate and relevant responses based on a custom knowledge base. The beauty of this solution is its adaptability. You can curate your knowledge base according to your own unique sales techniques and customer scenarios, ensuring SalesCopilot's responses are perfectly suited for your situation.An efficient vector storage solution is essential to working with large knowledge bases and connecting them to LLM's, allowing the LLM to offer knowledgeable, situation-specific advice. On top of that, Deep Lake's persistent storage means we only create the database once, which saves computational resources and time.In conclusion, the integration of SalesCopilot with Deep Lake creates a powerful tool that combines the speed and intelligence of LLM's with the rapid, precise information retrieval of a vector database. This hybrid system offers a highly adaptable, efficient, and effective solution to handling customer objections. The efficiency and simplicity Deep Lake brings to applications like this alongside its seamless integration make it a top choice for vector storage solutions.


Connecting Our Database to GPT-4

Now, all we need to do is connect our LLM to the database. First, we need to create a DeepLakeLoader instance with the path to our data.1db = DeepLakeLoader('data/salestesting.txt') 2CopyNext, we take the detected objection and use it to query the database:1results = db.query_db(detected_objection) 2CopyTo have our LLM generate a message based off these results and the objection, we use LangChain. In this example, we use a placeholder for the prompt - if you want to check out the prompts used in SalesCopilot, check out the prompts.py file.1from langchain.chat_models import ChatOpenAI 2from langchain.schema import SystemMessage, HumanMessage, AIMessage 3 4chat = ChatOpenAI() 5system_message = SystemMessage(content=objection_prompt) 6human_message = HumanMessage(content=f'Customer objection: {detected_objection} | Relevant guidelines: {results}') 7 8response = chat([system_message, human_message]) 9CopyTo print the response:1print(response.content) 2CopyAnd we're done! In just a few lines of code, we've got our response from the LLM, informed by our own custom knowledge base.If you want to check out the full code for SalesCopilot, click here to visit the GitHub repo.


Announcing Gen AI 360: Activeloop Collaborates with Towards AI to Launch 3 Free Skill Certification Courses on LangChain, Vector Databases, & Foundational Models

Mountain View, CA. June 20 2023 - Activeloop is proud to announce the launch of industry-pioneering educational certifications on Large Language Models, Deep Lake, the vector database for all AI data, and LangChain, a framework for developing applications powered by language models. The certification program, available at no cost, is designed as part of the Gen AI 360: Foundational Model Certification program in collaboration with Intel and Towards AI. The first installment of the course combines a comprehensive overview of foundational model theory and practical projects with LangChain and Deep Lake, equipping AI developers with the tools they need to apply Large Language Models across industries.'Reaching over 385,000 AI developers monthly, we're passionate about educating and upskilling engineers in this rapidly growing field. That is why we designed a practical course engineers can take to implement AI into their company processes or use Foundational Models to build entirely new products,' said Louie Peters, CEO of Towards AI.'Every company will be adding Foundational Models and vector databases to their day-to- day operations and the products they build very soon. Upon course completion, Deep Lake Certified developers can harness the full potential of Large Language Models and advanced AI technologies like Deep Lake and LangChain. Companies that want to achieve a competitive advantage should include these frameworks in their toolset. We're already seeing solutions powered by LangChain and Deep Lake rapidly gaining adoption,' said Davit Buniatyan, CEO of Activeloop.'I believe engineers and technology executives could greatly benefit from taking this course to stay at the forefront of AI,' said Arijit Bandyopadhyay, CTO – Enterprise Analytics & AI, Head of Strategy – Cloud & Enterprise, DCAI Group at Intel Corporation. 'Intel continues to be at the vanguard of AI and new technology adoption. This Foundational Model Certification could help better equip the next generation of innovators with what they need to succeed with Generative AI and Large Language Models. It could also contribute to the broader adoption of AI applications and solutions across various industries.'Activeloop's Deep Lake is designed to combine the best of both data lakes and vector databases, enabling companies to build their own data flywheels to fine-tune their Large Language Models on complex data beyond embeddings - such as text, image, video, or audio, and power AI products. LangChain, in its turn, seamlessly connects Deep Lake datasets with Foundational Models in diverse use cases - from understanding GitHub repositories to analyzing financial statements.Created with support from industry leaders, the certification will foster the adoption of cutting-edge AI technology within any engineering organization across industries. To learn more about the Deep Lake Foundational Model Certification and to sign up for free, visit Gen AI 360 Certification page.


About Activeloop

Activeloop is building Deep Lake, a vector database for all AI data. With Deep Lake, ML teams can store and manage complex data, such as embeddings, text, images, audio, videos, annotations, or tabular data, in a deep learning-native format and stream them to ML frameworks in real time. Matterport, Yale University, MILA Quebec, & others use Deep Lake.


About Towards AI

Towards AI simplifies AI learning & AI product building with 30-50 weekly blogs & tutorials for 385k followers and their 60K-member Discord community. Towards AI addresses AI students' and practitioners' challenges through community-driven practical courses and a Jobs board.


Didn't Work: Instructing GPT-4 To Generate Stable Diffusion Prompts

Initially, it seemed like it might be possible to send the LLM the text of our book and tell it to generate a prompt for each page. However, this didn't work for a few reasons: Stable Diffusion released in 2022: While it might seem like Stable Diffusion is already 'old news', to GPT-3.5 and GPT-4 it's in the future. Look at GPT-4's response to the question, 'What is Stable Diffusion?': Teaching the LLM how to prompt is difficult: It's possible to instruct the LLM to generate prompts without the LLM knowing what Stable Diffusion is; giving it the exact format to generate a prompt with has decent results. Unfortunately, the often injects plot details or non-visual content into the prompts, no matter how often you tell it not to. These details skew the relevance of the prompts and negatively impact the quality of the generated images.


What Did and Didn't Work while Building FableForge?

Before we look at the exact solution we eventually decided on, let's take a glance at the approaches that didn't work and what we learned from them:


How FableForge Uses Functions

For our Stable Diffusion prompts, we need structured data that strictly adheres to specific rules - a function is perfect for that! Let's take a look at one of the functions we used:1get_visual_description_function = [{ 2 'name': 'get_passage_setting', 3 'description': 'Generate and describe the visuals of a passage in a book. Visuals only, no characters, plot, or people.', 4 'parameters': { 5 'type': 'object', 6 'properties': { 7 'setting': { 8 'type': 'string', 9 'description': 'The visual setting of the passage, e.g. a green forest in the pacific northwest', 10 }, 11 'time_of_day': { 12 'type': 'string', 13 'description': 'The time of day of the passage, e.g. nighttime, daytime. If unknown, leave blank.', 14 }, 15 'weather': { 16 'type': 'string', 17 'description': 'The weather of the passage, eg. rain. If unknown, leave blank.', 18 }, 19 'key_elements': { 20 'type': 'string', 21 'description': 'The key visual elements of the passage, eg tall trees', 22 }, 23 'specific_details': { 24 'type': 'string', 25 'description': 'The specific visual details of the passage, eg moonlight', 26 } 27 }, 28 'required': ['setting', 'time_of_day', 'weather', 'key_elements', 'specific_details'] 29 } 30}] 31CopyWith this, we can send the chat model a page from our book, the function, and instructions to infer the details from the provided page. In return, we get structured data that we can use to form a great Stable Diffusion prompt!


Meet FableForge, AI Picture Books Generator powered by OpenAI, LangChain, Stable Diffusion, & Deep Lake

Imagine a world where children's picture books are created on-demand by children from a single prompt. With each generated image, the text and prompt pairs are stored for further finetuning if the child likes the story - to fit one human's imagination perfectly.This is the grand vision of FableForge.FableForge is an open-source app that generates children's picture books from a single prompt. First, GPT-3.5/4 is instructed to write a short children's book. Then, using the new function calling feature OpenAI just announced, the text from each book page is transformed into a prompt for Stable Diffusion. These prompts are sent to Replicate, corresponding images are generated, and all the elements are combined for a complete picture book. The matching images and prompts are stored in a Deep Lake vector database, allowing easy storing and visualizing of multimodal data (image and text pairs). Beyond that, the generated data can be streamed to machine learning frameworks in real time while training, to finetune our generative AI model. While the latter is beyond the scope of this example, we'd love to cover how it all works together.But first…


LangChain and OpenAI Function Calling

When we created FableForge, OpenAI announced the new function calling capabilities. Since then, LangChain - the open-source library we use to interact with OpenAI's Large Language Models - has added even better support for using functions. Our implementation of functions using LangChain is as follows: Define our function: First, we define our function, as we did above with get_visual_description_function. Give the chat model access to our function: Next, we call our chat model, including our function within the functions parameter, like so: 1 2response= self.chat([HumanMessage(content=f'{page}')],functions=get_visual_description_function) 3 4Copy Parse the JSON object: When the chat model uses our function, it provides the output as a JSON object. To convert the JSON object into a Python dictionary containing the function output, we can do the following: 1 2function_dict = json.loads(response.additional_kwargs['function_call']['arguments']) 3 4CopyIn the function, we defined earler, ‘setting' was one of the parameters. To access this, we can write:1setting = function_dict['setting'] 2CopyAnd we're done! We can follow the same steps for the each of the other parameters to extract them.


What is OpenAI Function Calling?

On June 13th, OpenAI announced a huge update to the chat completions API - function calling!. This means we can provide the chat model with a function, and the chat model will output a JSON object according to that function's parameters.Now, the chat models can interpret natural language input into a structured format suitable for external tools, APIs, or database queries. The chat models are designed to detect when a function needs to be called based on the user's input and can then respond with JSON that conforms to the described function's signature.In essence, function calling is a way to bridge the gap between unstructured language input and structured, actionable output that other systems, tools, or services can use.


Perfecting the Process: Using Deep Lake for Storage and Analysis

The final step breakthrough for perfecting FableForge was using Deep Lake to store the generated images and text. With Deep Lake, we could store multiple modalities of data, such as images and text, in the cloud. The web-based UI provided by Deep Lake made it incredibly straightforward to display, analyze, and optimize the generated images and prompts, improving the quality of our picture book output. For future Stable Diffusion endeavors, we now have a decently-sized dataset showing us what prompts work, and what prompts don't!


Second Component: Combining Text and Images

Now that we have our text and images, we can open up MS Paint and copy-paste the text onto each corresponding image. That would be different, and it's also time-consuming; instead, let's do it programmatically. In pdf_gen_utils.py, we turn our ingredients into a proper book in these steps: Text Addition and Image Conversion: First, we take each image, resize it, and apply a fading mask to the bottom - a white space for us to place our text. We then add the text to the faded area, convert it into a PDF, and save it. Cover Generation: A book needs a cover that follows a different format than the rest of the pages. Instead of a fading mask, we take the cover image and place a white box over a portion for the title to be placed within. The other steps (resizing and saving as PDF) are the same as above. PDF Assembly: Once we have completed all the pages, we combine them into a single PDF and delete the files we no longer need.


Third Component: Saving to Deep Lake

Now that we have finalized our picture book, we want to store the images and prompts in Deep Lake. For this, we created a SaveToDeepLake class:1import deeplake 2 3class SaveToDeepLake: 4 def __init__(self, buildbook_instance, name=None, dataset_path=None): 5 self.dataset_path = dataset_path 6 try: 7 self.ds = deeplake.load(dataset_path, read_only=False) 8 self.loaded = True 9 except: 10 self.ds = deeplake.empty(dataset_path) 11 self.loaded = False 12 13 self.prompt_list = buildbook_instance.sd_prompts_list 14 self.images = buildbook_instance.source_files 15 16 def fill_dataset(self): 17 if not self.loaded: 18 self.ds.create_tensor('prompts', htype='text') 19 self.ds.create_tensor('images', htype='image', sample_compression='png') 20 for i, prompt in enumerate(self.prompt_list): 21 self.ds.append({'prompts': prompt, 'images': deeplake.read(self.images[i])}) 22 23CopyWhen initialized, the class first tries to load a Deep Lake dataset from the provided path. If the dataset doesn't exist, a new one is created.If the dataset already existed, we simply added the prompts and images. The images can be easily uploaded using deeplake.read(), as Deep Lake is built to handle multimodal data.If the dataset is empty, we must first create the tensors to store our data. In this case, we create a tensor ‘prompts' for our prompts and ‘images' for our images. Our images are in PNG format, so we set sample_compression to 'png'.Once uploaded, we can view them in the UI, as shown above.All code can be found in the deep_lake_utils.py file.


First Component: AI Book Generation

All code for this component can be found in the api_utils.py file. Text Generation: To generate the text for the children's book, we use LangChain and the ChatOpenAI chat model. 1def get_pages(self): 2 pages = self.chat([HumanMessage(content=f'{self.book_text_prompt} Topic: {self.input_text}')]).content 3 return pages 4 5Copyself.book_text_prompt is a simple prompt instructing the model to generate a children's story. We specify the number of pages inside the prompt and what format the text should come in. The full prompt can be found in the prompts.py file. Visual Prompts Generation: To produce the prompts we will use with Stable Diffusion, we use functions, as outlined above. First, we send the whole book to the model: 1 def get_prompts(self): 2 base_atmosphere = self.chat([HumanMessage(content=f'Generate a visual description of the overall lightning/atmosphere of this book using the function.' 3 f'{self.book_text}')], functions=get_lighting_and_atmosphere_function) 4 summary = self.chat([HumanMessage(content=f'Generate a concise summary of the setting and visual details of the book')]).content 5 6CopySince we want our book to have a consistent style throughout, we will take the contents of base_atmosphere and append it to each individual prompt we generate later on. To further ensure our visuals stay consistent, we generate a concise summary of the visuals of the book. This summary will be sent to the model later on, accompanying each individual page, to generate our Stable Diffusion prompts.1 def generate_prompt(page, base_dict): 2 prompt = self.chat([HumanMessage(content=f'General book info: {base_dict}. Passage: {page}. Infer details about passage if they are missing, ' 3 f'use function with inferred detailsm as if you were illustrating the passage.')], 4 functions=get_visual_description_function) 5CopyThis method will be called for each individual page of the book. We send the model the info we just gathered along with a page from the book, and give it access to the get_visual_description_function function. The output of this will be a JSON object containing all the elements we need to form our prompts!1 for i, prompt in enumerate(prompt_list): 2 entry = f'{prompt['setting']}, {prompt['time_of_day']}, {prompt['weather']}, {prompt['key_elements']}, {prompt['specific_details']}, ' \ 3 f'{base_dict['lighting']}, {base_dict['mood']}, {base_dict['color_palette']}, in the style of {style}' 4CopyHere, we combine everything. Now that we have our prompts, we can send them to Replicate's Stable Diffusion API and get our images. Once those are downloaded, we can move on to the next step.


Building FableForge

FableForge's open-sourced code is located here.FableForge consists of four main components: The generation of the text and images The combining of the text and images to create the book Saving the images and prompts to the Deep Lake dataset The UI Let's take a look at each component individually, starting with the generation of the text and images. Here's a high-level overview of the architecture:


Final Component: Streamlit UI

To create a quick and simple UI, we used Streamlit. The complete code can be found in main.py.Our UI has three main features: Prompt Format: In this text input box, we allow the user to specify the prompt to generate the book based on. This could be anything from a theme, a plot, a time, and so on. Book Generation: Once the user has input their prompt, they can click the Generate button to generate the book. The app will run through all of the steps outlined above until it completes the generation. The user will then have a button to download their finished book. Saving to Deep Lake: The user can click the Save to Deep Lake checkbox to save the prompts and images to their Deep Lake vector database. Once the book is generated, this will run in the background, filling the user's dataset with all their generated prompts and images. Streamlit is an excellent choice for quick prototyping and smaller projects like FableForge - the entire UI is less than 60 lines of code!


Conclusion: The Future of AI-Generated Picture Books with FableForge & Deep Lake

Developing FableForge was a perfect example of how new AI tools and methodologies can be leveraged to overcome hurdles. By leveraging the power of LangChain, OpenAI's function calling feature, Stable Diffusion's image generation abilities, and Deep Lake's multimodal dataset storage and analysis capabilities, we created an app that opens up a new frontier in children's picture book creation.Everyone can create an app like this - we did it, too. What will matter for you in the end, however, is having the data as the moat - and using the data you gether from your users to finetune models - providing them personal, curated experiences as they immerse themsleves into fiction. This is where Deep Lake comes into place. With its ‘data lake' features of visualization of multimodal data and streaming capability, Deep Lake enables teams to finetune their LLM performance or train entirely new ML models in a cost-effective manner.


Getting the Data for the Song Recommendation Engine

To get our songs, we scraped https://www.disneyclips.com/lyrics/, a website containing all the lyrics for all Disney songs ever made. The code is here, and it relies on asyncio to speed up things. We won't focus too much on it, since it's not central to our story (plays Encanto music we don't talk about asyncio, no, no, no…).Then, we used Spotify Python APIs to get all the embedding URLs for each song into the 'Disney Hits' Playlist. We removed all the songs we had scraped but were not in this playlist. By doing so, we end up with 85 songs.We end up with a json looking like this.1json 2{ 3 'Aladdin': [ 4 { 5 'name': 'Arabian Nights', 6 'text': 'Oh, I come from a land, from a faraway place. Where the caravan camels roam... ', 7 'embed_url': 'https://open.spotify.com/embed/track/0CKmN3Wwk8W4zjU0pqq2cv?utm_source=generator' 8 }, 9 ... 10 ], 11Copy


Data Encoding for the Recommendation Engine

We were looking for the best way to retrieve the songs. We evaluated different approaches. We used Activeloop DeepLake Vector Database - more specifically, its implementation in LangChain.Creating the dataset is pretty straightforward. Given the previous json file, we proceed to embed the text field using langchain.embeddings.openai.OpenaAIEmbeddings and add all the rest of the keys/values as metadata1python 2from langchain.embeddings.openai import OpenAIEmbeddings 3from langchain.llms import OpenAI 4from langchain.vectorstores import DeepLake 5 6def create_db(dataset_path: str, json_filepath: str) -> DeepLake: 7 with open(json_filepath, 'r') as f: 8 data = json.load(f) 9 10 texts = [] 11 metadatas = [] 12 13 for movie, lyrics in data.items(): 14 for lyric in lyrics: 15 texts.append(lyric['text']) 16 metadatas.append( 17 { 18 'movie': movie, 19 'name': lyric['name'], 20 'embed_url': lyric['embed_url'], 21 } 22 ) 23 24 embeddings = OpenAIEmbeddings(model='text-embedding-ada-002') 25 26 db = DeepLake.from_texts( 27 texts, embeddings, metadatas=metadatas, dataset_path=dataset_path 28 ) 29 30 return db 31CopyTo load it, we can simply:1def load_db(dataset_path: str, *args, **kwargs) -> DeepLake: 2 db = DeepLake(dataset_path, *args, **kwargs) 3 return db 4CopyMy dataset_path is hub://<ACTIVELOOP_ORGANIZATION_ID>/<DATASET_NAME>, but you can also store it locally. To store Deep Lake datasets locally, check out this doc here.


Similarity Search of Direct Embeddings

This approach was straightforward. We create embeddings for the lyrics and the user input with gpt3 and do a similarity search. Unfortunately, we noticed terrible suggestions because we want to match the user's emotions to the songs rather than precisely what it says.For example, if we search for similar songs using 'I am Sad', we will see very similar scores across all documents:1 db.similarity_search_with_score('I am happy', distance_metric='cos', k=100) 2CopyIf we plot the scores using a box plot, we will see they mostly are around 0.74.While the first ten songs do not match so well1The World Es Mi Familia 0.7777353525161743 2Go the Distance 0.7724394202232361 3Waiting on a Miracle 0.7692896127700806 4Happy Working Song 0.7679054141044617 5In Summer 0.7620900273323059 6So Close 0.7601353526115417 7When I Am Older 0.7582702040672302 8How Far I'll Go 0.7560539245605469 9You're Welcome 0.7539903521537781 10What Else Can I Do? 0.7535801529884338 11Copy


Using ChatGPT as a Retrieval System

We also tried to nuke the whole lyrics into ChatGPT and asked it to return matching songs with the user input. We had first to create a one-sentence summary of each lyric to fit into 4096 tokens. It resulted in around 3k tokens per request (0.006$). It follows the prompt template, which is very simple but very long. The {songs} variable holds the JSON with all the songs.1You act like a song retrieval system. We want to propose three songs based on the user input. We provide you a list of songs with their themes in the format <MOVIE_NAME>;<SONG_TITLE>:<SONG_THEMES>. To match the user input to the song, try to find themes/emotions from it and imagine what emotions the user may have and what song may be lovely to listen to. Add a bit of randomness to your decision. 2If you don't find a match, provide your best guess. Try to look at each song's themes to offer more variations in the match. Please only output songs contained in the following list. 3 4{songs} 5 6Given an input, output three songs as a list that goes well with the input. The list of songs will be used to retrieve them from our database. The type of reply is List[str, str, str]. Please follow the following example formats. 7 8Examples: 9Input: 'Today I am not feeling great.' 10['<MOVIE_NAME>;<SONG_TITLE>', '<MOVIE_NAME>;<SONG_TITLE>', '<MOVIE_NAME>;<SONG_TITLE>'] 11Input: 'I am great today' 12['<MOVIE_NAME>;<SONG_TITLE>', '<MOVIE_NAME>;<SONG_TITLE>', '<MOVIE_NAME>;<SONG_TITLE>'] 13 14The user input is {user_input} 15CopyThat did work okayish but was overkill. Later on, we also tried emotional encoding we will discuss in the next section, which had comparable performance.


What Did Work: Similarity Search of Emotions Embeddings

Finally, we arrived at an inexpensive approach to run, which gives good results.  We convert each lyric to a list of 8 emotions using ChatGPT.  The prompt is the following1I am building a retrieval system.  Given the following song lyric 2 3{song} 4 5You are tasked to produce a list of 8 emotions that I will later use to retrieve the song.  6 7Please provide only a list of comma-separated emotions.  8CopyFor example, using the 'Arabian Nights' from Aladdin (shown in the previous section), we obtained 'nostalgic, adventurous, exotic, intense, romantic, mysterious, whimsical, passionate'.We then embedded each emotion for each song with GPT3.5-turbo and stored it with Deep Lake.The entire script is hereThen, we need to convert the user input to a list of emotions.  We used ChatGPT again with a custom prompt.1text 2We have a simple song retrieval system.  It accepts eight emotions.  You are tasked to suggest between 1 and 4 emotions to match the users' feelings.  Suggest more emotions for longer sentences and just one or two for small ones, trying to condense the central theme of the input.  3 4Examples: 5 6Input: 'I had a great day!' 7'Joy' 8Input: 'I am exhausted today and not feeling well.' 9'Exhaustion, Discomfort, and Fatigue' 10Input: 'I am in Love' 11'Love' 12 13Please, suggest emotions for input = '{user_input}', and reply ONLY with a list of emotions/feelings/vibes.  14CopyHere we tasked the model to provide between one and four emotions.  This worked best empirically, given the fact that most inputs are short.Let's see some examples:1'I'm happy and sad today' -> 'Happiness, Sadness' 2'hey, rock you' -> 'Energy, excitement, enthusiasm.' 3'I need to cry' -> 'Sadness, Grief, Sorrow, Despair.' 4CopyThen we used these emotions to perform the similarity search on the vector database.1python 2user_input = 'I am happy' 3# We use chatGPT to get emotions from a user's input 4emotions = chain.run(user_input=user_input) 5# We find the k more similar song 6matches = db.similarity_search_with_score(emotions, distance_metric='cos', k=k) 7CopyThese are the scores obtained from that search (k=100).  They are more spread apart.And the songs make more sense.1Down in New Orleans (Finale) 0.9068354368209839 2Happy Working Song 0.9066014885902405 3Love is an Open Door 0.8957026600837708 4Circle of Life 0.8907418251037598 5Where You Are 0.8890194892883301 6In Summer 0.8889626264572144 7Dig a Little Deeper 0.8887585401535034 8When We're Human 0.8860496282577515 9Hakuna Matata 0.8856213688850403 10The World Es Mi Familia 0.884093165397644 11CopyWe also implement some postprocessing.  We first filter out the low-scoring ones.1python 2def filter_scores(matches: Matches, th: float = 0.8) -> Matches: 3 return [(doc, score) for (doc, score) in matches if score > th] 4 5matches = filter_scores(matches, 0.8) 6CopyTo add more variations, aka only sometimes recommend the first one, we need to sample from the list of candidate matches


To do so, we first ensure the scores sum to one by dividing by their sum.1python 2def normalize_scores_by_sum(matches: Matches) -> Matches: 3 scores = [score for _, score in matches] 4 tot = sum(scores) 5 return [(doc, (score / tot)) for doc, score in matches] 6CopyThen we sample n songs using a modified version of np.random.choice(..., p=scores), basically everything we sample we remove the element we have sampled .  This ensures we don't sample two times the same element.1python 2docs, scores = zip(*matches) 3docs = weighted_random_sample( 4 np.array(docs), np.array(scores), n=number_of_displayed_songs 5).tolist() 6for doc in docs: 7 print(doc.metadata['name']) 8CopyAnd finally, we have our songs.  Then, we created a web app using Streamlit, and we hosted the app on an Hugging Face space.  Go give it a try! :)


Conclusion: Technology Choice Matters When Building a Recommendation Engine with LangChain and Deep Lake

While we explained how to mix these technologies to create a song recommendation system, you can apply the same principles to more use cases. With Deep Lake's multi-modality, you can embed store multiple embeddings to the same set of lyrics, or even incorporate additional factors such as embeddings based on song tempo, instruments used, and more!The main takeaway is understanding how to leverage LLMs to make the data work for you by transforming it to fit your task better. This was crucial for us since only after we converted both users' inputs and songs' lyrics to a list of emotions were we able to have suitable matches.That's all, folks 🎉Thanks for reading, and see you in the next one 💜 Francesco


3 Approaches to Matching Moods to Songs

The next step was to find a way to match our songs with a given user input. In this tutorial, we tried 3 approaches so you don't have to! Ultimately, we found a cheap way that worked qualitatively well. So let's start with the failures 😅


# Multimodal Financial Document Analysis and Recall; Tesla Investor Presentations
In this lesson, we will explore the application of the Retrieval-augmented Generation (RAG) method in processing a company's financial information contained within a PDF document. The process includes extracting critical data from a PDF file (like text, tables, graphs, etc.) and saving them in a vector store database such as Deep Lake for quick and efficient retrieval. Next, a RAG-enabled bot can access stored information to respond to end-user queries. This task requires diverse tools, including [Unstructured.io](http://unstructures.io/) for text/table extraction, OpenAI's GPT-4V for extracting information from graphs, and LlamaIndex for developing a bot with retrieval capabilities. As previously mentioned, data preprocessing plays a significant role in the RAG process. So, we start by pulling data from a PDF document. The content of this lesson focuses on demonstrating how to extract data from a single PDF document for ease of understanding. Nevertheless, the accompanying notebook provided after the lesson will analyze three separate reports, offering a broader scope of information.


# Extracting Data
Extracting textual data is relatively straightforward, but processing graphical elements such as line or bar charts can be more challenging. The latest OpenAI model equipped with vision processing, GPT-4V, is valuable for visual elements. We can feed the slides to the model and ask it to describe it in detail, which then will be used to complement the textual information. This lesson uses Tesla's [Q3 financial report](https://digitalassets.tesla.com/tesla-contents/image/upload/IR/TSLA-Q3-2023-Update-3.pdf) as the source document. It is possible to download the document using the `wget` command. ```bash wget https://digitalassets.tesla.com/tesla-contents/image/upload/IR/TSLA-Q3-2023-Update-3.pdf ``` <aside> 💡 The preprocessing tasks outlined in the next section might be time-consuming and necessitate API calls to OpenAI endpoints, which come with associated costs. To mitigate this, we have made the preprocessed dataset and the checkpoints of the output of each section available at the end of this lesson, allowing you to utilize them with the provided notebook. </aside>


# Extracting Data
## 1. Text/Tables
The `unstructured` package is an effective tool for extracting information from PDF files. It requires two tools, `poppler`and `tesseract`, that help render PDF documents. We suggest setting up these packages on [Google Colab](https://colab.research.google.com/), freely available for students to execute and experiment with code. We will briefly mention the installation of these packages on other operating systems. Let's install the utilities and their dependencies using the following commands. ```bash apt-get -qq install poppler-utils apt-get -qq install tesseract-ocr pip install -q unstructured[all-docs]==0.11.0 fastapi==0.103.2 kaleido==0.2.1 uvicorn==0.24.0.post1 typing-extensions==4.5.0 pydantic==1.10.13 ``` <aside> 💡 These packages are easy to install on Linux and Mac operating systems using `apt-get` and `brew`. However, they are more complex to install on Windows OS. You can follow the below instructions for a step-by-step guide if you use Windows. [[Installing Poppler on Windows](https://towardsdatascience.com/poppler-on-windows-179af0e50150)] [[Installing Tesseract on Windows](https://indiantechwarrior.com/how-to-install-tesseract-on-windows/)] </aside> The process is simple after installing all the necessary packages and dependencies. We simply use the `partition_pdf` function, which extracts text and table data from the PDF and divides it into multiple chunks. We can customize the size of these chunks based on the number of characters. ```python from unstructured.partition.pdf import partition_pdf raw_pdf_elements = partition_pdf( filename="./TSLA-Q3-2023-Update-3.pdf", # Use layout model (YOLOX) to get bounding boxes (for tables) and find titles # Titles are any sub-section of the document infer_table_structure=True, # Post processing to aggregate text once we have the title chunking_strategy="by_title", # Chunking params to aggregate text blocks # Attempt to create a new chunk 3800 chars # Attempt to keep chunks > 2000 chars # Hard max on chunks max_characters=4000, new_after_n_chars=3800, combine_text_under_n_chars=2000 ) ``` The previous code identifies and extracts various elements from the PDF, which can be classified into CompositeElements (the textual content) and Tables. We use the `[Pydantic](https://docs.pydantic.dev/latest/)` package to create a new data structure that stores information about each element, including their `type` and `text`. The code below iterates through all extracted elements, keeping them in a list where each item is an instance of the `Element` type. ```python from pydantic import BaseModel from typing import Any # Define data structure class Element(BaseModel): type: str text: Any # Categorize by type categorized_elements = [] for element in raw_pdf_elements: if "unstructured.documents.elements.Table" in str(type(element)): categorized_elements.append(Element(type="table", text=str(element))) elif "unstructured.documents.elements.CompositeElement" in str(type(element)): categorized_elements.append(Element(type="text", text=str(element))) ``` Creating the `Element` data structure enables convenient storage of the additional information, which can be beneficial for identifying the source of each answer, whether it is derived from texts, tables, or figures.


# Extracting Data
## 2. Graphs
The next step is gathering information from the charts to add context. The primary challenge is extracting images from the pages to feed into OpenAI's endpoint. A practical approach is to convert the PDF to images and pass each page to the model, inquiring if it detects any graphs. If it identifies one or more charts, the model can describe the data and the trends they represent. If no graphs are detected, the model will return an empty array as an indication. <aside> 💡 A drawback of this approach is that it increases the number of requests to the model, consequently leading to higher costs. The issue is that each page must be processed, regardless of whether it contains graphs, which is not an efficient approach. It is possible to reduce the cost by manually flagging the pages. </aside> The initial step involves installing the `pdf2image` package to convert the PDF into images. This also requires the `poppler` tool, which we have already installed. ```bash !pip install -q pdf2image==1.16.3 ``` The code below uses the `convert_from_path` function, which takes the path of a PDF file. We can iterate over each page and save it as a PNG file using the `.save()` method. These images will be saved in the `./pages` directory. Additionally, we define the `pages_png` variable that holds the path of each image. ```python import os from pdf2image import convert_from_path os.mkdir("./pages") convertor = convert_from_path('./TSLA-Q3-2023-Update-3.pdf') for idx, image in enumerate( convertor ): image.save(f"./pages/page-{idx}.png") pages_png = [file for file in os.listdir("./pages") if file.endswith('.png')] ``` Defining a few helper functions and variables is necessary before sending the image files to the OpenAI API. The `headers` variable will contain the OpenAI API Key, enabling the server to authenticate our requests. The `payload` carries configurations such as the model name, the maximum token limit, and the prompts. It instructs the model to describe the graphs and generate responses in JSON format, addressing scenarios like encountering multiple graphs on a single page or finding no graphs at all. We will add the images to the `payload` before sending the requests. Finally, there is the `encode_image()` function, which encodes the images in base64 format, allowing them to be processed by OpenAI. ```python headers = { "Content-Type": "application/json", "Authorization": "Bearer " + str( os.environ["OPENAI_API_KEY"] ) } payload = { "model": "gpt-4-vision-preview", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "You are an assistant that find charts, graphs, or diagrams from an image and summarize their information. There could be multiple diagrams in one image, so explain each one of them separately. ignore tables." }, { "type": "text", "text": 'The response must be a JSON in following format {"graphs": [<chart_1>, <chart_2>, <chart_3>]} where <chart_1>, <chart_2>, and <chart_3> placeholders that describe each graph found in the image. Do not append or add anything other than the JSON format response.' }, { "type": "text", "text": 'If could not find a graph in the image, return an empty list JSON as follows: {"graphs": []}. Do not append or add anything other than the JSON format response. Dont use coding "```" marks or the word json.' }, { "type": "text", "text": "Look at the attached image and describe all the graphs inside it in JSON format. ignore tables and be concise." } ] } ], "max_tokens": 1000 } # Function to encode the image to base64 format def encode_image(image_path): with open(image_path, "rb") as image_file: return base64.b64encode(image_file.read()).decode('utf-8') ``` The remaining steps include: 1) utilizing the `pages_png` variable to loop through the images, 2) encoding the image into base64 format, 3) adding the image into the payload, and finally, 4) sending


# Extracting Data
## 2. Graphs
the request to OpenAI and handling its responses. We will use the same `Element` data structure to store each image's type (graph) and the text (descriptions of the graphs). ```python graphs_description = [] for idx, page in tqdm( enumerate( pages_png ) ): # Getting the base64 string base64_image = encode_image(f"./pages/{page}") # Adjust Payload tmp_payload = copy.deepcopy(payload) tmp_payload['messages'][0]['content'].append({ "type": "image_url", "image_url": { "url": f "data:image/png;base64,{base64_image}" } }) try: response = requests.post("https://api.openai.com/v1/chat/completions", headers=headers, json=tmp_payload) response = response.json() graph_data = json.loads( response['choices'][0]['message']['content'] )['graphs'] desc = [f"{page}\n" + '\n'.join(f"{key}: {item[key]}" for key in item.keys()) for item in graph_data] graphs_description.extend( desc ) except: # Skip the page if there is an error. print("skipping... error in decoding.") continue; graphs_description = [Element(type="graph", text=str(item)) for item in graphs_description] ```


# Store on Deep Lake
This section will utilize the Deep Lake vector database to store the collected information and their embeddings. These embedding vectors convert pieces of text into numerical representations that capture their meaning, enabling similarity metrics such as cosine similarity to identify documents with close relationships. For instance, a prompt inquiring about a company's total revenue would result in high cosine similarity with a database document stating the revenue amount as X dollars. The data preparation is complete with the extraction of all crucial information from the PDF. The next step involves combining the output from the previous sections, resulting in a list containing 41 entries. ```python all_docs = categorized_elements + graphs_description print( len( all_docs ) ) ``` ``` 41 ``` Given that we are using LlamaIndex, we can use its integration with Deep Lake to create and store the dataset. Begin by installing LlamaIndex and deeplake packages along with their dependencies. ```bash !pip install -q llama_index==0.9.8 deeplake==3.8.8 cohere==4.37 ``` Before using the libraries, it's essential to configure the `OPENAI_API_KEY` and `ACTIVELOOP_TOKEN` variables in the environment. Remember to substitute the placeholder values with your actual keys from the respective platforms. ```python import os os.environ["OPENAI_API_KEY"] = "<Your_OpenAI_Key>" os.environ["ACTIVELOOP_TOKEN"] = "<Your_Activeloop_Key>" ``` The integration of LlamaIndex enables the use of `DeepLakeVectorStore` class, which is designed to create a new dataset. Simply enter your organization ID, which by default is your Activeloop username, in the code provided below. This code will generate an empty dataset, ready to store documents. ```python from llama_index.vector_stores import DeepLakeVectorStore # TODO: use your organization id here. (by default, org id is your username) my_activeloop_org_id = "<YOUR-ACTIVELOOP-ORG-ID>" my_activeloop_dataset_name = "tsla_q3" dataset_path = f"hub://{my_activeloop_org_id}/{my_activeloop_dataset_name}" vector_store = DeepLakeVectorStore( dataset_path=dataset_path, runtime={"tensor_db": True}, overwrite=False) ``` ``` Your Deep Lake dataset has been successfully created! ``` Next, we must pass the created vector store to a `StorageContext` class. This class serves as a wrapper to create storage from various data types. In our case, we're generating the storage from a vector database, which is accomplished simply by passing the created database instance using the `.from_defaults()` method. ```python from llama_index.storage.storage_context import StorageContext storage_context = StorageContext.from_defaults(vector_store=vector_store) ``` To store our preprocessed data, we must transform them into LlamaIndex `Documents` for compatibility with the library. The LlamaIndex `Document` is an abstract class that acts as a wrapper for various data types, including text files, PDFs, and database outputs. This wrapper facilitates the storage of valuable information with each sample. In our case, we can include a metadata tag to hold extra details like the data type (text, table, or graph) or denote document relationships. This approach simplifies the retrieval of these details later. As shown in the code below, you can employ built-in classes like `SimpleDirectoryReader` to automatically read files from a specified path or proceed manually. It will loop through our list containing all the extracted information and assign text and a category to each document. ```python from llama_index import Document documents = [Document(text=t.text, metadata={"category": t.type},) for t in categorized_elements] ``` Lastly, we can utilize the `VectorStoreIndex` class to generate embeddings for the documents and employ the database instance to store these values. By default, it uses OpenAI's Ada model to create the embeddings. ```python from llama_index import VectorStoreIndex index = VectorStoreIndex.from_documents( documents, storage_context=storage_context ) ``` ``` Uploading data to deeplake dataset. 100%|██████████| 29/29 [00:00<00:00, 46.26it/s] \Dataset(path='hub://alafalaki/tsla_q3-nograph', tensors=['text', 'metadata', 'embedding', 'id']) tensor htype shape dtype compression ------- ------- ------- ------- ------- text text (29, 1) str None metadata json (29, 1) str None embedding embedding (29, 1536) float32 None id text (29, 1) str None ``` <aside> 💡 The dataset has already been created and is hosted under


# Store on Deep Lake
the GenAI360 organization on the Activeloop hub. If you prefer not to use OpenAI APIs for generating embeddings, you can test the remaining codes using these publicly accessible datasets. Just substitute the `dataset_path` variable with the following: `hub://genai360/tsla_q3`. </aside>


# Activate DeepMemory
The deep memory feature from Activeloop enhances the retriever's accuracy. This improvement allows the model to access higher-quality data, leading to more detailed and informative responses. In earlier lessons, we already covered the basics of deep memory, so we will not dive into more details. The process begins by fetching chunks of data from the cloud and using GPT-3.5 to create specific questions for each chunk. These generated questions are then utilized in the deep memory training procedure to enhance the embedding quality. In our experience, this approach led to a 25% enhancement in performance. <aside> 💡 Activeloop recommends using a dataset containing a minimum of 100 chunks, ensuring sufficient context for the model to enhance the embedding space effectively. So, the codes in this section are based on three PDF documents. For the complete code and execution steps to process three documents instead of one, please refer to the accompanying notebook. The processed dataset is available in the cloud on the GenAI360 organization. You can access using the following key: `hub://genai360/tesla_quarterly_2023`. </aside> The initial phase involves loading the pre-existing dataset and reading the text of each chunk along with its corresponding ID. ```python from llama_index.vector_stores import DeepLakeVectorStore # TODO: use your organization id here. (by default, org id is your username) my_activeloop_org_id = "<YOUR-ACTIVELOOP-ORG-ID>" my_activeloop_dataset_name = "LlamaIndex_tsla_q3" dataset_path = f"hub://{my_activeloop_org_id}/{my_activeloop_dataset_name}" db = DeepLakeVectorStore( dataset_path=dataset_path, runtime={"tensor_db": True}, read_only=True ) # fetch dataset docs and ids if they exist (optional you can also ingest) docs = db.vectorstore.dataset.text.data(fetch_chunks=True, aslist=True)['value'] ids = db.vectorstore.dataset.id.data(fetch_chunks=True, aslist=True)['value'] print(len(docs)) ``` ``` Deep Lake Dataset in hub://genai360/tesla_quarterly_2023 already exists, loading from the storage 127 ``` The following code segment outlines a function designed to use GPT-3.5 for generating questions corresponding to each data chunk. This involves crafting a specialized tool tailored for the OpenAI API. Primarily, the code configures suitable prompts for API requests to produce the questions and compiles them with their associated chunk IDs into a list. ```python import json import random from tqdm import tqdm from openai import OpenAI client = OpenAI() # Set the function JSON Schema for openai function calling feature tools = [ { "type": "function", "function": { "name": "create_question_from_text", "parameters": { "type": "object", "properties": { "question": { "type": "string", "description": "Question created from the given text", }, }, "required": ["question"], }, "description": "Create question from a given text.", }, } ] def generate_question(tools, text): try: response = client.chat.completions.create( model="gpt-3.5-turbo", tools=tools, tool_choice={ "type": "function", "function": {"name": "create_question_from_text"}, }, messages=[ {"role": "system", "content": "You are a world class expert for generating questions based on provided context. You make sure the question can be answered by the text."}, { "role": "user", "content": text, }, ], ) json_response = response.choices[0].message.tool_calls[0].function.arguments parsed_response = json.loads(json_response) question_string = parsed_response["question"] return question_string except: question_string = "No question generated" return question_string def generate_queries(docs: list[str], ids: list[str], n: int): questions = [] relevances = [] pbar = tqdm(total=n) while len(questions) < n: # 1. randomly draw a piece of text and relevance id r = random.randint(0, len(docs)-1) text, label = docs[r], ids[r] # 2. generate queries and assign and relevance id generated_qs = [generate_question(tools, text)] if generated_qs == ["No question generated"]: continue questions.extend(generated_qs) relevances.extend([[(label, 1)] for _ in generated_qs]) pbar.update(len(generated_qs)) return questions[:n], relevances[:n] questions, relevances = generate_queries(docs, ids, n=20) ``` ``` 100%|██████████| 20/20 [00:19<00:00, 1.02it/s] ``` Now, we can use the questions and the reference ids to activate the deep memory using the `.deep_memory.train()` method to improve the embedding representations. You can see the status of the training process using the `.info` method. ```python from langchain.embeddings.openai import OpenAIEmbeddings embeddings = OpenAIEmbeddings() job_id = db.vectorstore.deep_memory.train( queries=questions, relevance=relevances, embedding_function=embeddings.embed_documents, ) print( db.vectorstore.dataset.embedding.info )


# Activate DeepMemory
``` ``` Starting DeepMemory training job Your Deep Lake dataset has been successfully created! Preparing training data for deepmemory: Creating 20 embeddings in 1 batches of size 20:: 100%|██████████| 1/1 [00:03<00:00, 3.23s/it] DeepMemory training job started. Job ID: 6581e3056a1162b64061a9a4 {'deepmemory': {'6581e3056a1162b64061a9a4_0.npy': {'base_recall@10': 0.25, 'deep_memory_version': '0.2', 'delta': 0.25, 'job_id': '6581e3056a1162b64061a9a4_0', 'model_type': 'npy', 'recall@10': 0.5}, 'model.npy': {'base_recall@10': 0.25, 'deep_memory_version': '0.2', 'delta': 0.25, 'job_id': '6581e3056a1162b64061a9a4_0', 'model_type': 'npy', 'recall@10': 0.5}}} ``` The dataset is now prepared and compatible with the deep memory feature. It's crucial to note that the deep memory option must be actively set to true when using the dataset for inference.


# Chatbot In Action
In this section, we will use the created dataset as the retrieval object, providing the necessary context for the `GPT-3.5-turbo` model (the default choice for LlamaIndex) to answer the questions. Keep in mind that the inference outcomes presented in the subsequent section are derived from processing three PDF files, which are consistent with the sample codes provided in the notebook. To access the processed dataset containing all the PDF documents, use `hub://genai360/tesla_quarterly_2023` as the dataset path in the code below. The `DeepLakeVectorStore` class also handles loading a dataset from the hub. The key distinction in the code below, compared to the previous sections, lies in the use of the `.from_vector_store()` method. This method creates indexes directly from the database rather than variables. ```python from llama_index.vector_stores import DeepLakeVectorStore from llama_index.storage.storage_context import StorageContext from llama_index import VectorStoreIndex vector_store = DeepLakeVectorStore(dataset_path=dataset_path, overwrite=False) storage_context = StorageContext.from_defaults(vector_store=vector_store) index = VectorStoreIndex.from_vector_store( vector_store, storage_context=storage_context ) ``` We can now use the `.as_query_engine()` method of the index variables to establish a query engine. This will allow us to ask questions from various data sources. Notice the `vector_store_kwargs` argument, which activates the `deep_memory` feature by setting it to True. This step is essential for enabling the feature on the retriever. The `.query()` method takes a prompt and searches for the most relevant data points within the database to construct an answer. ```python query_engine = index.as_query_engine(vector_store_kwargs={"deep_memory": True}) response = query_engine.query( "What are the trends in vehicle deliveries?", ) ``` ``` The trends in vehicle deliveries on the Quarter 3 report show an increasing trend over the quarters. ``` ![Screenshot referenced graph.](Multimodal%20Financial%20Document%20Analysis%20and%20Recall;%20974bbe2cce5d4402a7ac0bec9022a7f3/Untitled.png) Screenshot referenced graph. As observed, the chatbot effectively utilized the data from the descriptions of the graphs we generated in the report. On the right, there's a screenshot of the bar chart which the chatbot referenced to generate its response. Additionally, we conducted an experiment where we compiled the same dataset but excluded the graph descriptions. This dataset can be accessed via `hub://genai360/tesla_quarterly_2023-nograph` path. The purpose was to determine whether including the descriptions aids the chatbot's performance. ``` In quarter 3, there was a decrease in Model S/X deliveries compared to the previous quarter, with a 14% decline. However, there was an increase in Model 3/Y deliveries, with a 29% growth. Overall, total deliveries in quarter 3 increased by 27% compared to the previous quarter. ``` You'll observe that the chatbot points to incorrect text segments. Despite the answer being contextually similar, it doesn't provide the correct answer. The graph shows an upward trend, a detail that might not have been mentioned in the report's text.


# Conclusion
In this lesson, we explored the steps of developing a chatbot capable of utilizing PDF files as a knowledge base to answer questions. Additionally, we employed the vision capability of GPT-4V to identify and describe graphs from each page. Describing the charts and their illustrated trends improves the chatbot's accuracy in answering and providing additional context. --- >> [Notebook](https://colab.research.google.com/drive/1JHevaKUazdjSptTMzjFR9BCQ2RTBq73o?usp=sharing). >> Preprocessed Text/Label: [categorized_elements.pkl](Multimodal%20Financial%20Document%20Analysis%20and%20Recall;%20974bbe2cce5d4402a7ac0bec9022a7f3/categorized_elements.pkl) >> Preprocessed Graphs: [graphs_description.pkl](Multimodal%20Financial%20Document%20Analysis%20and%20Recall;%20974bbe2cce5d4402a7ac0bec9022a7f3/graphs_description.pkl)


Production-Ready RAG Solutions with LlamaIndex

# Introduction
[LlamaIndex](https://www.llamaindex.ai/) is a framework for developing data-driven LLM applications, offering data ingestion, indexing, and querying tools. It plays a key role in incorporating additional data sources into LLMs, which is essential for RAG systems. In this lesson, we will explore how RAG-based applications can be improved by focusing on building production-ready code with a focus on data considerations. We'll discuss how to improve RAG retrieval performance through clear data definition and state management. Additionally, we will cover how to use LLMs to extract metadata to boost retrieval efficiency. The lesson also covers the concerns about how embedding references and summaries in text chunks can significantly improve retrieval performance and the capability of LLMs to infer metadata filters for structured retrieval. We'll also discuss fine-tuning embedding representations in LLM applications to achieve optimal retrieval performance.


# Challenges of RAG Systems
Retrieval-Augmented Generation (RAG) applications present unique challenges crucial for their successful implementation. In this section, we explore the dynamic management of data, ensuring varied and effective data representation and adhering to regulatory standards, highlighting the intricate balance required in RAG systems. ### **Document Updates and Stored Vectors** A significant challenge in RAG systems is keeping up with changes in documents and ensuring these updates are accurately reflected in the stored vectors. When documents are modified, added, or removed, the corresponding vectors need to be updated to maintain the accuracy and relevance of the retrieval system. Not addressing this can lead to outdated or irrelevant data retrieval, negatively impacting the system's effectiveness. Implementing dynamic updating mechanisms for vectors can greatly improve the system's ability to provide relevant and current information, enhancing its overall performance. ### Chunking **and Data Distribution** The granularity level is vital in achieving accurate retrieval results. If the chunk size is too large, important details might be missed; if it's too small, the system might get bogged down in details and miss the bigger picture. This setting requires testing and refinement tailored to the specific characteristics of the data and its application. ### **Diverse Representations in Latent Space** The presence of different representations in the same latent space can be challenging (e.g., for representing a paragraph of text versus representing a table or an image). These diverse representations can cause conflicts or inconsistencies when retrieving information, leading to less accurate results. ### **Compliance** Compliance is another critical issue, especially when implementing RAG systems in regulated industries or environments with strict data handling requirements, particularly for private documents with limited access. Non-compliance can lead to legal issues (think about a finance application), data breaches, or misuse of sensitive information. Ensuring the system adheres to relevant laws, regulations, and ethical guidelines prevents these risks. It increases the system's reliability and trustworthiness, vital for its successful deployment.