Allowing us to retrieve relevant information and feed this information into the generation model as a secondary source of information .  We will demonstrate option 2.  Building a Knowledge Base With option 2 the retrieval of relevant information requires an external 'Knowledge Base', a place where we can store and use to efficiently retrieve information.  We can think of this as the external long-term memory of our LLM.  We will need to retrieve information that is semantically related to our queries, to do this we need to use 'dense vector embeddings'.  These can be thought of as numerical representations of the meaning behind our sentences.  To create these dense vectors we use the text-embedding-ada-002 model.  We have already authenticated our OpenAI connection, to create an embedding we just do:embed_model = 'text-embedding-ada-002'res = openai.Embedding.create( input=[ 'Sample document text goes here', 'there will be several phrases in each batch' ], engine=embed_model)In the response res we will find a JSON-like object containing our new embeddings within the 'data' field.res.keys()Inside 'data' we will find two records, one for each of the two sentences we just embedded.  Each vector embedding contains 1536 dimensions (the output dimensionality of the text-embedding-ada-002 model.len(res['data'])len(res['data'][0]['embedding']), len(res['data'][1]['embedding'])We will apply this same embedding logic to a dataset containing information relevant to our query (and many other queries on the topics of ML and AI).  Data Preparation The dataset we will be using is the jamescalam/youtube-transcriptions from Hugging Face Datasets.  It contains transcribed audio from several ML and tech YouTube channels.  We download it with:from datasets import load_datasetdata = load_dataset('jamescalam/youtube-transcriptions', split='train')datadata[0]The dataset contains many small snippets of text data.  We will need to merge many snippets from each video to create more substantial chunks of text that contain more information.from tqdm.auto import tqdmnew_data = []window = 20 # number of sentences to combinestride = 4 # number of sentences to 'stride' over, used to create overlapfor i in tqdm(range(0, len(data), stride)): i_end = min(len(data)-1, i+window) if data[i]['title'] != data[i_end]['title']: # in this case we skip this entry as we have start/end of two videos continue text = ' '.join(data[i:i_end]['text']) # create the new merged dataset new_data.append({ 'start': data[i]['start'], 'end': data[i_end]['end'], 'title': data[i]['title'], 'text': text, 'id': data[i]['id'], 'url': data[i]['url'], 'published': data[i]['published'], 'channel_id': data[i]['channel_id'] })new_data[0]Now we need a place to store these embeddings and enable a efficient vector search through them all


Retrieval Augmented Generative Question Answering with Pinecone

To do that we use Pinecone, we can get a free API key and enter it below where we will initialize our connection to Pinecone and create a new index.import pineconeindex_name = 'openai-youtube-transcriptions'# initialize connection to pinecone (get API key at app.pinecone.io)pinecone.init( api_key='PINECONE_API_KEY', environment='us-east1-gcp' # may be different, check at app.pinecone.io)# check if index already exists (it shouldn't if this is first time)if index_name not in pinecone.list_indexes(): # if does not exist, create index pinecone.create_index( index_name, dimension=len(res['data'][0]['embedding']), metric='cosine', metadata_config={'indexed': ['channel_id', 'published']} )# connect to indexindex = pinecone.Index(index_name)# view index statsindex.describe_index_stats()We can see the index is currently empty with a total_vector_count of 0 .  We can begin populating it with OpenAI text-embedding-ada-002 built embeddings like so:from tqdm.auto import tqdmfrom time import sleepbatch_size = 100 # how many embeddings we create and insert at oncefor i in tqdm(range(0, len(new_data), batch_size)): # find end of batch i_end = min(len(new_data), i+batch_size) meta_batch = new_data[i:i_end] # get ids ids_batch = [x['id'] for x in meta_batch] # get texts to encode texts = [x['text'] for x in meta_batch] # create embeddings (try-except added to avoid RateLimitError) done = False while not done: try: res = openai.Embedding.create(input=texts, engine=embed_model) done = True except: sleep(5) embeds = [record['embedding'] for record in res['data']] # cleanup metadata meta_batch = [{ 'start': x['start'], 'end': x['end'], 'title': x['title'], 'text': x['text'], 'url': x['url'], 'published': x['published'], 'channel_id': x['channel_id'] } for x in meta_batch] to_upsert = list(zip(ids_batch, embeds, meta_batch)) # upsert to Pinecone index.upsert(vectors=to_upsert)Now we search, for this we need to create a query vector xq:res = openai.Embedding.create( input=[query], engine=embed_model)# retrieve from Pineconexq = res['data'][0]['embedding']# get relevant contexts (including the questions)res = index.query(xq, top_k=2, include_metadata=True)reslimit = 3750def retrieve(query): res = openai.Embedding.create( input=[query], engine=embed_model ) # retrieve from Pinecone xq = res['data'][0]['embedding'] # get relevant contexts res = index.query(xq, top_k=3, include_metadata=True) contexts = [ x['metadata']['text'] for x in res['matches'] ] # build our prompt with the retrieved contexts included prompt_start = ( 'Answer the question based on the context below.\n\n'+ 'Context:\n' ) prompt_end = ( f'\n\nQuestion: {query}\nAnswer:' ) # append contexts until hitting limit for i in range(1, len(contexts)): if len('\n\n---\n\n'.join(contexts[:i])) >= limit: prompt = ( prompt_start + '\n\n---\n\n'.join(contexts[:i-1]) + prompt_end ) break elif i == len(contexts)-1: prompt = ( prompt_start + '\n\n---\n\n'.join(contexts) + prompt_end ) return prompt# first we retrieve relevant items from Pineconequery_with_contexts = retrieve(query)query_with_contexts# then we complete the context-infused querycomplete(query_with_contexts)And we get a pretty great answer straight away, specifying to use multiple-rankings loss (also called multiple negatives ranking loss).


Data Preparation

The dataset we will be using is the jamescalam/youtube-transcriptions from Hugging Face Datasets. It contains transcribed audio from several ML and tech YouTube channels. We download it with:


Building a Knowledge Base

With option 2 the retrieval of relevant information requires an external 'Knowledge Base', a place where we can store and use to efficiently retrieve information. We can think of this as the external long-term memory of our LLM.We will need to retrieve information that is semantically related to our queries, to do this we need to use 'dense vector embeddings'. These can be thought of as numerical representations of the meaning behind our sentences.To create these dense vectors we use the text-embedding-ada-002 model.We have already authenticated our OpenAI connection, to create an embedding we just do:


Fixing LLMs that Hallucinate

In this notebook we will learn how to query relevant contexts to our queries from Pinecone, and pass these to a generative OpenAI model to generate an answer backed by real data sources.A common problem with using GPT-3 to factually answer questions is that GPT-3 can sometimes make things up. The GPT models have a broad range of general knowledge, but this does not necessarily apply to more specific information. For that we use the Pinecone vector database as our 'external knowledge base' — like long-term memory for GPT-3.Required installs for this notebook are:


Visualizing embeddings in Weights and Biases

Open in GithubWe will upload the data to Weights & Biases and use an Embedding Projector to visualize the embeddings using common dimension reduction algorithms like PCA, UMAP, and t-SNE. The dataset is created in the Get_embeddings_from_dataset Notebook.What is Weights & Biases? Weights & Biases is a machine learning platform used by OpenAI and other ML teams to build better models faster. They use it to quickly track experiments, evaluate model performance, reproduce models, visualize results, and share findings with colleagues.1. Log the data to W&B We create a W&B Table with the original data and the embeddings. Each review is a new row and the 1536 embedding floats are given their own column named emb_{i}.import pandas as pdfrom sklearn.manifold import TSNEimport numpy as npfrom ast import literal_eval# Load the embeddingsdatafile_path = 'data/fine_food_reviews_with_embeddings_1k.csv'df = pd.read_csv(datafile_path)# Convert to a list of lists of floatsmatrix = np.array(df.embedding.apply(literal_eval).to_list())import wandboriginal_cols = df.columns[1:-1].tolist()embedding_cols = ['emb_'+str(idx) for idx in range(len(matrix[0]))]table_cols = original_cols + embedding_colswith wandb.init(project='openai_embeddings'): table = wandb.Table(columns=table_cols) for i, row in enumerate(df.to_dict(orient='records')): original_data = [row[col_name] for col_name in original_cols] embedding_data = matrix[i].tolist() table.add_data(*(original_data + embedding_data)) wandb.log({'openai_embedding_table': table})


What is Weights & Biases?

Weights & Biases is a machine learning platform used by OpenAI and other ML teams to build better models faster. They use it to quickly track experiments, evaluate model performance, reproduce models, visualize results, and share findings with colleagues.


2. Render as 2D Projection

After navigating to the W&B run link, we click the ⚙ icon in the top right of the Table and change 'Render As:' to 'Combined 2D Projection'.Example: http://wandb.me/openai_embeddings


1. Log the data to W&B

We create a W&B Table with the original data and the embeddings. Each review is a new row and the 1536 embedding floats are given their own column named emb_{i}.


Use cases for embeddings

Open in GithubThe OpenAI API embeddings endpoint can be used to measure relatedness or similarity between pieces of text. By leveraging GPT-3's understanding of text, these embeddings achieved state-of-the-art results on benchmarks in unsupervised learning and transfer learning settings. Embeddings can be used for semantic search, recommendations, cluster analysis, near-duplicate detection, and more. For more information, read OpenAI's blog post announcements: Introducing Text and Code Embeddings (Jan 2022) New and Improved Embedding Model (Dec 2022) For comparison with other embedding models, see Massive Text Embedding Benchmark (MTEB) Leaderboard Semantic search Embeddings can be used for search either by themselves or as a feature in a larger system. The simplest way to use embeddings for search is as follows: Before the search (precompute): Split your text corpus into chunks smaller than the token limit (8,191 tokens for text-embedding-ada-002) Embed each chunk of text Store those embeddings in your own database or in a vector search provider like Pinecone, Weaviate or Qdrant At the time of the search (live compute): Embed the search query Find the closest embeddings in your database Return the top results An example of how to use embeddings for search is shown in Semantic_text_search_using_embeddings.ipynb. In more advanced search systems, the cosine similarity of embeddings can be used as one feature among many in ranking search results. Question answering The best way to get reliably honest answers from GPT-3 is to give it source documents in which it can locate correct answers. Using the semantic search procedure above, you can cheaply search through a corpus of documents for relevant information and then give that information to GPT-3 via the prompt to answer a question. We demonstrate this in Question_answering_using_embeddings.ipynb. Recommendations Recommendations are quite similar to search, except that instead of a free-form text query, the inputs are items in a set. An example of how to use embeddings for recommendations is shown in Recommendation_using_embeddings.ipynb. Similar to search, these cosine similarity scores can either be used on their own to rank items or as features in larger ranking algorithms. Customizing Embeddings Although OpenAI's embedding model weights cannot be fine-tuned, you can nevertheless use training data to customize embeddings to your application. In Customizing_embeddings.ipynb, we provide an example method for customizing your embeddings using training data. The idea of the method is to train a custom matrix to multiply embedding vectors by in order to get new customized embeddings. With good training data, this custom matrix will help emphasize the features relevant to your training labels. You can equivalently consider the matrix multiplication as (a) a modification of the embeddings or (b) a modification of the distance function used to measure the distances between embeddings.


Semantic search

Embeddings can be used for search either by themselves or as a feature in a larger system.The simplest way to use embeddings for search is as follows: Before the search (precompute): Split your text corpus into chunks smaller than the token limit (8,191 tokens for text-embedding-ada-002) Embed each chunk of text Store those embeddings in your own database or in a vector search provider like Pinecone, Weaviate or Qdrant At the time of the search (live compute): Embed the search query Find the closest embeddings in your database Return the top results An example of how to use embeddings for search is shown in Semantic_text_search_using_embeddings.ipynb.In more advanced search systems, the cosine similarity of embeddings can be used as one feature among many in ranking search results.


Question answering

The best way to get reliably honest answers from GPT-3 is to give it source documents in which it can locate correct answers. Using the semantic search procedure above, you can cheaply search through a corpus of documents for relevant information and then give that information to GPT-3 via the prompt to answer a question. We demonstrate this in Question_answering_using_embeddings.ipynb.


Customizing Embeddings

Although OpenAI's embedding model weights cannot be fine-tuned, you can nevertheless use training data to customize embeddings to your application.In Customizing_embeddings.ipynb, we provide an example method for customizing your embeddings using training data. The idea of the method is to train a custom matrix to multiply embedding vectors by in order to get new customized embeddings. With good training data, this custom matrix will help emphasize the features relevant to your training labels. You can equivalently consider the matrix multiplication as (a) a modification of the embeddings or (b) a modification of the distance function used to measure the distances between embeddings.


Recommendations

Recommendations are quite similar to search, except that instead of a free-form text query, the inputs are items in a set.An example of how to use embeddings for recommendations is shown in Recommendation_using_embeddings.ipynb.Similar to search, these cosine similarity scores can either be used on their own to rank items or as features in larger ranking algorithms.


Embedding texts that are longer than the model's maximum context length

Open in GithubOpenAI's embedding models cannot embed text that exceeds a maximum length.  The maximum length varies by model, and is measured by tokens, not string length.  If you are unfamiliar with tokenization, check out How to count tokens with tiktoken.  This notebook shows how to handle texts that are longer than a model's maximum context length.  We'll demonstrate using embeddings from text-embedding-ada-002, but the same ideas can be applied to other models and tasks.  To learn more about embeddings, check out the OpenAI Embeddings Guide.1.  Model context length First, we select the model and define a function to get embeddings from the API.import openaifrom tenacity import retry, wait_random_exponential, stop_after_attempt, retry_if_not_exception_typeEMBEDDING_MODEL = 'text-embedding-ada-002'EMBEDDING_CTX_LENGTH = 8191EMBEDDING_ENCODING = 'cl100k_base'# let's make sure to not retry on an invalid request, because that is what we want to demonstrate@retry(wait=wait_random_exponential(min=1, max=20), stop=stop_after_attempt(6), retry=retry_if_not_exception_type(openai.InvalidRequestError))def get_embedding(text_or_tokens, model=EMBEDDING_MODEL): return openai.Embedding.create(input=text_or_tokens, model=model)['data'][0]['embedding']The text-embedding-ada-002 model has a context length of 8191 tokens with the cl100k_base encoding, and we can see that going over that limit causes an error.long_text = 'AGI ' * 5000try: get_embedding(long_text)except openai.InvalidRequestError as e: print(e)Clearly we want to avoid these errors, particularly when handling programmatically with a large number of embeddings.  Yet, we still might be faced with texts that are longer than the maximum context length.  Below we describe and provide recipes for the main approaches to handling these longer texts: (1) simply truncating the text to the maximum allowed length, and (2) chunking the text and embedding each chunk individually.1.  Truncating the input text The simplest solution is to truncate the input text to the maximum allowed length.  Because the context length is measured in tokens, we have to first tokenize the text before truncating it.  The API accepts inputs both in the form of text or tokens, so as long as you are careful that you are using the appropriate encoding, there is no need to convert the tokens back into string form.  Below is an example of such a truncation function.import tiktokendef truncate_text_tokens(text, encoding_name=EMBEDDING_ENCODING, max_tokens=EMBEDDING_CTX_LENGTH): '''Truncate a string to have `max_tokens` according to the given encoding.''' encoding = tiktoken.get_encoding(encoding_name) return encoding.encode(text)[:max_tokens]Our example from before now works without error.truncated = truncate_text_tokens(long_text)len(get_embedding(truncated))2.  Chunking the input text Though truncation works, discarding potentially relevant text is a clear drawback.  Another approach is to divide the input text into chunks and then embed each chunk individually.  Then, we can either use the chunk embeddings separately, or combine them in some way, such as averaging (weighted by the size of each chunk).  We will take a function from Python's own cookbook that breaks up a sequence into chunks.from itertools import islicedef batched(iterable, n): '''Batch data into tuples of length n


1. Model context length

First, we select the model and define a function to get embeddings from the API.


1. Truncating the input text

The simplest solution is to truncate the input text to the maximum allowed length. Because the context length is measured in tokens, we have to first tokenize the text before truncating it. The API accepts inputs both in the form of text or tokens, so as long as you are careful that you are using the appropriate encoding, there is no need to convert the tokens back into string form. Below is an example of such a truncation function.


The last batch may be shorter.''' # batched('ABCDEFG', 3) --> ABC DEF G if n < 1: raise ValueError('n must be at least one') it = iter(iterable) while (batch := tuple(islice(it, n))): yield batchNow we define a function that encodes a string into tokens and then breaks it up into chunks.def chunked_tokens(text, encoding_name, chunk_length): encoding = tiktoken.get_encoding(encoding_name) tokens = encoding.encode(text) chunks_iterator = batched(tokens, chunk_length) yield from chunks_iteratorFinally, we can write a function that safely handles embedding requests, even when the input text is longer than the maximum context length, by chunking the input tokens and embedding each chunk individually .  The average flag can be set to True to return the weighted average of the chunk embeddings, or False to simply return the unmodified list of chunk embeddings.import numpy as npdef len_safe_get_embedding(text, model=EMBEDDING_MODEL, max_tokens=EMBEDDING_CTX_LENGTH, encoding_name=EMBEDDING_ENCODING, average=True): chunk_embeddings = [] chunk_lens = [] for chunk in chunked_tokens(text, encoding_name=encoding_name, chunk_length=max_tokens): chunk_embeddings.append(get_embedding(chunk, model=model)) chunk_lens.append(len(chunk)) if average: chunk_embeddings = np.average(chunk_embeddings, axis=0, weights=chunk_lens) chunk_embeddings = chunk_embeddings / np.linalg.norm(chunk_embeddings) # normalizes length to 1 chunk_embeddings = chunk_embeddings.tolist() return chunk_embeddingsOnce again, we can now handle long input texts.average_embedding_vector = len_safe_get_embedding(long_text, average=True)chunks_embedding_vectors = len_safe_get_embedding(long_text, average=False)print(f'Setting average=True gives us a single {len(average_embedding_vector)}-dimensional embedding vector for our long text.')print(f'Setting average=False gives us {len(chunks_embedding_vectors)} embedding vectors, one for each of the chunks.')In some cases, it may make sense to split chunks on paragraph boundaries or sentence boundaries to help preserve the meaning of the text.


2. Chunking the input text

Though truncation works, discarding potentially relevant text is a clear drawback. Another approach is to divide the input text into chunks and then embed each chunk individually. Then, we can either use the chunk embeddings separately, or combine them in some way, such as averaging (weighted by the size of each chunk).We will take a function from Python's own cookbook that breaks up a sequence into chunks.


Azure completions example

Open in GithubIn this example we'll try to go over all operations needed to get completions working using the Azure endpoints. This example focuses on completions but also touches on some other operations that are also available using the API. This example is meant to be a quick way of showing simple operations and is not meant as a tutorial.import osimport openaiSetup For the following sections to work properly we first have to setup some things. Let's start with the api_base and api_version. To find your api_base go to https://portal.azure.com, find your resource and then under 'Resource Management' -> 'Keys and Endpoints' look for the 'Endpoint' value.openai.api_version = '2023-05-15'openai.api_base = '' # Please add your endpoint hereWe next have to setup the api_type and api_key. We can either get the key from the portal or we can get it through Microsoft Active Directory Authentication. Depending on this the api_type is either azure or azure_ad.Setup: Portal Let's first look at getting the key from the portal. Go to https://portal.azure.com, find your resource and then under 'Resource Management' -> 'Keys and Endpoints' look for one of the 'Keys' values.openai.api_type = 'azure'openai.api_key = os.environ['OPENAI_API_KEY'] Note: In this example, we configured the library to use the Azure API by setting the variables in code. For development, consider setting the environment variables instead: OPENAI_API_BASE OPENAI_API_KEY OPENAI_API_TYPE OPENAI_API_VERSION(Optional) Setup: Microsoft Active Directory Authentication Let's now see how we can get a key via Microsoft Active Directory Authentication. Uncomment the following code if you want to use Active Directory Authentication instead of keys from the portal.# from azure.identity import DefaultAzureCredential# default_credential = DefaultAzureCredential()# token = default_credential.get_token('https://cognitiveservices.azure.com/.default')# openai.api_type = 'azure_ad'# openai.api_key = token.tokenA token is valid for a period of time, after which it will expire. To ensure a valid token is sent with every request, you can refresh an expiring token by hooking into requests.auth:import typingimport timeimport requestsif typing.TYPE_CHECKING: from azure.core.credentials import TokenCredentialclass TokenRefresh(requests.auth.AuthBase): def __init__(self, credential: 'TokenCredential', scopes: typing.List[str]) -> None: self.credential = credential self.scopes = scopes self.cached_token: typing.Optional[str] = None def __call__(self, req): if not self.cached_token or self.cached_token.expires_on - time.time() < 300: self.cached_token = self.credential.get_token(*self.scopes) req.headers['Authorization'] = f'Bearer {self.cached_token.token}' return reqsession = requests.Session()session.auth = TokenRefresh(default_credential, ['https://cognitiveservices.azure.com/.default'])openai.requestssession = sessionDeployments In this section we are going to create a deployment using the text-davinci-002 model that we can then use to create completions.Deployments: Create manually Create a new deployment by going to your Resource in your portal under 'Resource Management' -> 'Model deployments'. Select text-davinci-002 as the model.deployment_id = '' # Fill in the deployment id from the portal hereCompletions Now let's send a sample completion to the deployment.prompt = 'The food was delicious and the waiter'completion = openai.Completion.create(deployment_id=deployment_id, prompt=prompt, stop='.', temperature=0) print(f'{prompt}{completion['choices'][0]['text']}.')


Setup

For the following sections to work properly we first have to setup some things. Let's start with the api_base and api_version. To find your api_base go to https://portal.azure.com, find your resource and then under 'Resource Management' -> 'Keys and Endpoints' look for the 'Endpoint' value.


(Optional) Setup: Microsoft Active Directory Authentication

Let's now see how we can get a key via Microsoft Active Directory Authentication. Uncomment the following code if you want to use Active Directory Authentication instead of keys from the portal.


Setup: Portal

Let's first look at getting the key from the portal. Go to https://portal.azure.com, find your resource and then under 'Resource Management' -> 'Keys and Endpoints' look for one of the 'Keys' values.


Completions

Now let's send a sample completion to the deployment.


Deployments: Create manually

Create a new deployment by going to your Resource in your portal under 'Resource Management' -> 'Model deployments'. Select text-davinci-002 as the model.


Deployments

In this section we are going to create a deployment using the text-davinci-002 model that we can then use to create completions.


Unit test writing using a multi-step prompt

The code example illustrates a few embellishments on the chained, multi-step prompt: Conditional branching (e.g., asking for elaboration only if the first plan is too short) The choice of different models for different steps A check that re-runs the function if the output is unsatisfactory (e.g., if the output code cannot be parsed by Python's ast module) Streaming output so that you can start reading the output before it's fully generated (handy for long, multi-step outputs) # imports needed to run the code in this notebookimport ast # used for detecting whether generated Python code is validimport openai # used for calling the OpenAI APIcolor_prefix_by_role = { 'system': '\033[0m', # gray 'user': '\033[0m', # gray 'assistant': '\033[92m', # green}def print_messages(messages, color_prefix_by_role=color_prefix_by_role) -> None: '''Prints messages sent to or from GPT.''' for message in messages: role = message['role'] color_prefix = color_prefix_by_role[role] content = message['content'] print(f'{color_prefix}\n[{role}]\n{content}')def print_message_delta(delta, color_prefix_by_role=color_prefix_by_role) -> None: '''Prints a chunk of messages streamed back from GPT.''' if 'role' in delta: role = delta['role'] color_prefix = color_prefix_by_role[role] print(f'{color_prefix}\n[{role}]\n', end='') elif 'content' in delta: content = delta['content'] print(content, end='') else: pass# example of a function that uses a multi-step prompt to write unit testsdef unit_tests_from_function( function_to_test: str, # Python function to test, as a string unit_test_package: str = 'pytest', # unit testing package; use the name as it appears in the import statement approx_min_cases_to_cover: int = 7, # minimum number of test case categories to cover (approximate) print_text: bool = False, # optionally prints text; helpful for understanding the function & debugging explain_model: str = 'gpt-3.5-turbo', # model used to generate text plans in step 1 plan_model: str = 'gpt-3.5-turbo', # model used to generate text plans in steps 2 and 2b execute_model: str = 'gpt-3.5-turbo', # model used to generate code in step 3 temperature: float = 0.4, # temperature = 0 can sometimes get stuck in repetitive loops, so we use 0.4 reruns_if_fail: int = 1, # if the output code cannot be parsed, this will re-run the function up to N times) -> str: '''Returns a unit test for a given Python function, using a 3-step GPT prompt.''' # Step 1: Generate an explanation of the function # create a markdown-formatted message that asks GPT to explain the function, formatted as a bullet list explain_system_message = { 'role': 'system', 'content': 'You are a world-class Python developer with an eagle eye for unintended bugs and edge cases .  You carefully explain code with great detail and accuracy.  You organize your explanations in markdown-formatted, bulleted lists.', } explain_user_message = { 'role': 'user', 'content': f'''Please explain the following Python function.  Review what each element of the function is doing precisely and what the author's intentions may have been


Organize your explanation as a markdown-formatted, bulleted list.```python{function_to_test}```''', } explain_messages = [explain_system_message, explain_user_message] if print_text: print_messages(explain_messages) explanation_response = openai.ChatCompletion.create( model=explain_model, messages=explain_messages, temperature=temperature, stream=True, ) explanation = '' for chunk in explanation_response: delta = chunk['choices'][0]['delta'] if print_text: print_message_delta(delta) if 'content' in delta: explanation += delta['content'] explain_assistant_message = {'role': 'assistant', 'content': explanation} # Step 2: Generate a plan to write a unit test # Asks GPT to plan out cases the units tests should cover, formatted as a bullet list plan_user_message = { 'role': 'user', 'content': f'''A good unit test suite should aim to:- Test the function's behavior for a wide range of possible inputs- Test edge cases that the author may not have foreseen- Take advantage of the features of `{unit_test_package}` to make the tests easy to write and maintain- Be easy to read and understand, with clean code and descriptive names- Be deterministic, so that the tests always pass or fail in the same wayTo help unit test the function above, list diverse scenarios that the function should be able to handle (and under each scenario, include a few examples as sub-bullets).''', } plan_messages = [ explain_system_message, explain_user_message, explain_assistant_message, plan_user_message, ] if print_text: print_messages([plan_user_message]) plan_response = openai.ChatCompletion.create( model=plan_model, messages=plan_messages, temperature=temperature, stream=True, ) plan = '' for chunk in plan_response: delta = chunk['choices'][0]['delta'] if print_text: print_message_delta(delta) if 'content' in delta: plan += delta['content'] plan_assistant_message = {'role': 'assistant', 'content': plan} # Step 2b: If the plan is short, ask GPT to elaborate further # this counts top-level bullets (e.g., categories), but not sub-bullets (e.g., test cases) num_bullets = max(plan.count('\n-'), plan.count('\n*')) elaboration_needed = num_bullets < approx_min_cases_to_cover if elaboration_needed: elaboration_user_message = { 'role': 'user', 'content': f'''In addition to those scenarios above, list a few rare or unexpected edge cases (and as before, under each edge case, include a few examples as sub-bullets).''', } elaboration_messages = [ explain_system_message, explain_user_message, explain_assistant_message, plan_user_message, plan_assistant_message, elaboration_user_message, ] if print_text: print_messages([elaboration_user_message]) elaboration_response = openai.ChatCompletion.create( model=plan_model, messages=elaboration_messages, temperature=temperature, stream=True, ) elaboration = '' for chunk in elaboration_response: delta = chunk['choices'][0]['delta'] if print_text: print_message_delta(delta) if 'content' in delta: elaboration += delta['content'] elaboration_assistant_message = {'role': 'assistant', 'content': elaboration} # Step 3: Generate the unit test # create a markdown-formatted prompt that asks GPT to complete a unit test package_comment = '' if unit_test_package == 'pytest': package_comment = '# below, each test case is represented by a tuple passed to the @pytest.mark.parametrize decorator' execute_system_message = { 'role': 'system', 'content': 'You are a world-class Python developer with an eagle eye for unintended bugs and edge cases .  You write careful, accurate unit tests.  When asked to reply only with code, you write all of your code in a single block.', } execute_user_message = { 'role': 'user', 'content': f'''Using Python and the `{unit_test_package}` package, write a suite of unit tests for the function, following the cases above.  Include helpful comments to explain each line


Open in GithubComplex tasks, such as writing unit tests, can benefit from multi-step prompts.  In contrast to a single prompt, a multi-step prompt generates text from GPT and then feeds that output text back into subsequent prompts.  This can help in cases where you want GPT to reason things out before answering, or brainstorm a plan before executing it.  In this notebook, we use a 3-step prompt to write unit tests in Python using the following steps: Explain: Given a Python function, we ask GPT to explain what the function is doing and why.  Plan: We ask GPT to plan a set of unit tests for the function.  If the plan is too short, we ask GPT to elaborate with more ideas for unit tests.  Execute: Finally, we instruct GPT to write unit tests that cover the planned cases


Reply only with code, formatted as follows:```python# importsimport {unit_test_package} # used for our unit tests{{insert other imports as needed}}# function to test{function_to_test}# unit tests{package_comment}{{insert unit test code here}}```''', } execute_messages = [ execute_system_message, explain_user_message, explain_assistant_message, plan_user_message, plan_assistant_message, ] if elaboration_needed: execute_messages += [elaboration_user_message, elaboration_assistant_message] execute_messages += [execute_user_message] if print_text: print_messages([execute_system_message, execute_user_message]) execute_response = openai.ChatCompletion.create( model=execute_model, messages=execute_messages, temperature=temperature, stream=True, ) execution = '' for chunk in execute_response: delta = chunk['choices'][0]['delta'] if print_text: print_message_delta(delta) if 'content' in delta: execution += delta['content'] # check the output for errors code = execution.split('```python')[1].split('```')[0].strip() try: ast.parse(code) except SyntaxError as e: print(f'Syntax error in generated code: {e}') if reruns_if_fail > 0: print('Rerunning...') return unit_tests_from_function( function_to_test=function_to_test, unit_test_package=unit_test_package, approx_min_cases_to_cover=approx_min_cases_to_cover, print_text=print_text, explain_model=explain_model, plan_model=plan_model, execute_model=execute_model, temperature=temperature, reruns_if_fail=reruns_if_fail - 1, # decrement rerun counter when calling again ) # return the unit test as a string return codeexample_function = '''def pig_latin(text): def translate(word): vowels = 'aeiou' if word[0] in vowels: return word + 'way' else: consonants = '' for letter in word: if letter not in vowels: consonants += letter else: break return word[len(consonants):] + consonants + 'ay' words = text.lower().split() translated_words = [translate(word) for word in words] return ' '.join(translated_words)'''unit_tests = unit_tests_from_function( example_function, approx_min_cases_to_cover=10, print_text=True)print(unit_tests)Make sure to check any code before using it, as GPT makes plenty of mistakes (especially on character-based tasks like this one) .  For best results, use the most powerful model (GPT-4, as of May 2023).


Clustering for Transaction Classification

Open in GithubThis notebook covers use cases where your data is unlabelled but has features that can be used to cluster them into meaningful categories. The challenge with clustering is making the features that make those clusters stand out human-readable, and that is where we'll look to use GPT-3 to generate meaningful cluster descriptions for us. We can then use these to apply labels to a previously unlabelled dataset. To feed the model we use embeddings created using the approach displayed in the notebook Multiclass classification for transactions Notebook, applied to the full 359 transactions in the dataset to give us a bigger pool for learning


Setup

# importsimport openaiimport pandas as pdimport numpy as npfrom sklearn.cluster import KMeansfrom sklearn.manifold import TSNEimport matplotlibimport matplotlib.pyplot as pltimport osfrom ast import literal_evalopenai.api_key = os.getenv('OPENAI_API_KEY')COMPLETIONS_MODEL = 'text-davinci-003'# This path leads to a file with data and precomputed embeddingsembedding_path = 'data/library_transactions_with_embeddings_359.csv'Clustering We'll reuse the approach from the Clustering Notebook, using K-Means to cluster our dataset using the feature embeddings we created previously. We'll then use the Completions endpoint to generate cluster descriptions for us and judge their effectivenessdf = pd.read_csv(embedding_path)embedding_df = pd.read_csv(embedding_path)embedding_df['embedding'] = embedding_df.embedding.apply(literal_eval).apply(np.array)matrix = np.vstack(embedding_df.embedding.values)matrix.shapen_clusters = 5kmeans = KMeans(n_clusters=n_clusters, init='k-means++', random_state=42)kmeans.fit(matrix)labels = kmeans.labels_embedding_df['Cluster'] = labelstsne = TSNE( n_components=2, perplexity=15, random_state=42, init='random', learning_rate=200)vis_dims2 = tsne.fit_transform(matrix)x = [x for x, y in vis_dims2]y = [y for x, y in vis_dims2]for category, color in enumerate(['purple', 'green', 'red', 'blue','yellow']): xs = np.array(x)[embedding_df.Cluster == category] ys = np.array(y)[embedding_df.Cluster == category] plt.scatter(xs, ys, color=color, alpha=0.3) avg_x = xs.mean() avg_y = ys.mean() plt.scatter(avg_x, avg_y, marker='x', color=color, s=100)plt.title('Clusters identified visualized in language 2d using t-SNE')# We'll read 10 transactions per cluster as we're expecting some variationtransactions_per_cluster = 10for i in range(n_clusters): print(f'Cluster {i} Theme:', end=' ') transactions = '\n'.join( embedding_df[embedding_df.Cluster == i] .combined.str.replace('Supplier: ', '') .str.replace('Description: ', ': ') .str.replace('Value: ', ': ') .sample(transactions_per_cluster, random_state=42) .values ) response = openai.Completion.create( model=COMPLETIONS_MODEL, # We'll include a prompt to instruct the model what sort of description we're looking for prompt=f'''We want to group these transactions into meaningful clusters so we can target the areas we are spending the most money. What do the following transactions have in common?\n\nTransactions:\n'''\n{transactions}\n'''\n\nTheme:''', temperature=0, max_tokens=100, top_p=1, frequency_penalty=0, presence_penalty=0, ) print(response['choices'][0]['text'].replace('\n', '')) sample_cluster_rows = embedding_df[embedding_df.Cluster == i].sample(transactions_per_cluster, random_state=42) for j in range(transactions_per_cluster): print(sample_cluster_rows.Supplier.values[j], end=', ') print(sample_cluster_rows.Description.values[j], end='\n ') #print(str(sample_cluster_rows.Value).values[j]) print('-' * 100)Conclusion We now have five new clusters that we can use to describe our data. Looking at the visualisation some of our clusters have some overlap and we'll need some tuning to get to the right place, but already we can see that GPT-3 has made some effective inferences. In particular, it picked up that items including legal deposits were related to literature archival, which is true but the model was given no clues on. Very cool, and with some tuning we can create a base set of clusters that we can then use with a multiclass classifier to generalise to other transactional datasets we might use.


Clustering

We'll reuse the approach from the Clustering Notebook, using K-Means to cluster our dataset using the feature embeddings we created previously. We'll then use the Completions endpoint to generate cluster descriptions for us and judge their effectiveness


Conclusion

We now have five new clusters that we can use to describe our data. Looking at the visualisation some of our clusters have some overlap and we'll need some tuning to get to the right place, but already we can see that GPT-3 has made some effective inferences. In particular, it picked up that items including legal deposits were related to literature archival, which is true but the model was given no clues on. Very cool, and with some tuning we can create a base set of clusters that we can then use with a multiclass classifier to generalise to other transactional datasets we might use.


Multiclass Classification for Transactions

Open in GithubFor this notebook we will be looking to classify a public dataset of transactions into a number of categories that we have predefined. These approaches should be replicable to any multiclass classification use case where we are trying to fit transactional data into predefined categories, and by the end of running through this you should have a few approaches for dealing with both labelled and unlabelled datasets. The different approaches we'll be taking in this notebook are: Zero-shot Classification: First we'll do zero shot classification to put transactions in one of five named buckets using only a prompt for guidance Classification with Embeddings: Following this we'll create embeddings on a labelled dataset, and then use a traditional classification model to test their effectiveness at identifying our categories Fine-tuned Classification: Lastly we'll produce a fine-tuned model trained on our labelled dataset to see how this compares to the zero-shot and few-shot classification approaches


Setup

%load_ext autoreload%autoreload %pip install openai 'openai[datalib]' 'openai[embeddings]' transformersimport openaiimport pandas as pdimport numpy as npimport jsonimport osopenai.api_key = os.getenv('OPENAI_API_KEY')COMPLETIONS_MODEL = 'text-davinci-002'Load dataset We're using a public transaction dataset of transactions over £25k for the Library of Scotland.  The dataset has three features that we'll be using: Supplier: The name of the supplier Description: A text description of the transaction Value: The value of the transaction in GBP Source: https://data.nls.uk/data/organisational-data/transactions-over-25k/transactions = pd.read_csv('./data/25000_spend_dataset_current.csv', encoding= 'unicode_escape')len(transactions)transactions.head() .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Date Supplier Description Transaction value (£) 0 21/04/2016 M & J Ballantyne Ltd George IV Bridge Work 35098.0 1 26/04/2016 Private Sale Literary & Archival Items 30000.0 2 30/04/2016 City Of Edinburgh Council Non Domestic Rates 40800.0 3 09/05/2016 Computacenter Uk Kelvin Hall 72835.0 4 09/05/2016 John Graham Construction Ltd Causewayside Refurbishment 64361.0 def request_completion(prompt): completion_response = openai.Completion.create( prompt=prompt, temperature=0, max_tokens=5, top_p=1, frequency_penalty=0, presence_penalty=0, model=COMPLETIONS_MODEL ) return completion_responsedef classify_transaction(transaction,prompt): prompt = prompt.replace('SUPPLIER_NAME',transaction['Supplier']) prompt = prompt.replace('DESCRIPTION_TEXT',transaction['Description']) prompt = prompt.replace('TRANSACTION_VALUE',str(transaction['Transaction value (£)'])) classification = request_completion(prompt)['choices'][0]['text'].replace('\n','') return classification# This function takes your training and validation outputs from the prepare_data function of the Finetuning API, and# confirms that each have the same number of classes.# If they do not have the same number of classes the fine-tune will fail and return an errordef check_finetune_classes(train_file,valid_file): train_classes = set() valid_classes = set() with open(train_file, 'r') as json_file: json_list = list(json_file) print(len(json_list)) for json_str in json_list: result = json.loads(json_str) train_classes.add(result['completion']) #print(f'result: {result['completion']}') #print(isinstance(result, dict)) with open(valid_file, 'r') as json_file: json_list = list(json_file) print(len(json_list)) for json_str in json_list: result = json.loads(json_str) valid_classes.add(result['completion']) #print(f'result: {result['completion']}') #print(isinstance(result, dict)) if len(train_classes) == len(valid_classes): print('All good') else: print('Classes do not match, please prepare data again')Zero-shot Classification We'll first assess the performance of the base models at classifying these transactions using a simple prompt.  We'll provide the model with 5 categories and a catch-all of 'Could not classify' for ones that it cannot place.zero_shot_prompt = '''You are a data expert working for the National Library of Scotland.  You are analysing all transactions over £25,000 in value and classifying them into one of five categories.The five categories are Building Improvement, Literature & Archive, Utility Bills, Professional Services and Software/IT.If you can't tell what it is, say Could not classify Transaction: Supplier: SUPPLIER_NAMEDescription: DESCRIPTION_TEXTValue: TRANSACTION_VALUE The classification is:'''# Get a test transactiontransaction = transactions.iloc[0]# Interpolate the values into the promptprompt = zero_shot_prompt.replace('SUPPLIER_NAME',transaction['Supplier'])prompt = prompt.replace('DESCRIPTION_TEXT',transaction['Description'])prompt = prompt.replace('TRANSACTION_VALUE',str(transaction['Transaction value (£)']))# Use our completion function to return a predictioncompletion_response = request_completion(prompt)print(completion_response['choices'][0]['text'])Our first attempt is correct, M & J Ballantyne Ltd are a house builder and the work they performed is indeed Building Improvement


Setup

Lets expand the sample size to 25 and see how it performs, again with just a simple prompt to guide ittest_transactions = transactions.iloc[:25]test_transactions['Classification'] = test_transactions.apply(lambda x: classify_transaction(x,zero_shot_prompt),axis=1)test_transactions['Classification'].value_counts()test_transactions.head(25) .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Date Supplier Description Transaction value (£) Classification 0 21/04/2016 M & J Ballantyne Ltd George IV Bridge Work 35098.0 Building Improvement 1 26/04/2016 Private Sale Literary & Archival Items 30000.0 Literature & Archive 2 30/04/2016 City Of Edinburgh Council Non Domestic Rates 40800.0 Utility Bills 3 09/05/2016 Computacenter Uk Kelvin Hall 72835.0 Software/IT 4 09/05/2016 John Graham Construction Ltd Causewayside Refurbishment 64361.0 Building Improvement 5 09/05/2016 A McGillivray Causewayside Refurbishment 53690.0 Building Improvement 6 16/05/2016 John Graham Construction Ltd Causewayside Refurbishment 365344.0 Building Improvement 7 23/05/2016 Computacenter Uk Kelvin Hall 26506.0 Software/IT 8 23/05/2016 ECG Facilities Service Facilities Management Charge 32777.0 Building Improvement 9 23/05/2016 ECG Facilities Service Facilities Management Charge 32777.0 Building Improvement 10 30/05/2016 ALDL ALDL Charges 32317.0 Could not classify 11 10/06/2016 Wavetek Ltd Kelvin Hall 87589.0 Could not classify 12 10/06/2016 John Graham Construction Ltd Causewayside Refurbishment 381803.0 Building Improvement 13 28/06/2016 ECG Facilities Service Facilities Management Charge 32832.0 Building Improvement 14 30/06/2016 Glasgow City Council Kelvin Hall 1700000.0 Building Improvement 15 11/07/2016 Wavetek Ltd Kelvin Hall 65692.0 Could not classify 16 11/07/2016 John Graham Construction Ltd Causewayside Refurbishment 139845.0 Building Improvement 17 15/07/2016 Sotheby'S Literary & Archival Items 28500.0 Literature & Archive 18 18/07/2016 Christies Literary & Archival Items 33800.0 Literature & Archive 19 25/07/2016 A McGillivray Causewayside Refurbishment 30113.0 Building Improvement 20 31/07/2016 ALDL ALDL Charges 32317.0 Could not classify 21 08/08/2016 ECG Facilities Service Facilities Management Charge 32795.0 Building Improvement 22 15/08/2016 Creative Video Productions Ltd Kelvin Hall 26866.0 Could not classify 23 15/08/2016 John Graham Construction Ltd Causewayside Refurbishment 196807.0 Building Improvement 24 24/08/2016 ECG Facilities Service Facilities Management Charge 32795.0 Building Improvement Initial results are pretty good even with no labelled examples! The ones that it could not classify were tougher cases with few clues as to their topic, but maybe if we clean up the labelled dataset to give more examples we can get better performance.Classification with Embeddings Lets create embeddings from the small set that we've classified so far - we've made a set of labelled examples by running the zero-shot classifier on 101 transactions from our dataset and manually correcting the 15 Could not classify results that we got Create embeddings This initial section reuses the approach from the Get_embeddings_from_dataset Notebook to create embeddings from a combined field concatenating all of our featuresdf = pd.read_csv('./data/labelled_transactions.csv')df.head() .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Date Supplier Description Transaction value (£) Classification 0 15/08/2016 Creative Video Productions Ltd Kelvin Hall 26866 Other 1 29/05/2017 John Graham Construction Ltd Causewayside Refurbishment 74806 Building Improvement 2 29/05/2017 Morris & Spottiswood Ltd George IV Bridge Work 56448 Building Improvement 3 31/05/2017 John Graham Construction Ltd Causewayside Refurbishment 164691 Building Improvement 4 24/07/2017 John Graham Construction Ltd Causewayside Refurbishment 27926 Building Improvement df['combined'] = 'Supplier: ' + df['Supplier'].str.strip() + '; Description: ' + df['Description'].str.strip() + '; Value: ' + str(df['Transaction value (£)']).strip()df.head(2) .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Date Supplier Description Transaction value (£) Classification combined 0 15/08/2016 Creative Video Productions Ltd Kelvin Hall 26866 Other Supplier: Creative Video Productions Ltd; Desc..


Setup

Making the model predict a single token rather than multiple consecutive ones like 'Building Improvement' should give us better results We also need to add a common prefix and suffix to each example to aid the model in making predictions - in our case our text is already started with 'Supplier' and we'll add a suffix of '\n\n###\n\n' Lastly we'll aid a leading whitespace onto each of our target classes for classification, again to aid the model ft_prep_df = fs_df.copy()len(ft_prep_df)ft_prep_df.head() .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Unnamed: 0 Date Supplier Description Transaction value (£) Classification combined n_tokens babbage_similarity babbage_search 0 0 15/08/2016 Creative Video Productions Ltd Kelvin Hall 26866 Other Supplier: Creative Video Productions Ltd; Desc.. .  12 [-0.009630300104618073, 0.009887108579277992, ...  [-0.008217384107410908, 0.025170527398586273, ...  1 1 29/05/2017 John Graham Construction Ltd Causewayside Refurbishment 74806 Building Improvement Supplier: John Graham Construction Ltd; Descri...  16 [-0.006144719664007425, -0.0018709596479311585...  [-0.007424891460686922, 0.008475713431835175, ...  2 2 29/05/2017 Morris & Spottiswood Ltd George IV Bridge Work 56448 Building Improvement Supplier: Morris & Spottiswood Ltd; Descriptio...  17 [-0.005225738976150751, 0.015156379900872707, ...  [-0.007611643522977829, 0.030322374776005745, ...  3 3 31/05/2017 John Graham Construction Ltd Causewayside Refurbishment 164691 Building Improvement Supplier: John Graham Construction Ltd; Descri...  16 [-0.006144719664007425, -0.0018709596479311585...  [-0.007424891460686922, 0.008475713431835175, ...  4 4 24/07/2017 John Graham Construction Ltd Causewayside Refurbishment 27926 Building Improvement Supplier: John Graham Construction Ltd; Descri...  16 [-0.006144719664007425, -0.0018709596479311585...  [-0.007424891460686922, 0.008475713431835175, ...  classes = list(set(ft_prep_df['Classification']))class_df = pd.DataFrame(classes).reset_index()class_df.columns = ['class_id','class']class_df , len(class_df)ft_df_with_class = ft_prep_df.merge(class_df,left_on='Classification',right_on='class',how='inner')# Adding a leading whitespace onto each completion to help the modelft_df_with_class['class_id'] = ft_df_with_class.apply(lambda x: ' ' + str(x['class_id']),axis=1)ft_df_with_class = ft_df_with_class.drop('class', axis=1)# Adding a common separator onto the end of each prompt so the model knows when a prompt is terminatingft_df_with_class['prompt'] = ft_df_with_class.apply(lambda x: x['combined'] + '\n\n###\n\n',axis=1)ft_df_with_class.head() .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Unnamed: 0 Date Supplier Description Transaction value (£) Classification combined n_tokens babbage_similarity babbage_search class_id prompt 0 0 15/08/2016 Creative Video Productions Ltd Kelvin Hall 26866 Other Supplier: Creative Video Productions Ltd; Desc...  12 [-0.009630300104618073, 0.009887108579277992, ...  [-0.008217384107410908, 0.025170527398586273, ...  4 Supplier: Creative Video Productions Ltd; Desc...  1 51 31/03/2017 NLS Foundation Grant Payment 177500 Other Supplier: NLS Foundation; Description: Grant P...  11 [-0.022305507212877274, 0.008543581701815128, ...  [-0.020519884303212166, 0.01993306167423725, -...  4 Supplier: NLS Foundation; Description: Grant P...  2 70 26/06/2017 British Library Legal Deposit Services 50056 Other Supplier: British Library; Description: Legal ...  11 [-0.01019938476383686, 0.015277703292667866, -...  [-0.01843327097594738, 0.03343546763062477, -0...  4 Supplier: British Library; Description: Legal ...  3 71 24/07/2017 ALDL Legal Deposit Services 27067 Other Supplier: ALDL; Description: Legal Deposit Ser...  11 [-0.008471488021314144, 0.004098685923963785, ...  [-0.012966590002179146, 0.01299362163990736, 0...  4 Supplier: ALDL; Description: Legal Deposit Ser...  4 100 24/07/2017 AM Phillip Vehicle Purchase 26604 Other Supplier: AM Phillip; Description: Vehicle Pur..


Setup

If we had a few hundred or thousand we may get better results, but lets do one last test on a holdout set to confirm that it doesn't generalise well to a new set of observationsholdout_df = transactions.copy().iloc[101:]holdout_df.head() .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Date Supplier Description Transaction value (£) 101 23/10/2017 City Building LLP Causewayside Refurbishment 53147.0 102 30/10/2017 ECG Facilities Service Facilities Management Charge 35758.0 103 30/10/2017 ECG Facilities Service Facilities Management Charge 35758.0 104 06/11/2017 John Graham Construction Ltd Causewayside Refurbishment 134208.0 105 06/11/2017 ALDL Legal Deposit Services 27067.0 holdout_df['combined'] = 'Supplier: ' + holdout_df['Supplier'].str.strip() + '; Description: ' + holdout_df['Description'].str.strip() + '\n\n###\n\n' # + '; Value: ' + str(df['Transaction value (£)']).strip()holdout_df['prediction_result'] = holdout_df.apply(lambda x: openai.Completion.create(model=fine_tuned_model, prompt=x['combined'], max_tokens=1, temperature=0, logprobs=5),axis=1)holdout_df['pred'] = holdout_df.apply(lambda x : x['prediction_result']['choices'][0]['text'],axis=1)holdout_df.head(10) .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Date Supplier Description Transaction value (£) combined prediction_result pred 101 23/10/2017 City Building LLP Causewayside Refurbishment 53147.0 Supplier: City Building LLP; Description: Caus.. .  {'id': 'cmpl-63YDadbYLo8xKsGY2vReOFCMgTOvG', '...  2 102 30/10/2017 ECG Facilities Service Facilities Management Charge 35758.0 Supplier: ECG Facilities Service; Description:...  {'id': 'cmpl-63YDbNK1D7UikDc3xi5ATihg5kQEt', '...  2 103 30/10/2017 ECG Facilities Service Facilities Management Charge 35758.0 Supplier: ECG Facilities Service; Description:...  {'id': 'cmpl-63YDbwfiHjkjMWsfTKNt6naeqPzOe', '...  2 104 06/11/2017 John Graham Construction Ltd Causewayside Refurbishment 134208.0 Supplier: John Graham Construction Ltd; Descri...  {'id': 'cmpl-63YDbWAndtsRqPTi2ZHZtPodZvOwr', '...  2 105 06/11/2017 ALDL Legal Deposit Services 27067.0 Supplier: ALDL; Description: Legal Deposit Ser...  {'id': 'cmpl-63YDbDu7WM3svYWsRAMdDUKtSFDBu', '...  2 106 27/11/2017 Maggs Bros Ltd Literary & Archival Items 26500.0 Supplier: Maggs Bros Ltd; Description: Literar...  {'id': 'cmpl-63YDbxNNI8ZH5CJJNxQ0IF9Zf925C', '...  0 107 30/11/2017 Glasgow City Council Kelvin Hall 42345.0 Supplier: Glasgow City Council; Description: K...  {'id': 'cmpl-63YDb8R1FWu4bjwM2xE775rouwneV', '...  2 108 11/12/2017 ECG Facilities Service Facilities Management Charge 35758.0 Supplier: ECG Facilities Service; Description:...  {'id': 'cmpl-63YDcAPsp37WhbPs9kwfUX0kBk7Hv', '...  2 109 11/12/2017 John Graham Construction Ltd Causewayside Refurbishment 159275.0 Supplier: John Graham Construction Ltd; Descri...  {'id': 'cmpl-63YDcML2welrC3wF0nuKgcNmVu1oQ', '...  2 110 08/01/2018 ECG Facilities Service Facilities Management Charge 35758.0 Supplier: ECG Facilities Service; Description:...  {'id': 'cmpl-63YDc95SSdOHnIliFB2cjMEEm7Z2u', '...  2 holdout_df['pred'].value_counts()Well those results were similarly underwhelming - so we've learned that with a dataset with a small number of labelled observations, either zero-shot classification or traditional classification with embeddings return better results than a fine-tuned model.  A fine-tuned model is still a great tool, but is more effective when you have a larger number of labelled examples for each class that you're looking to classify


Setup

1 29/05/2017 John Graham Construction Ltd Causewayside Refurbishment 74806 Building Improvement Supplier: John Graham Construction Ltd; Descri.. .  from transformers import GPT2TokenizerFasttokenizer = GPT2TokenizerFast.from_pretrained('gpt2')df['n_tokens'] = df.combined.apply(lambda x: len(tokenizer.encode(x)))len(df)embedding_path = './data/transactions_with_embeddings_100.csv'from openai.embeddings_utils import get_embeddingdf['babbage_similarity'] = df.combined.apply(lambda x: get_embedding(x, engine='text-similarity-babbage-001'))df['babbage_search'] = df.combined.apply(lambda x: get_embedding(x, engine='text-search-babbage-doc-001'))df.to_csv(embedding_path)Use embeddings for classification Now that we have our embeddings, let see if classifying these into the categories we've named gives us any more success.  For this we'll use a template from the Classification_using_embeddings notebookfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import classification_report, accuracy_scorefrom ast import literal_evalfs_df = pd.read_csv(embedding_path)fs_df['babbage_similarity'] = fs_df.babbage_similarity.apply(literal_eval).apply(np.array)fs_df.head() .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Unnamed: 0 Date Supplier Description Transaction value (£) Classification combined n_tokens babbage_similarity babbage_search 0 0 15/08/2016 Creative Video Productions Ltd Kelvin Hall 26866 Other Supplier: Creative Video Productions Ltd; Desc...  136 [-0.009802100248634815, 0.022551486268639565, ...  [-0.00232666521333158, 0.019198870286345482, 0...  1 1 29/05/2017 John Graham Construction Ltd Causewayside Refurbishment 74806 Building Improvement Supplier: John Graham Construction Ltd; Descri...  140 [-0.009065819904208183, 0.012094118632376194, ...  [0.005169447045773268, 0.00473341578617692, -0...  2 2 29/05/2017 Morris & Spottiswood Ltd George IV Bridge Work 56448 Building Improvement Supplier: Morris & Spottiswood Ltd; Descriptio...  141 [-0.009000026620924473, 0.02405017428100109, -...  [0.0028343256562948227, 0.021166473627090454, ...  3 3 31/05/2017 John Graham Construction Ltd Causewayside Refurbishment 164691 Building Improvement Supplier: John Graham Construction Ltd; Descri...  140 [-0.009065819904208183, 0.012094118632376194, ...  [0.005169447045773268, 0.00473341578617692, -0...  4 4 24/07/2017 John Graham Construction Ltd Causewayside Refurbishment 27926 Building Improvement Supplier: John Graham Construction Ltd; Descri...  140 [-0.009065819904208183, 0.012094118632376194, ...  [0.005169447045773268, 0.00473341578617692, -0...  X_train, X_test, y_train, y_test = train_test_split( list(fs_df.babbage_similarity.values), fs_df.Classification, test_size=0.2, random_state=42)clf = RandomForestClassifier(n_estimators=100)clf.fit(X_train, y_train)preds = clf.predict(X_test)probas = clf.predict_proba(X_test)report = classification_report(y_test, preds)print(report)Performance for this model is pretty strong, so creating embeddings and using even a simpler classifier looks like an effective approach as well, with the zero-shot classifier helping us do the initial classification of the unlabelled dataset.  Lets take it one step further and see if a fine-tuned model trained on this same labelled datasets gives us comparable resultsFine-tuned Transaction Classification For this use case we're going to try to improve on the few-shot classification from above by training a fine-tuned model on the same labelled set of 101 transactions and applying this fine-tuned model on group of unseen transactionsBuilding Fine-tuned Classifier We'll need to do some data prep first to get our data ready.  This will take the following steps: First we'll list out our classes and replace them with numeric identifiers


Zero-shot Classification

We'll first assess the performance of the base models at classifying these transactions using a simple prompt. We'll provide the model with 5 categories and a catch-all of 'Could not classify' for ones that it cannot place.


Setup

10 [-0.003459023078903556, 0.004626389592885971, .. .  [-0.0010945454705506563, 0.008626140654087067,...  4 Supplier: AM Phillip; Description: Vehicle Pur...  # This step is unnecessary if you have a number of observations in each class# In our case we don't, so we shuffle the data to give us a better chance of getting equal classes in our train and validation sets# Our fine-tuned model will error if we have less classes in the validation set, so this is a necessary stepimport random labels = [x for x in ft_df_with_class['class_id']]text = [x for x in ft_df_with_class['prompt']]ft_df = pd.DataFrame(zip(text, labels), columns = ['prompt','class_id']) #[:300]ft_df.columns = ['prompt','completion']ft_df['ordering'] = ft_df.apply(lambda x: random.randint(0,len(ft_df)), axis = 1)ft_df.set_index('ordering',inplace=True)ft_df_sorted = ft_df.sort_index(ascending=True)ft_df_sorted.head() .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } prompt completion ordering 0 Supplier: Sothebys; Description: Literary & Ar...  0 1 Supplier: Sotheby'S; Description: Literary & A...  0 2 Supplier: City Of Edinburgh Council; Descripti...  1 2 Supplier: John Graham Construction Ltd; Descri...  2 3 Supplier: John Graham Construction Ltd; Descri...  2 # This step is to remove any existing files if we've already produced training/validation sets for this classifier#!rm transactions_grouped*# We output our shuffled dataframe to a .jsonl file and run the prepare_data function to get us our input filesft_df_sorted.to_json('transactions_grouped.jsonl', orient='records', lines=True)!openai tools fine_tunes.prepare_data -f transactions_grouped.jsonl -q# This functions checks that your classes all appear in both prepared files# If they don't, the fine-tuned model creation will failcheck_finetune_classes('transactions_grouped_prepared_train.jsonl','transactions_grouped_prepared_valid.jsonl')# This step creates your model!openai api fine_tunes.create -t 'transactions_grouped_prepared_train.jsonl' -v 'transactions_grouped_prepared_valid.jsonl' --compute_classification_metrics --classification_n_classes 5 -m curie# You can use following command to get fine tuning job status and model name, replace the job name with your job#!openai api fine_tunes.get -i ft-YBIc01t4hxYBC7I5qhRF3Qdx# Congrats, you've got a fine-tuned model!# Copy/paste the name provided into the variable below and we'll take it for a spinfine_tuned_model = 'curie:ft-personal-2022-10-20-10-42-56'Applying Fine-tuned Classifier Now we'll apply our classifier to see how it performs.  We only had 31 unique observations in our training set and 8 in our validation set, so lets see how the performance istest_set = pd.read_json('transactions_grouped_prepared_valid.jsonl', lines=True)test_set.head() .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } prompt completion 0 Supplier: Wavetek Ltd; Description: Kelvin Hal...  2 1 Supplier: ECG Facilities Service; Description:...  1 2 Supplier: M & J Ballantyne Ltd; Description: G...  2 3 Supplier: Private Sale; Description: Literary ...  0 4 Supplier: Ex Libris; Description: IT equipment...  3 test_set['predicted_class'] = test_set.apply(lambda x: openai.Completion.create(model=fine_tuned_model, prompt=x['prompt'], max_tokens=1, temperature=0, logprobs=5),axis=1)test_set['pred'] = test_set.apply(lambda x : x['predicted_class']['choices'][0]['text'],axis=1)test_set['result'] = test_set.apply(lambda x: str(x['pred']).strip() == str(x['completion']).strip(), axis = 1)test_set['result'].value_counts()Performance is not great - unfortunately this is expected.  With only a few examples of each class, the above approach with embeddings and a traditional classifier worked better.  A fine-tuned model works best with a great number of labelled observations


Create embeddings

This initial section reuses the approach from the Get_embeddings_from_dataset Notebook to create embeddings from a combined field concatenating all of our features


Classification with Embeddings

Lets create embeddings from the small set that we've classified so far - we've made a set of labelled examples by running the zero-shot classifier on 101 transactions from our dataset and manually correcting the 15 Could not classify results that we got


Fine-tuned Transaction Classification

For this use case we're going to try to improve on the few-shot classification from above by training a fine-tuned model on the same labelled set of 101 transactions and applying this fine-tuned model on group of unseen transactions


Load dataset

We're using a public transaction dataset of transactions over £25k for the Library of Scotland. The dataset has three features that we'll be using: Supplier: The name of the supplier Description: A text description of the transaction Value: The value of the transaction in GBP Source:https://data.nls.uk/data/organisational-data/transactions-over-25k/


Building Fine-tuned Classifier

We'll need to do some data prep first to get our data ready. This will take the following steps: First we'll list out our classes and replace them with numeric identifiers. Making the model predict a single token rather than multiple consecutive ones like 'Building Improvement' should give us better results We also need to add a common prefix and suffix to each example to aid the model in making predictions - in our case our text is already started with 'Supplier' and we'll add a suffix of '\n\n###\n\n' Lastly we'll aid a leading whitespace onto each of our target classes for classification, again to aid the model


Use embeddings for classification

Now that we have our embeddings, let see if classifying these into the categories we've named gives us any more success.For this we'll use a template from the Classification_using_embeddings notebook


Applying Fine-tuned Classifier

Now we'll apply our classifier to see how it performs. We only had 31 unique observations in our training set and 8 in our validation set, so lets see how the performance is


How to handle rate limits

Open in GithubWhen you call the OpenAI API repeatedly, you may encounter error messages that say 429: 'Too Many Requests' or RateLimitError.  These error messages come from exceeding the API's rate limits.  This guide shares tips for avoiding and handling rate limit errors.  To see an example script for throttling parallel requests to avoid rate limit errors, see api_request_parallel_processor.py.  Why rate limits exist Rate limits are a common practice for APIs, and they're put in place for a few different reasons.  First, they help protect against abuse or misuse of the API.  For example, a malicious actor could flood the API with requests in an attempt to overload it or cause disruptions in service.  By setting rate limits, OpenAI can prevent this kind of activity.  Second, rate limits help ensure that everyone has fair access to the API.  If one person or organization makes an excessive number of requests, it could bog down the API for everyone else.  By throttling the number of requests that a single user can make, OpenAI ensures that everyone has an opportunity to use the API without experiencing slowdowns.  Lastly, rate limits can help OpenAI manage the aggregate load on its infrastructure.  If requests to the API increase dramatically, it could tax the servers and cause performance issues.  By setting rate limits, OpenAI can help maintain a smooth and consistent experience for all users.  Although hitting rate limits can be frustrating, rate limits exist to protect the reliable operation of the API for its users.Default rate limits As of Jan 2023, the default rate limits are: For reference, 1,000 tokens is roughly a page of text.  Other rate limit resources Read more about OpenAI's rate limits in these other resources: Guide: Rate limits Help Center: Is API usage subject to any rate limits? Help Center: How can I solve 429: 'Too Many Requests' errors? Requesting a rate limit increase If you'd like your organization's rate limit increased, please fill out the following form: OpenAI Rate Limit Increase Request form Example rate limit error A rate limit error will occur when API requests are sent too quickly.  If using the OpenAI Python library, they will look something like: RateLimitError: Rate limit reached for default-codex in organization org-{id} on requests per min.  Limit: 20.000000 / min.  Current: 24.000000 / min.  Contact support@openai.com if you continue to have issues or if you'd like to request an increase.  Below is example code for triggering a rate limit error.import openai # for making OpenAI API requests# request a bunch of completions in a loopfor _ in range(100): openai.ChatCompletion.create( model='gpt-3.5-turbo', messages=[{'role': 'user', 'content': 'Hello'}], max_tokens=10, )How to avoid rate limit errors Retrying with exponential backoff One easy way to avoid rate limit errors is to automatically retry requests with a random exponential backoff


How to handle rate limits

Retrying with exponential backoff means performing a short sleep when a rate limit error is hit, then retrying the unsuccessful request .  If the request is still unsuccessful, the sleep length is increased and the process is repeated.  This continues until the request is successful or until a maximum number of retries is reached.  This approach has many benefits: Automatic retries means you can recover from rate limit errors without crashes or missing data Exponential backoff means that your first retries can be tried quickly, while still benefiting from longer delays if your first few retries fail Adding random jitter to the delay helps retries from all hitting at the same time Note that unsuccessful requests contribute to your per-minute limit, so continuously resending a request won't work.  Below are a few example solutions.Example #1: Using the Tenacity library Tenacity is an Apache 2.0 licensed general-purpose retrying library, written in Python, to simplify the task of adding retry behavior to just about anything.  To add exponential backoff to your requests, you can use the tenacity.retry decorator.  The following example uses the tenacity.wait_random_exponential function to add random exponential backoff to a request.  Note that the Tenacity library is a third-party tool, and OpenAI makes no guarantees about its reliability or security.import openai # for OpenAI API callsfrom tenacity import ( retry, stop_after_attempt, wait_random_exponential,) # for exponential backoff@retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(6))def completion_with_backoff(**kwargs): return openai.Completion.create(**kwargs)completion_with_backoff(model='text-davinci-002', prompt='Once upon a time,')Example #2: Using the backoff library Another library that provides function decorators for backoff and retry is backoff.  Like Tenacity, the backoff library is a third-party tool, and OpenAI makes no guarantees about its reliability or security.import backoff # for exponential backoffimport openai # for OpenAI API calls@backoff.on_exception(backoff.expo, openai.error.RateLimitError)def completions_with_backoff(**kwargs): return openai.Completion.create(**kwargs)completions_with_backoff(model='text-davinci-002', prompt='Once upon a time,')Example 3: Manual backoff implementation If you don't want to use third-party libraries, you can implement your own backoff logic.# importsimport randomimport timeimport openai# define a retry decoratordef retry_with_exponential_backoff( func, initial_delay: float = 1, exponential_base: float = 2, jitter: bool = True, max_retries: int = 10, errors: tuple = (openai.error.RateLimitError,),): '''Retry a function with exponential backoff.''' def wrapper(*args, **kwargs): # Initialize variables num_retries = 0 delay = initial_delay # Loop until a successful response or max_retries is hit or an exception is raised while True: try: return func(*args, **kwargs) # Retry on specified errors except errors as e: # Increment retries num_retries += 1 # Check if max retries has been reached if num_retries > max_retries: raise Exception( f'Maximum number of retries ({max_retries}) exceeded.' ) # Increment the delay delay *= exponential_base * (1 + jitter * random.random()) # Sleep for the delay time.sleep(delay) # Raise exceptions for any errors not specified except Exception as e: raise e return wrapper@retry_with_exponential_backoffdef completions_with_backoff(**kwargs): return openai.Completion.create(**kwargs)completions_with_backoff(model='text-davinci-002', prompt='Once upon a time,')How to maximize throughput of batch processing given rate limits If you're processing real-time requests from users, backoff and retry is a great strategy to minimize latency while avoiding rate limit errors


Why rate limits exist

Rate limits are a common practice for APIs, and they're put in place for a few different reasons. First, they help protect against abuse or misuse of the API. For example, a malicious actor could flood the API with requests in an attempt to overload it or cause disruptions in service. By setting rate limits, OpenAI can prevent this kind of activity. Second, rate limits help ensure that everyone has fair access to the API. If one person or organization makes an excessive number of requests, it could bog down the API for everyone else. By throttling the number of requests that a single user can make, OpenAI ensures that everyone has an opportunity to use the API without experiencing slowdowns. Lastly, rate limits can help OpenAI manage the aggregate load on its infrastructure. If requests to the API increase dramatically, it could tax the servers and cause performance issues. By setting rate limits, OpenAI can help maintain a smooth and consistent experience for all users. Although hitting rate limits can be frustrating, rate limits exist to protect the reliable operation of the API for its users.


How to handle rate limits

However, if you're processing large volumes of batch data, where throughput matters more than latency, there are a few other things you can do in addition to backoff and retry .  Proactively adding delay between requests If you are constantly hitting the rate limit, then backing off, then hitting the rate limit again, then backing off again, it's possible that a good fraction of your request budget will be 'wasted' on requests that need to be retried.  This limits your processing throughput, given a fixed rate limit.  Here, one potential solution is to calculate your rate limit and add a delay equal to its reciprocal (e.g., if your rate limit 20 requests per minute, add a delay of 3–6 seconds to each request).  This can help you operate near the rate limit ceiling without hitting it and incurring wasted requests.  Example of adding delay to a request# importsimport timeimport openai# Define a function that adds a delay to a Completion API calldef delayed_completion(delay_in_seconds: float = 1, **kwargs): '''Delay a completion by a specified amount of time.''' # Sleep for the delay time.sleep(delay_in_seconds) # Call the Completion API and return the result return openai.Completion.create(**kwargs)# Calculate the delay based on your rate limitrate_limit_per_minute = 20delay = 60.0 / rate_limit_per_minutedelayed_completion( delay_in_seconds=delay, model='text-davinci-002', prompt='Once upon a time,')Batching requests The OpenAI API has separate limits for requests per minute and tokens per minute.  If you're hitting the limit on requests per minute, but have headroom on tokens per minute, you can increase your throughput by batching multiple tasks into each request.  This will allow you to process more tokens per minute, especially with the smaller models.  Sending in a batch of prompts works exactly the same as a normal API call, except that pass in a list of strings to prompt parameter instead of a single string.  Warning: the response object may not return completions in the order of the prompts, so always remember to match responses back to prompts using the index field.  Example without batchingimport openai # for making OpenAI API requestsnum_stories = 10prompt = 'Once upon a time,'# serial example, with one story completion per requestfor _ in range(num_stories): response = openai.Completion.create( model='curie', prompt=prompt, max_tokens=20, ) # print story print(prompt + response.choices[0].text)


Example rate limit error

A rate limit error will occur when API requests are sent too quickly. If using the OpenAI Python library, they will look something like:RateLimitError: Rate limit reached for default-codex in organization org-{id} on requests per min. Limit: 20.000000 / min. Current: 24.000000 / min. Contact support@openai.com if you continue to have issues or if you'd like to request an increase.Below is example code for triggering a rate limit error.


Requesting a rate limit increase

If you'd like your organization's rate limit increased, please fill out the following form: OpenAI Rate Limit Increase Request form


Other rate limit resources

Read more about OpenAI's rate limits in these other resources: Guide: Rate limits Help Center: Is API usage subject to any rate limits? Help Center: How can I solve 429: 'Too Many Requests' errors?


Example #1: Using the Tenacity library

Tenacity is an Apache 2.0 licensed general-purpose retrying library, written in Python, to simplify the task of adding retry behavior to just about anything.To add exponential backoff to your requests, you can use the tenacity.retry decorator. The following example uses the tenacity.wait_random_exponential function to add random exponential backoff to a request.Note that the Tenacity library is a third-party tool, and OpenAI makes no guarantees about its reliability or security.


Retrying with exponential backoff

One easy way to avoid rate limit errors is to automatically retry requests with a random exponential backoff. Retrying with exponential backoff means performing a short sleep when a rate limit error is hit, then retrying the unsuccessful request. If the request is still unsuccessful, the sleep length is increased and the process is repeated. This continues until the request is successful or until a maximum number of retries is reached.This approach has many benefits: Automatic retries means you can recover from rate limit errors without crashes or missing data Exponential backoff means that your first retries can be tried quickly, while still benefiting from longer delays if your first few retries fail Adding random jitter to the delay helps retries from all hitting at the same time Note that unsuccessful requests contribute to your per-minute limit, so continuously resending a request won't work.Below are a few example solutions.


Default rate limits

As of Jan 2023, the default rate limits are:For reference, 1,000 tokens is roughly a page of text.


Example 3: Manual backoff implementation

If you don't want to use third-party libraries, you can implement your own backoff logic.


Example #2: Using the backoff library

Another library that provides function decorators for backoff and retry is backoff.Like Tenacity, the backoff library is a third-party tool, and OpenAI makes no guarantees about its reliability or security.


Batching requests

The OpenAI API has separate limits for requests per minute and tokens per minute.If you're hitting the limit on requests per minute, but have headroom on tokens per minute, you can increase your throughput by batching multiple tasks into each request. This will allow you to process more tokens per minute, especially with the smaller models.Sending in a batch of prompts works exactly the same as a normal API call, except that pass in a list of strings to prompt parameter instead of a single string.Warning: the response object may not return completions in the order of the prompts, so always remember to match responses back to prompts using the index field.


Example with batching

import openai # for making OpenAI API requestsnum_stories = 10prompts = ['Once upon a time,'] * num_stories# batched example, with 10 stories completions per requestresponse = openai.Completion.create( model='curie', prompt=prompts, max_tokens=20,)# match completions to prompts by indexstories = [''] * len(prompts)for choice in response.choices: stories[choice.index] = prompts[choice.index] + choice.text# print storiesfor story in stories: print(story)Example parallel processing script We've written an example script for parallel processing large quantities of API requests: api_request_parallel_processor.py. The script combines some handy features: Streams requests from file, to avoid running out of memory for giant jobs Makes requests concurrently, to maximize throughput Throttles both request and token usage, to stay under rate limits Retries failed requests, to avoid missing data Logs errors, to diagnose problems with requests Feel free to use it as is or modify it to suit your needs.


How to maximize throughput of batch processing given rate limits

If you're processing real-time requests from users, backoff and retry is a great strategy to minimize latency while avoiding rate limit errors.However, if you're processing large volumes of batch data, where throughput matters more than latency, there are a few other things you can do in addition to backoff and retry.


Example parallel processing script

We've written an example script for parallel processing large quantities of API requests: api_request_parallel_processor.py.The script combines some handy features: Streams requests from file, to avoid running out of memory for giant jobs Makes requests concurrently, to maximize throughput Throttles both request and token usage, to stay under rate limits Retries failed requests, to avoid missing data Logs errors, to diagnose problems with requests Feel free to use it as is or modify it to suit your needs.


Proactively adding delay between requests

If you are constantly hitting the rate limit, then backing off, then hitting the rate limit again, then backing off again, it's possible that a good fraction of your request budget will be 'wasted' on requests that need to be retried. This limits your processing throughput, given a fixed rate limit.Here, one potential solution is to calculate your rate limit and add a delay equal to its reciprocal (e.g., if your rate limit 20 requests per minute, add a delay of 3–6 seconds to each request). This can help you operate near the rate limit ceiling without hitting it and incurring wasted requests.


Azure embeddings example

Create a new deployment by going to your Resource in your portal under 'Resource Management' -> 'Model deployments'.deployment_id = '' # Fill in the deployment id from the portal hereDeployments: Listing Now because creating a new deployment takes a long time, let's look in the subscription for an already finished deployment that succeeded.print('While deployment running, selecting a completed one that supports embeddings.')deployment_id = Noneresult = openai.Deployment.list()for deployment in result.data: if deployment['status'] != 'succeeded': continue model = openai.Model.retrieve(deployment['model']) if model['capabilities']['embeddings'] != True: continue deployment_id = deployment['id'] breakif not deployment_id: print('No deployment with status: succeeded found.')else: print(f'Found a succeeded deployment that supports embeddings with id: {deployment_id}.')Embeddings Now let's send a sample embedding to the deployment.embeddings = openai.Embedding.create(deployment_id=deployment_id, input='The food was delicious and the waiter...') print(embeddings)


Azure embeddings example

Open in GithubIn this example we'll try to go over all operations for embeddings that can be done using the Azure endpoints.  This example focuses on embeddings but also touches some other operations that are also available using the API.  This example is meant to be a quick way of showing simple operations and is not meant as a tutorial.import osimport openaiSetup For the following sections to work properly we first have to setup some things.  Let's start with the api_base and api_version.  To find your api_base go to https://portal.azure.com, find your resource and then under 'Resource Management' -> 'Keys and Endpoints' look for the 'Endpoint' value.openai.api_version = '2023-05-15'openai.api_base = '' # Please add your endpoint hereWe next have to setup the api_type and api_key.  We can either get the key from the portal or we can get it through Microsoft Active Directory Authentication.  Depending on this the api_type is either azure or azure_ad.Setup: Portal Let's first look at getting the key from the portal.  Go to https://portal.azure.com, find your resource and then under 'Resource Management' -> 'Keys and Endpoints' look for one of the 'Keys' values.openai.api_type = 'azure'openai.api_key = os.environ['OPENAI_API_KEY'] Note: In this example, we configured the library to use the Azure API by setting the variables in code.  For development, consider setting the environment variables instead: OPENAI_API_BASE OPENAI_API_KEY OPENAI_API_TYPE OPENAI_API_VERSION(Optional) Setup: Microsoft Active Directory Authentication Let's now see how we can get a key via Microsoft Active Directory Authentication.  Uncomment the following code if you want to use Active Directory Authentication instead of keys from the portal.# from azure.identity import DefaultAzureCredential# default_credential = DefaultAzureCredential()# token = default_credential.get_token('https://cognitiveservices.azure.com/.default')# openai.api_type = 'azure_ad'# openai.api_key = token.tokenA token is valid for a period of time, after which it will expire.  To ensure a valid token is sent with every request, you can refresh an expiring token by hooking into requests.auth:import typingimport timeimport requestsif typing.TYPE_CHECKING: from azure.core.credentials import TokenCredentialclass TokenRefresh(requests.auth.AuthBase): def __init__(self, credential: 'TokenCredential', scopes: typing.List[str]) -> None: self.credential = credential self.scopes = scopes self.cached_token: typing.Optional[str] = None def __call__(self, req): if not self.cached_token or self.cached_token.expires_on - time.time() < 300: self.cached_token = self.credential.get_token(*self.scopes) req.headers['Authorization'] = f'Bearer {self.cached_token.token}' return reqsession = requests.Session()session.auth = TokenRefresh(default_credential, ['https://cognitiveservices.azure.com/.default'])openai.requestssession = sessionDeployments In this section we are going to create a deployment that we can use to create embeddings.Deployments: Create manually Let's create a deployment using the text-similarity-curie-001 model


Setup: Portal

Let's first look at getting the key from the portal. Go to https://portal.azure.com, find your resource and then under 'Resource Management' -> 'Keys and Endpoints' look for one of the 'Keys' values.


Deployments: Create manually

Let's create a deployment using the text-similarity-curie-001 model. Create a new deployment by going to your Resource in your portal under 'Resource Management' -> 'Model deployments'.


Setup

For the following sections to work properly we first have to setup some things. Let's start with the api_base and api_version. To find your api_base go to https://portal.azure.com, find your resource and then under 'Resource Management' -> 'Keys and Endpoints' look for the 'Endpoint' value.


Let's now see how we can get a key via Microsoft Active Directory Authentication. Uncomment the following code if you want to use Active Directory Authentication instead of keys from the portal.


Embeddings

Now let's send a sample embedding to the deployment.


Deployments

In this section we are going to create a deployment that we can use to create embeddings.


Deployments: Listing

Now because creating a new deployment takes a long time, let's look in the subscription for an already finished deployment that succeeded.


Classification using embeddings

Open in GithubThere are many ways to classify text. This notebook shares an example of text classification using embeddings. For many text classification tasks, we've seen fine-tuned models do better than embeddings. See an example of fine-tuned models for classification in Fine-tuned_classification.ipynb. We also recommend having more examples than embedding dimensions, which we don't quite achieve here. In this text classification task, we predict the score of a food review (1 to 5) based on the embedding of the review's text. We split the dataset into a training and a testing set for all the following tasks, so we can realistically evaluate performance on unseen data. The dataset is created in the Get_embeddings_from_dataset Notebook.# importsimport pandas as pdimport numpy as npfrom ast import literal_evalfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import classification_report, accuracy_score# load datadatafile_path = 'data/fine_food_reviews_with_embeddings_1k.csv'df = pd.read_csv(datafile_path)df['embedding'] = df.embedding.apply(literal_eval).apply(np.array) # convert string to array# split data into train and testX_train, X_test, y_train, y_test = train_test_split( list(df.embedding.values), df.Score, test_size=0.2, random_state=42)# train random forest classifierclf = RandomForestClassifier(n_estimators=100)clf.fit(X_train, y_train)preds = clf.predict(X_test)probas = clf.predict_proba(X_test)report = classification_report(y_test, preds)print(report)We can see that the model has learnt to distinguish between the categories decently. 5-star reviews show the best performance overall, and this is not too surprising, since they are the most common in the dataset.from openai.embeddings_utils import plot_multiclass_precision_recallplot_multiclass_precision_recall(probas, y_test, [1, 2, 3, 4, 5], clf)Unsurprisingly 5-star and 1-star reviews seem to be easier to predict. Perhaps with more data, the nuances between 2-4 stars could be better predicted, but there's also probably more subjectivity in how people use the inbetween scores.


Clustering

Open in GithubWe use a simple k-means algorithm to demonstrate how clustering can be done. Clustering can help discover valuable, hidden groupings within the data. The dataset is created in the Get_embeddings_from_dataset Notebook.# importsimport numpy as npimport pandas as pdfrom ast import literal_eval# load datadatafile_path = './data/fine_food_reviews_with_embeddings_1k.csv'df = pd.read_csv(datafile_path)df['embedding'] = df.embedding.apply(literal_eval).apply(np.array) # convert string to numpy arraymatrix = np.vstack(df.embedding.values)matrix.shape


2. Text samples in the clusters & naming the clusters

Let's show random samples from each cluster. We'll use text-davinci-003 to name the clusters, based on a random sample of 5 reviews from that cluster.


1. Find the clusters using K-means

We show the simplest use of K-means. You can pick the number of clusters that fits your use case best.from sklearn.cluster import KMeansn_clusters = 4kmeans = KMeans(n_clusters=n_clusters, init='k-means++', random_state=42)kmeans.fit(matrix)labels = kmeans.labels_df['Cluster'] = labelsdf.groupby('Cluster').Score.mean().sort_values()from sklearn.manifold import TSNEimport matplotlibimport matplotlib.pyplot as plttsne = TSNE(n_components=2, perplexity=15, random_state=42, init='random', learning_rate=200)vis_dims2 = tsne.fit_transform(matrix)x = [x for x, y in vis_dims2]y = [y for x, y in vis_dims2]for category, color in enumerate(['purple', 'green', 'red', 'blue']): xs = np.array(x)[df.Cluster == category] ys = np.array(y)[df.Cluster == category] plt.scatter(xs, ys, color=color, alpha=0.3) avg_x = xs.mean() avg_y = ys.mean() plt.scatter(avg_x, avg_y, marker='x', color=color, s=100)plt.title('Clusters identified visualized in language 2d using t-SNE')Visualization of clusters in a 2d projection. In this run, the green cluster (#1) seems quite different from the others. Let's see a few samples from each cluster.2. Text samples in the clusters & naming the clusters Let's show random samples from each cluster. We'll use text-davinci-003 to name the clusters, based on a random sample of 5 reviews from that cluster.import openai# Reading a review which belong to each group.rev_per_cluster = 5for i in range(n_clusters): print(f'Cluster {i} Theme:', end=' ') reviews = '\n'.join( df[df.Cluster == i] .combined.str.replace('Title: ', '') .str.replace('\n\nContent: ', ': ') .sample(rev_per_cluster, random_state=42) .values ) response = openai.Completion.create( engine='text-davinci-003', prompt=f'What do the following customer reviews have in common?\n\nCustomer reviews:\n'''\n{reviews}\n'''\n\nTheme:', temperature=0, max_tokens=64, top_p=1, frequency_penalty=0, presence_penalty=0, ) print(response['choices'][0]['text'].replace('\n', '')) sample_cluster_rows = df[df.Cluster == i].sample(rev_per_cluster, random_state=42) for j in range(rev_per_cluster): print(sample_cluster_rows.Score.values[j], end=', ') print(sample_cluster_rows.Summary.values[j], end=': ') print(sample_cluster_rows.Text.str[:70].values[j]) print('-' * 100)It's important to note that clusters will not necessarily match what you intend to use them for. A larger amount of clusters will focus on more specific patterns, whereas a small number of clusters will usually focus on largest discrepencies in the data.