Method

Published by Antonia Creswell et al., one extension of the chain-of-thought technique is to split the single prompt for generating explanations and answers into smaller parts. First, a prompt selects a relevant subset of facts from the text ('selection prompt'). Then, a second prompt infers a conclusion from the selected facts ('inference prompt'). These prompts are then alternated in a loop to generate multiple steps of reasoning and eventually land on a final answer. The authors illustrate the idea in the following figure: Source: Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning by Antonia Creswell et al. (2022)


Method

In the original selection-inference technique, specialized 'selection' and 'inference' prompts are alternated to select facts and make inferences from those facts, combining to generate a sequence of reasoning steps.The authors extend this technique with two additional components.First, the authors add a 'halter' model that, after each inference step, is asked whether the inferences thus far are sufficient to answer the question. If yes, then the model generates a final answer.The halter models brings a couple of advantages: it can tell the selection-inference process to stop or keep going, as necessary. if the process never halts, you'll get no answer, which is often preferable to a hallucinated guess Source: Faithful Reasoning Using Large Language Models by Antonia Creswell et al. (2022) Source: Faithful Reasoning Using Large Language Models by Antonia Creswell et al. (2022)Second, the authors add a value function, which is used to assess the quality of reasoning steps and search over multiple reasoning trajectories. This echoes a common theme for increasing reliability; instead of generating a single answer from the model, generate a set of answers and then use some type of value function / discriminator / verifier model to pick the best one. Source: Faithful Reasoning Using Large Language Models by Antonia Creswell et al. (2022)In addition to these two extensions, the authors also use a trick to reduce hallucination of fake facts. Rather than asking the model to write out factual sentences, they fine-tune a model to work with sentence labels (e.g., sen1) instead. This helps prevent the model from hallucinating fake facts not mentioned in the prompt context. Source: Faithful Reasoning Using Large Language Models by Antonia Creswell et al. (2022)


Results

When applied to a 7B-parameter model, the authors found that selection-inference prompting substantially improved performance relative to chain-of-thought prompting on the bAbi and Proof Writer benchmark tasks (both of which require longer sequences of reasoning steps). The best performance they achieved combined both selection-inference prompting with fine-tuning. Source: Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning by Antonia Creswell et al. (2022)


Results

The authors evaluated their technique on two benchmarks: the ProofWriter task (not shown) and EntailmentBankQA (shown). The technique increased accuracy substantially, especially on harder reasoning problems. Source: Faithful Reasoning Using Large Language Models by Antonia Creswell et al. (2022)](https://arxiv.org/abs/2208.14271)In addition, their sentence label manipulation trick essentially eliminated hallucination! Source: Faithful Reasoning Using Large Language Models by Antonia Creswell et al. (2022)](https://arxiv.org/abs/2208.14271)


Faithful reasoning architecture

A few months after publishing the selection-inference prompting technique, the authors extended the technique in a follow-up paper, with ideas for: figuring out when the selection-inference cycle should stop or continue adding a value function to help search over multiple reasoning paths reducing hallucination of fake facts by fine-tuning a model to reason about sentence labels (e.g., sen1) rather than writing out the sentences themselves


Implications

This paper illustrates a number of helpful lessons for improving the reliability of large language models: Split complex tasks into smaller, more reliable subtasks Generate your answer in a step-by-step fashion, evaluating it along the way Generate many possible answers and use another model or function to pick the ones that look best Reduce hallucination by constraining what the model can say (e.g., by using sentence labels instead of sentences) Maximize performance of models by fine-tuning them on specialized tasks To learn more, read the full paper.


Implications

Although the gains on these benchmarks were large, these benchmarks were specifically chosen because they required longer sequences of reasoning. On problems that don't require reasoning with many steps, the gains are likely smaller.The results highlight a couple of general lessons for working with large language models. One, splitting up complex tasks into smaller tasks is a great way to improve reliability and performance; the more atomic the task, the less room there is for the model to err. Two, getting maximum performance often means combining fine-tuning with whatever approach you've chosen.To learn more, read the full paper.


Method

Least-to-most prompting is another technique that splits up reasoning tasks into smaller, more reliable subtasks. The idea is to elicit a subtask from the model by prompting it with something like To solve {question}, we need to first solve: '. Then, with that subtask in hand, the model can generate a solution. The solution is appended to the original question and the process is repeated until a final answer is produced. Source: Least-to-most Prompting Enables Complex Reasoning in Large Language Models by Denny Zhou et al. (2022)


Results

When applied to benchmarks involving long reasoning chains using code-davinci-002 (which is optimized for code but can still understand text), the authors measured gains as large as 16% -> 99.7%! Source: Least-to-most Prompting Enables Complex Reasoning in Large Language Models by Denny Zhou et al. (2022)


Implications

Although the above gains from least-to-most prompting are impressive, they are measured on a very narrow set of tasks that require long reasoning chains.Still, they illustrate a common theme: increase reliability by (a) breaking complex tasks into smaller subtasks and (b) giving the model more time and space to work out the answer.To learn more, read the full paper.


Method

In contrast to the previous techniques, which try to maximize the likelihood of correct answers, another approach is to use GPT-3 to generate a tree of possible explanations (both correct and incorrect), and then analyze their relationships to guess at which set is correct. This technique was coined maieutic prompting by Jaehun Jung et al. in May 2022 (maieutic means relating to the Socratic method of asking questions to elicit ideas).The method is complicated, and works as follows: First, build a maieutic tree, where each node is a statement that could be true or false: Start with a multiple-choice question or true/false statement (e.g. War cannot have a tie) For each possible answer to the question, use the model to generate a corresponding explanation (with a prompt like War cannot have a tie? True, because) Then, prompt the model with the question and the generated explanation, and ask it to produce the answer. If reversing the explanation (with a prefix like It is wrong to say that {explanation}) reverses the answer, then the explanation is considered 'logically integral.' If an explanation is not logically integral, then repeat the above process recursively, with each explanation turned into a True or False question, and generate more explanations for each new question. After all of the recursive explaining is done, you end up with a tree of explanations, where each leaf on the tree has the property that reversing the explanation reverses the model's answer. Second, convert the tree into a graph of relations: For each node in the tree, calculate the model's relative belief in each node (inferred from the probability of getting an answer of True to given an explanation) For each pair of nodes in the tree, use the model to identify whether they are entailed (implied) or contradicted Third, find the most consistent set of beliefs and take those to be true: Specifically, using the strength of belief in each node and the logical relationships between them, formulate the problem as a weighted maximum satisfiability problem (MAX-SAT) Use a solver to the find the most self-consistent set of beliefs, and take those as true Source: Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations by Jaehun Jung et al. (2022)


Results

Source: Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations by Jaehun Jung et al. (2022)


Least-to-most prompting

In addition to doing poorly on long reasoning chains (where selection-inference shines), chain-of-thought prompting can especially struggle when the examples are short but the task is long.


Method

For tasks with a discrete set of answers, one simple way to improve reliability is to sample multiple explanations & answers from the model (using a positive temperature) and then pick the final answer that appears most often. Source: Self-Consistency Improves Chain of Thought Reasoning in Language Models by Xuezhi Wang et al. (2022)


Results

This technique lifted accuracies by anywhere from 1 to 24 percentage points on a suite of math and reasoning benchmarks. (Plotted below are results from Google's LaMDA model; using Google's larger PaLM model, the baselines were higher but the gains were a bit smaller.) Source: Self-Consistency Improves Chain of Thought Reasoning in Language Models by Xuezhi Wang et al. (2022)


Implications

Beyond the complexity, one limitation of this method is that it appears to only apply to questions that can be posed as multiple-choice.To learn more, read the full paper.


Implications

Although this technique is simple to implement, it can be costly. Generating a set of 10 answers will increase your costs by 10x.Also, as with many of these techniques, it applies only to tasks with a limited set of answers. For open-ended tasks where each answer is unique (such as writing a poem), it's not obvious what it would mean to pick the most common answer.Lastly, this technique ought to be most beneficial when there are multiple paths or phrasings to reach an answer; if there's only one path, then the technique may not help at all. An extreme example: If the task was to generate a single token answer, then taking the most common token from 100 generations would be no different than taking the token with the highest logprobs (which you can get with a single generation at temperature=0).


Verifiers

Another key technique for improving task performance is to train a verifier or discriminator model to evaluate the outputs of the main generative model. If the discriminator rejects the output, then you can resample the generative model until you get an acceptable output. In many cases, it's easier to judge an answer than it is to create an answer, which helps explain the power of this method.


Method

In 2021, OpenAI researchers applied this technique to grade school math problems, using the following procedure: First, they fine-tuned a model on questions and solutions For each problem in the training set, they generated 100 solutions Each of those 100 solutions was automatically labeled as either correct or incorrect, based on whether the final answer was correct Using those solutions, with some labeled correct and some labeled incorrect, they fine-tuned a verifier model to classify whether a question and candidate solution was correct or incorrect Finally, at test time, the generative model creates 100 solutions to each problem, and the one with the highest score according to the verifier model is picked as the final answer Source: Training Verifiers to Solve Math Word Problems by Karl Cobbe et al. (2021)


Implications

Similar to the self-consistency technique, this method can get expensive, as generating, say, 100 solutions per task will increase your costs by roughly ~100x.


Theories of reliability

Although the techniques above vary in their approach, they all share the goal of improving reliability on complex tasks. Mainly they do this by: decomposing unreliable operations into smaller, more reliable operations (e.g., selection-inference prompting) using multiple steps or multiple relationships to make the system's reliability greater than any individual component (e.g., maieutic prompting)


Chain of thought prompting

Source: Language Model Cascades by David Dohan et al. (2022)


Results

With a 175B GPT-3 model and 8,000 training examples, this technique substantially lifted grade school math accuracy from ~33% to ~55%. Source: Training Verifiers to Solve Math Word Problems by Karl Cobbe et al. (2021)


Probabilistic graphical models

This paradigm of trying to build a reliable system out of less reliable components is reminiscent of probabilistic programming, and many of the analysis techniques of that field can be applied to this one.In the paper Language Model Cascades, David Dohan et al. interpret the above techniques in the paradigm of probabilistic graphical models:


Fine-tuned chain of thought prompting / Self-taught reasoner


Implications

Although formulating these techniques as probabilistic graphical models may not be immediately useful for solving any particular problem, the framework may be helpful in selecting, combining, and discovering new techniques.


Selection-inference prompting


Verifiers


Bibliography

AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model PromptsTraining Verifiers to Solve Math Word ProblemsChain of Thought Prompting Elicits Reasoning in Large Language ModelsSelf-Consistency Improves Chain of Thought Reasoning in Language ModelsSTaR: Bootstrapping Reasoning With ReasoningLarge Language Models are Zero-Shot ReasonersSelection-Inference: Exploiting Large Language Models for Interpretable Logical ReasoningLeast-to-most Prompting Enables Complex Reasoning in Large Language ModelsMaieutic Prompting: Logically Consistent Reasoning with Recursive ExplanationsLanguage Model CascadesFaithful Reasoning Using Large Language Models


Closing thoughts

Research into large language models is very active and evolving rapidly. Not only do researchers continue to improve the models, they also continue to improve our understanding of how to best employ the models. To underscore the pace of these developments, note that all of the papers shared above were published within the past 12 months (as I write in Sep 2022).In the future, expect better models and better techniques to be published. Even if the specific techniques here are eclipsed by future best practices, the general principles behind them will likely remain a key part of any expert user's toolkit.


How to work with large language models

Open in GithubHow large language models work Large language models are functions that map text to text.  Given an input string of text, a large language model predicts the text that should come next.  The magic of large language models is that by being trained to minimize this prediction error over vast quantities of text, the models end up learning concepts useful for these predictions.  For example, they learn: how to spell how grammar works how to paraphrase how to answer questions how to hold a conversation how to write in many languages how to code etc.  None of these capabilities are explicitly programmed in—they all emerge as a result of training.  GPT-3 powers hundreds of software products, including productivity apps, education apps, games, and more.  How to control a large language model Of all the inputs to a large language model, by far the most influential is the text prompt.  Large language models can be prompted to produce output in a few ways: Instruction: Tell the model what you want Completion: Induce the model to complete the beginning of what you want Demonstration: Show the model what you want, with either: A few examples in the prompt Many hundreds or thousands of examples in a fine-tuning training dataset An example of each is shown below.  Instruction prompts Instruction-following models (e.g., text-davinci-003 or any model beginning with text-) are specially designed to follow instructions.  Write your instruction at the top of the prompt (or at the bottom, or both), and the model will do its best to follow the instruction and then stop.  Instructions can be detailed, so don't be afraid to write a paragraph explicitly detailing the output you want.  Example instruction prompt: Extract the name of the author from the quotation below.  'Some humans theorize that intelligent species go extinct before they can expand into outer space.  If they're correct, then the hush of the night sky is the silence of the graveyard.' ― Ted Chiang, Exhalation Output: Ted Chiang Completion prompt example Completion-style prompts take advantage of how large language models try to write text they think is mostly likely to come next.  To steer the model, try beginning a pattern or sentence that will be completed by the output you want to see.  Relative to direct instructions, this mode of steering large language models can take more care and experimentation.  In addition, the models won't necessarily know where to stop, so you will often need stop sequences or post-processing to cut off text generated beyond the desired output.  Example completion prompt: 'Some humans theorize that intelligent species go extinct before they can expand into outer space


How to work with large language models

If they're correct, then the hush of the night sky is the silence of the graveyard.' ― Ted Chiang, Exhalation The author of this quote is Output: Ted Chiang Demonstration prompt example (few-shot learning) Similar to completion-style prompts, demonstrations can show the model what you want it to do .  This approach is sometimes called few-shot learning, as the model learns from a few examples provided in the prompt.  Example demonstration prompt: Quote: 'When the reasoning mind is forced to confront the impossible again and again, it has no choice but to adapt.' ― N.K.  Jemisin, The Fifth Season Author: N.K.  Jemisin Quote: 'Some humans theorize that intelligent species go extinct before they can expand into outer space.  If they're correct, then the hush of the night sky is the silence of the graveyard.' ― Ted Chiang, Exhalation Author: Output: Ted Chiang Fine-tuned prompt example With enough training examples, you can fine-tune a custom model.  In this case, instructions become unnecessary, as the model can learn the task from the training data provided.  However, it can be helpful to include separator sequences (e.g., -> or ### or any string that doesn't commonly appear in your inputs) to tell the model when the prompt has ended and the output should begin.  Without separator sequences, there is a risk that the model continues elaborating on the input text rather than starting on the answer you want to see.  Example fine-tuned prompt (for a model that has been custom trained on similar prompt-completion pairs): 'Some humans theorize that intelligent species go extinct before they can expand into outer space.  If they're correct, then the hush of the night sky is the silence of the graveyard.' ― Ted Chiang, Exhalation ### Output: Ted Chiang Code Capabilities Large language models aren't only great at text - they can be great at code too.  OpenAI's specialized code model is called Codex.  Codex powers more than 70 products, including: GitHub Copilot (autocompletes code in VS Code and other IDEs) Pygma (turns Figma designs into code) Replit (has an 'Explain code' button and other features) Warp (a smart terminal with AI command search) Machinet (writes Java unit test templates) Note that unlike instruction-following text models (e.g., text-davinci-002), Codex is not trained to follow instructions.  As a result, designing good prompts can take more care.  More prompt advice For more prompt examples, visit OpenAI Examples.  In general, the input prompt is the best lever for improving model outputs.  You can try tricks like: Give more explicit instructions.  E.g., if you want the output to be a comma separated list, ask it to return a comma separated list.  If you want it to say 'I don't know' when it doesn't know the answer, tell it 'Say 'I don't know' if you do not know the answer.' Supply better examples


How to work with large language models

If you're demonstrating examples in your prompt, make sure that your examples are diverse and high quality .  Ask the model to answer as if it was an expert.  Explicitly asking the model to produce high quality output or output as if it was written by an expert can induce the model to give higher quality answers that it thinks an expert would write.  E.g., 'The following answer is correct, high-quality, and written by an expert.' Prompt the model to write down the series of steps explaining its reasoning.  E.g., prepend your answer with something like 'Let's think step by step.' Prompting the model to give an explanation of its reasoning before its final answer can increase the likelihood that its final answer is consistent and correct.


How to control a large language model

Of all the inputs to a large language model, by far the most influential is the text prompt.Large language models can be prompted to produce output in a few ways: Instruction: Tell the model what you want Completion: Induce the model to complete the beginning of what you want Demonstration: Show the model what you want, with either: A few examples in the prompt Many hundreds or thousands of examples in a fine-tuning training dataset An example of each is shown below.


Instruction prompts

Instruction-following models (e.g., text-davinci-003 or any model beginning with text-) are specially designed to follow instructions. Write your instruction at the top of the prompt (or at the bottom, or both), and the model will do its best to follow the instruction and then stop. Instructions can be detailed, so don't be afraid to write a paragraph explicitly detailing the output you want.Example instruction prompt:Extract the name of the author from the quotation below.'Some humans theorize that intelligent species go extinct before they can expand into outer space. If they're correct, then the hush of the night sky is the silence of the graveyard.'― Ted Chiang, ExhalationOutput:Ted Chiang


How large language models work

Large language models are functions that map text to text. Given an input string of text, a large language model predicts the text that should come next.The magic of large language models is that by being trained to minimize this prediction error over vast quantities of text, the models end up learning concepts useful for these predictions. For example, they learn: how to spell how grammar works how to paraphrase how to answer questions how to hold a conversation how to write in many languages how to code etc. None of these capabilities are explicitly programmed in—they all emerge as a result of training.GPT-3 powers hundreds of software products, including productivity apps, education apps, games, and more.


Completion prompt example

Completion-style prompts take advantage of how large language models try to write text they think is mostly likely to come next. To steer the model, try beginning a pattern or sentence that will be completed by the output you want to see. Relative to direct instructions, this mode of steering large language models can take more care and experimentation. In addition, the models won't necessarily know where to stop, so you will often need stop sequences or post-processing to cut off text generated beyond the desired output.Example completion prompt:'Some humans theorize that intelligent species go extinct before they can expand into outer space. If they're correct, then the hush of the night sky is the silence of the graveyard.'― Ted Chiang, ExhalationThe author of this quote isOutput: Ted Chiang


Fine-tuned prompt example

With enough training examples, you can fine-tune a custom model. In this case, instructions become unnecessary, as the model can learn the task from the training data provided. However, it can be helpful to include separator sequences (e.g., -> or ### or any string that doesn't commonly appear in your inputs) to tell the model when the prompt has ended and the output should begin. Without separator sequences, there is a risk that the model continues elaborating on the input text rather than starting on the answer you want to see.Example fine-tuned prompt (for a model that has been custom trained on similar prompt-completion pairs):'Some humans theorize that intelligent species go extinct before they can expand into outer space. If they're correct, then the hush of the night sky is the silence of the graveyard.'― Ted Chiang, Exhalation###Output: Ted Chiang


Code Capabilities

Large language models aren't only great at text - they can be great at code too. OpenAI's specialized code model is called Codex.Codex powers more than 70 products, including: GitHub Copilot (autocompletes code in VS Code and other IDEs) Pygma (turns Figma designs into code) Replit (has an 'Explain code' button and other features) Warp (a smart terminal with AI command search) Machinet (writes Java unit test templates) Note that unlike instruction-following text models (e.g., text-davinci-002), Codex is not trained to follow instructions. As a result, designing good prompts can take more care.


More prompt advice

For more prompt examples, visit OpenAI Examples.In general, the input prompt is the best lever for improving model outputs. You can try tricks like: Give more explicit instructions. E.g., if you want the output to be a comma separated list, ask it to return a comma separated list. If you want it to say 'I don't know' when it doesn't know the answer, tell it 'Say 'I don't know' if you do not know the answer.' Supply better examples. If you're demonstrating examples in your prompt, make sure that your examples are diverse and high quality. Ask the model to answer as if it was an expert. Explicitly asking the model to produce high quality output or output as if it was written by an expert can induce the model to give higher quality answers that it thinks an expert would write. E.g., 'The following answer is correct, high-quality, and written by an expert.' Prompt the model to write down the series of steps explaining its reasoning. E.g., prepend your answer with something like 'Let's think step by step.' Prompting the model to give an explanation of its reasoning before its final answer can increase the likelihood that its final answer is consistent and correct.


Demonstration prompt example (few-shot learning)

Similar to completion-style prompts, demonstrations can show the model what you want it to do. This approach is sometimes called few-shot learning, as the model learns from a few examples provided in the prompt.Example demonstration prompt:Quote:'When the reasoning mind is forced to confront the impossible again and again, it has no choice but to adapt.'― N.K. Jemisin, The Fifth SeasonAuthor: N.K. JemisinQuote:'Some humans theorize that intelligent species go extinct before they can expand into outer space. If they're correct, then the hush of the night sky is the silence of the graveyard.'― Ted Chiang, ExhalationAuthor:Output: Ted Chiang


How to fine-tune chat models

Open in GithubThis notebook provides a step-by-step guide for our new gpt-3.5-turbo fine-tuning. We'll perform entity extraction using the RecipeNLG dataset, which provides various recipes and a list of extracted generic ingredients for each. This is a common dataset for named entity recognition (NER) tasks. We will go through the following steps: Setup: Loading our dataset and filtering down to one domain to fine-tune on. Data preparation: Preparing your data for fine-tuning by creating training and validation examples, and uploading them to the Files endpoint. Fine-tuning: Creating your fine-tuned model. Inference: Using your fine-tuned model for inference on new inputs. By the end of this you should be able to train, evaluate and deploy a fine-tuned gpt-3.5-turbo model. For more information on fine-tuning, you can refer to our documentation guide, API reference or blog post


Setup

# make sure to use the latest version of the openai python package!pip install --upgrade openai import jsonimport openaiimport osimport pandas as pdfrom pprint import pprintOPENAI_API_KEY = os.getenv('OPENAI_API_KEY', '')Fine-tuning works best when focused on a particular domain.  It's important to make sure your dataset is both focused enough for the model to learn, but general enough that unseen examples won't be missed.  Having this in mind, we have extracted a subset from the RecipesNLG dataset to only contain documents from www.cookbooks.com.# Read in the dataset we'll use for this task.# This will be the RecipesNLG dataset, which we've cleaned to only contain documents from www.cookbooks.comrecipe_df = pd.read_csv('data/cookbook_recipes_nlg_10k.csv')recipe_df.head() .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } title ingredients directions link source NER 0 No-Bake Nut Cookies ['1 c.  firmly packed brown sugar', '1/2 c.  eva...  ['In a heavy 2-quart saucepan, mix brown sugar...  www.cookbooks.com/Recipe-Details.aspx?id=44874 www.cookbooks.com ['brown sugar', 'milk', 'vanilla', 'nuts', 'bu...  1 Jewell Ball'S Chicken ['1 small jar chipped beef, cut up', '4 boned ...  ['Place chipped beef on bottom of baking dish....  www.cookbooks.com/Recipe-Details.aspx?id=699419 www.cookbooks.com ['beef', 'chicken breasts', 'cream of mushroom...  2 Creamy Corn ['2 (16 oz.) pkg.  frozen corn', '1 (8 oz.) pkg...  ['In a slow cooker, combine all ingredients.  C...  www.cookbooks.com/Recipe-Details.aspx?id=10570 www.cookbooks.com ['frozen corn', 'cream cheese', 'butter', 'gar...  3 Chicken Funny ['1 large whole chicken', '2 (10 1/2 oz.) cans...  ['Boil and debone chicken.', 'Put bite size pi...  www.cookbooks.com/Recipe-Details.aspx?id=897570 www.cookbooks.com ['chicken', 'chicken gravy', 'cream of mushroo...  4 Reeses Cups(Candy) ['1 c.  peanut butter', '3/4 c.  graham cracker ...  ['Combine first four ingredients and press in ...  www.cookbooks.com/Recipe-Details.aspx?id=659239 www.cookbooks.com ['peanut butter', 'graham cracker crumbs', 'bu...  Data preparation We'll begin by preparing our data.  When fine-tuning with the ChatCompletion format, each training example is a simple list of messages.  For example, an entry could look like: [{'role': 'system', 'content': 'You are a helpful recipe assistant.  You are to extract the generic ingredients from each of the recipes provided.'}, {'role': 'user', 'content': 'Title: No-Bake Nut Cookies\n\nIngredients: ['1 c.  firmly packed brown sugar', '1/2 c.  evaporated milk', '1/2 tsp.  vanilla', '1/2 c.  broken nuts (pecans)', '2 Tbsp.  butter or margarine', '3 1/2 c.  bite size shredded rice biscuits']\n\nGeneric ingredients: '}, {'role': 'assistant', 'content': '['brown sugar', 'milk', 'vanilla', 'nuts', 'butter', 'bite size shredded rice biscuits']'}] During the training process this conversation will be split, with the final entry being the completion that the model will produce, and the remainder of the messages acting as the prompt.  Consider this when building your training examples - if your model will act on multi-turn conversations, then please provide representative examples so it doesn't perform poorly when the conversation starts to expand.  Please note that currently there is a 4096 token limit for each training example


Setup

Anything longer than this will be truncated at 4096 tokens.training_data = []system_message = 'You are a helpful recipe assistant .  You are to extract the generic ingredients from each of the recipes provided.'def create_user_message(row): return f'''Title: {row['title']}\n\nIngredients: {row['ingredients']}\n\nGeneric ingredients: '''def prepare_example_conversation(row): messages = [] messages.append({'role': 'system', 'content': system_message}) user_message = create_user_message(row) messages.append({'role': 'user', 'content': user_message}) messages.append({'role': 'assistant', 'content': row['NER']}) return {'messages': messages}pprint(prepare_example_conversation(recipe_df.iloc[0]))Let's now do this for a subset of the dataset to use as our training data.  You can begin with even 30-50 well-pruned examples.  You should see performance continue to scale linearly as you increase the size of the training set, but your jobs will also take longer.# use the first 100 rows of the dataset for trainingtraining_df = recipe_df.loc[0:100]# apply the prepare_example_conversation function to each row of the training_dftraining_data = training_df.apply(prepare_example_conversation, axis=1).tolist()for example in training_data[:5]: print(example)In addition to training data, we can also optionally provide validation data, which will be used to make sure that the model does not overfit your training set.validation_df = recipe_df.loc[101:200]validation_data = validation_df.apply(prepare_example_conversation, axis=1).tolist()We then need to save our data as .jsonl files, with each line being one training example conversation.def write_jsonl(data_list: list, filename: str) -> None: with open(filename, 'w') as out: for ddict in data_list: jout = json.dumps(ddict) + '\n' out.write(jout)training_file_name = 'tmp_recipe_finetune_training.jsonl'write_jsonl(training_data, training_file_name)validation_file_name = 'tmp_recipe_finetune_validation.jsonl'write_jsonl(validation_data, validation_file_name)This is what the first 5 lines of our training .jsonl file look like:# print the first 5 lines of the training file!head -n 5 tmp_recipe_finetune_training.jsonlUpload files You can now upload the files to our Files endpoint to be used by the fine-tuned model.training_response = openai.File.create( file=open(training_file_name, 'rb'), purpose='fine-tune')training_file_id = training_response['id']validation_response = openai.File.create( file=open(validation_file_name, 'rb'), purpose='fine-tune')validation_file_id = validation_response['id']print('Training file ID:', training_file_id)print('Validation file ID:', validation_file_id)Fine-tuning Now we can create our fine-tuning job with the generated files and an optional suffix to identify the model.  The response will contain an id which you can use to retrieve updates on the job.  Note: The files have to first be processed by our system, so you might get a File not ready error.  In that case, simply retry a few minutes later.response = openai.FineTuningJob.create( training_file=training_file_id, validation_file=validation_file_id, model='gpt-3.5-turbo', suffix='recipe-ner',)job_id = response['id']print('Job ID:', response['id'])print('Status:', response['status'])Check job status You can make a GET request to the https://api.openai.com/v1/alpha/fine-tunes endpoint to list your alpha fine-tune jobs.  In this instance you'll want to check that the ID you got from the previous step ends up as status: succeeded.  Once it is completed, you can use the result_files to sample the results from the validation set (if you uploaded one), and use the ID from the fine_tuned_model parameter to invoke your trained model.response = openai.FineTuningJob.retrieve(job_id)print('Job ID:', response['id'])print('Status:', response['status'])print('Trained Tokens:', response['trained_tokens'])We can track the progress of the fine-tune with the events endpoint.  You can rerun the cell below a few times until the fine-tune is ready.response = openai.FineTuningJob.list_events(id=job_id, limit=50)events = response['data']events.reverse()for event in events: print(event['message'])Now that it's done, we can get a fine-tuned model ID from the job:response = openai.FineTuningJob.retrieve(job_id)fine_tuned_model_id = response['fine_tuned_model']if fine_tuned_model_id is None: raise RuntimeError('Fine-tuned model ID not found


Upload files

You can now upload the files to our Files endpoint to be used by the fine-tuned model.


Data preparation

We'll begin by preparing our data. When fine-tuning with the ChatCompletion format, each training example is a simple list of messages. For example, an entry could look like:[{'role': 'system', 'content': 'You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided.'}, {'role': 'user', 'content': 'Title: No-Bake Nut Cookies\n\nIngredients: ['1 c. firmly packed brown sugar', '1/2 c. evaporated milk', '1/2 tsp. vanilla', '1/2 c. broken nuts (pecans)', '2 Tbsp. butter or margarine', '3 1/2 c. bite size shredded rice biscuits']\n\nGeneric ingredients: '}, {'role': 'assistant', 'content': '['brown sugar', 'milk', 'vanilla', 'nuts', 'butter', 'bite size shredded rice biscuits']'}]During the training process this conversation will be split, with the final entry being the completion that the model will produce, and the remainder of the messages acting as the prompt. Consider this when building your training examples - if your model will act on multi-turn conversations, then please provide representative examples so it doesn't perform poorly when the conversation starts to expand.Please note that currently there is a 4096 token limit for each training example. Anything longer than this will be truncated at 4096 tokens.


Inference

The last step is to use your fine-tuned model for inference. Similar to the classic FineTuning, you simply call ChatCompletions with your new fine-tuned model name filling the model parameter.test_df = recipe_df.loc[201:300]test_row = test_df.iloc[0]test_messages = []test_messages.append({'role': 'system', 'content': system_message})user_message = create_user_message(test_row)test_messages.append({'role': 'user', 'content': create_user_message(test_row)})pprint(test_messages)response = openai.ChatCompletion.create( model=fine_tuned_model_id, messages=test_messages, temperature=0, max_tokens=500)print(response['choices'][0]['message']['content'])Conclusion Congratulations, you are now ready to fine-tune your own models using the ChatCompletion format! We look forward to seeing what you build


Check job status

You can make a GET request to the https://api.openai.com/v1/alpha/fine-tunes endpoint to list your alpha fine-tune jobs. In this instance you'll want to check that the ID you got from the previous step ends up as status: succeeded.Once it is completed, you can use the result_files to sample the results from the validation set (if you uploaded one), and use the ID from the fine_tuned_model parameter to invoke your trained model.


Conclusion

Congratulations, you are now ready to fine-tune your own models using the ChatCompletion format! We look forward to seeing what you build


Fine-tuning

Now we can create our fine-tuning job with the generated files and an optional suffix to identify the model. The response will contain an id which you can use to retrieve updates on the job.Note: The files have to first be processed by our system, so you might get a File not ready error. In that case, simply retry a few minutes later.


Setup

Your job has likely not been completed yet.')print('Fine-tuned model ID:', fine_tuned_model_id)


How to evaluate a summarization task

Open in GithubIn this notebook we delve into the evaluation techniques for abstractive summarization tasks using a simple example. We explore traditional evaluation methods like ROUGE and BERTScore, in addition to showcasing a more novel approach using LLMs as evaluators. Evaluating the quality of summaries is a time-consuming process, as it involves different quality metrics such as coherence, conciseness, readability and content. Traditional automatic evaluation metrics such as ROUGE and BERTScore and others are concrete and reliable, but they may not correlate well with the actual quality of summaries. They show relatively low correlation with human judgments, especially for open-ended generation tasks (Liu et al., 2023). There's a growing need to lean on human evaluations, user feedback, or model-based metrics while being vigilant about potential biases. While human judgment provides invaluable insights, it is often not scalable and can be cost-prohibitive. In addition to these traditional metrics, we showcase a method (G-Eval) that leverages Large Language Models (LLMs) as a novel, reference-free metric for assessing abstractive summaries. In this case, we use gpt-4 to score candidate outputs. gpt-4 has effectively learned an internal model of language quality that allows it to differentiate between fluent, coherent text and low-quality text. Harnessing this internal scoring mechanism allows auto-evaluation of new candidate outputs generated by an LLM.


Setup

# Installing necessary packages for the evaluation# rouge: For evaluating with ROUGE metric# bert_score: For evaluating with BERTScore# openai: To interact with OpenAI's API!pip install rouge --quiet!pip install bert_score --quiet!pip install openai --quietimport openaiimport osimport reimport pandas as pd# Python Implementation of the ROUGE Metricfrom rouge import Rouge# BERTScore leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference sentences by cosine similarity.from bert_score import BERTScoreropenai.api_key = os.environ.get('OPENAI_API_KEY')Example task For the purposes of this notebook we'll use the example summarization below.  Notice that we provide two generated summaries to compare, and a reference human-written summary, which evaluation metrics like ROUGE and BERTScore require.  Excerpt (excerpt): OpenAI's mission is to ensure that artificial general intelligence (AGI) benefits all of humanity.  OpenAI will build safe and beneficial AGI directly, but will also consider its mission fulfilled if its work aids others to achieve this outcome.  OpenAI follows several key principles for this purpose.  First, broadly distributed benefits - any influence over AGI's deployment will be used for the benefit of all, and to avoid harmful uses or undue concentration of power.  Second, long-term safety - OpenAI is committed to doing the research to make AGI safe, and to promote the adoption of such research across the AI community.  Third, technical leadership - OpenAI aims to be at the forefront of AI capabilities.  Fourth, a cooperative orientation - OpenAI actively cooperates with other research and policy institutions, and seeks to create a global community working together to address AGI's global challenges.  Summaries: Reference Summary /ref_summary (human generated)Eval Summary 1 / eval_summary_1 (system generated)Eval Summary 2 / eval_summary_2 (system generated)OpenAI aims to ensure artificial general intelligence (AGI) is used for everyone's benefit, avoiding harmful uses or undue power concentration.  It is committed to researching AGI safety, promoting such studies among the AI community.  OpenAI seeks to lead in AI capabilities and cooperates with global research and policy institutions to address AGI's challenges.OpenAI aims to AGI benefits all humanity, avoiding harmful uses and power concentration.  It pioneers research into safe and beneficial AGI and promotes adoption globally.  OpenAI maintains technical leadership in AI while cooperating with global institutions to address AGI challenges.  It seeks to lead a collaborative worldwide effort developing AGI for collective good.OpenAI aims to ensure AGI is for everyone's use, totally avoiding harmful stuff or big power concentration.  Committed to researching AGI's safe side, promoting these studies in AI folks.  OpenAI wants to be top in AI things and works with worldwide research, policy groups to figure AGI's stuff.  Take a moment to figure out which summary you'd personally prefer and the one that captures OpenAI's mission really well.excerpt = 'OpenAI's mission is to ensure that artificial general intelligence (AGI) benefits all of humanity.  OpenAI will build safe and beneficial AGI directly, but will also consider its mission fulfilled if its work aids others to achieve this outcome


Setup

OpenAI follows several key principles for this purpose .  First, broadly distributed benefits - any influence over AGI's deployment will be used for the benefit of all, and to avoid harmful uses or undue concentration of power.  Second, long-term safety - OpenAI is committed to doing the research to make AGI safe, and to promote the adoption of such research across the AI community.  Third, technical leadership - OpenAI aims to be at the forefront of AI capabilities.  Fourth, a cooperative orientation - OpenAI actively cooperates with other research and policy institutions, and seeks to create a global community working together to address AGI's global challenges.'ref_summary = 'OpenAI aims to ensure artificial general intelligence (AGI) is used for everyone's benefit, avoiding harmful uses or undue power concentration.  It is committed to researching AGI safety, promoting such studies among the AI community.  OpenAI seeks to lead in AI capabilities and cooperates with global research and policy institutions to address AGI's challenges.'eval_summary_1 = 'OpenAI aims to AGI benefits all humanity, avoiding harmful uses and power concentration.  It pioneers research into safe and beneficial AGI and promotes adoption globally.  OpenAI maintains technical leadership in AI while cooperating with global institutions to address AGI challenges.  It seeks to lead a collaborative worldwide effort developing AGI for collective good.'eval_summary_2 = 'OpenAI aims to ensure AGI is for everyone's use, totally avoiding harmful stuff or big power concentration.  Committed to researching AGI's safe side, promoting these studies in AI folks.  OpenAI wants to be top in AI things and works with worldwide research, policy groups to figure AGI's stuff.'Evaluating using ROUGE ROUGE, which stands for Recall-Oriented Understudy for Gisting Evaluation, primarily gauges the overlap of words between a generated output and a reference text.  It's a prevalent metric for evaluating automatic summarization tasks.  Among its variants, ROUGE-L offers insights into the longest contiguous match between system-generated and reference summaries, gauging how well the system retains the original summary's essence.# function to calculate the Rouge scoredef get_rouge_scores(text1, text2): rouge = Rouge() return rouge.get_scores(text1, text2)rouge_scores_out = []# Calculate the ROUGE scores for both summaries using referenceeval_1_rouge = get_rouge_scores(eval_summary_1, ref_summary)eval_2_rouge = get_rouge_scores(eval_summary_2, ref_summary)for metric in ['rouge-1', 'rouge-2', 'rouge-l']: for label in ['F-Score']: eval_1_score = eval_1_rouge[0][metric][label[0].lower()] eval_2_score = eval_2_rouge[0][metric][label[0].lower()] row = { 'Metric': f'{metric} ({label})', 'Summary 1': eval_1_score, 'Summary 2': eval_2_score, } rouge_scores_out.append(row)def highlight_max(s): is_max = s == s.max() return [ 'background-color: lightgreen' if v else 'background-color: white' for v in is_max ]rouge_scores_out = ( pd.DataFrame(rouge_scores_out) .set_index('Metric') .style.apply(highlight_max, axis=1))rouge_scores_out #T_7e6ac_row0_col0, #T_7e6ac_row1_col1, #T_7e6ac_row2_col0 { background-color: white; } #T_7e6ac_row0_col1, #T_7e6ac_row1_col0, #T_7e6ac_row2_col1 { background-color: lightgreen; } Summary 1 Summary 2 Metric rouge-1 (F-Score) 0.488889 0.511628 rouge-2 (F-Score) 0.230769 0.163265 rouge-l (F-Score) 0.488889 0.511628 The table shows the ROUGE scores for evaluating two different summaries against a reference text


Setup

Consistency: Checks if the summary aligns with the facts in the source document .  Fluency: Rates the grammar and readability of the summary.  We craft prompts for each of these criteria, taking the original document and the summary as inputs, and leveraging chain-of-thought generation and guiding the model to output a numeric score from 1-5 for each criteria.  We generate scores from gpt-4 with the defined prompts, comparing them across summaries.  In this demonstration, we're using a direct scoring function where gpt-4 generates a discrete score (1-5) for each metric.  Normalizing the scores and taking a weighted sum could result in more robust, continuous scores that better reflect the quality and diversity of the summaries.# Evaluation prompt template based on G-EvalEVALUATION_PROMPT_TEMPLATE = '''You will be given one summary written for an article.  Your task is to rate the summary on one metric.Please make sure you read and understand these instructions very carefully.  Please keep this document open while reviewing, and refer to it as needed.Evaluation Criteria:{criteria}Evaluation Steps:{steps}Example:Source Text:{document}Summary:{summary}Evaluation Form (scores ONLY):- {metric_name}'''# Metric 1: RelevanceRELEVANCY_SCORE_CRITERIA = '''Relevance(1-5) - selection of important content from the source.  \The summary should include only important information from the source document.  \Annotators were instructed to penalize summaries which contained redundancies and excess information.'''RELEVANCY_SCORE_STEPS = '''1.  Read the summary and the source document carefully.2.  Compare the summary to the source document and identify the main points of the article.3.  Assess how well the summary covers the main points of the article, and how much irrelevant or redundant information it contains.4.  Assign a relevance score from 1 to 5.'''# Metric 2: CoherenceCOHERENCE_SCORE_CRITERIA = '''Coherence(1-5) - the collective quality of all sentences.  \We align this dimension with the DUC quality question of structure and coherence \whereby 'the summary should be well-structured and well-organized.  \The summary should not just be a heap of related information, but should build from sentence to a\coherent body of information about a topic.''''COHERENCE_SCORE_STEPS = '''1.  Read the article carefully and identify the main topic and key points.2.  Read the summary and compare it to the article.  Check if the summary covers the main topic and key points of the article,and if it presents them in a clear and logical order.3.  Assign a score for coherence on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria.'''# Metric 3: ConsistencyCONSISTENCY_SCORE_CRITERIA = '''Consistency(1-5) - the factual alignment between the summary and the summarized source.  \A factually consistent summary contains only statements that are entailed by the source document.  \Annotators were also asked to penalize summaries that contained hallucinated facts.'''CONSISTENCY_SCORE_STEPS = '''1.  Read the article carefully and identify the main facts and details it presents.2.  Read the summary and compare it to the article.  Check if the summary contains any factual errors that are not supported by the article.3


Setup

In the case of rouge-1, Summary 2 outperforms Summary 1, indicating a better overlap of individual words and for rouge-l, Summary 2 has a higher score, implying a closer match in the longest common subsequences, and thus a potentially better overall summarization in capturing the main content and order of the original text .  Since Summary 2 has many words and short phrases directly lifted from the excerpt, its overlap with the reference summary would likely be higher, leading to higher ROUGE scores.  While ROUGE and similar metrics, such as BLEU and METEOR, offer quantitative measures, they often fail to capture the true essence of a well-generated summary.  They also correlate worse with human scores.  Given the advancements in LLMs, which are adept at producing fluent and coherent summaries, traditional metrics like ROUGE may inadvertently penalize these models.  This is especially true if the summaries are articulated differently but still encapsulate the core information accurately.Evaluating using BERTScore ROUGE relies on the exact presence of words in both the predicted and reference texts, failing to interpret the underlying semantics.  This is where BERTScore comes in and leverages the contextual embeddings from the BERT model, aiming to evaluate the similarity between a predicted and a reference sentence in the context of machine-generated text.  By comparing embeddings from both sentences, BERTScore captures semantic similarities that might be missed by traditional n-gram based metrics.# Instantiate the BERTScorer object for English languagescorer = BERTScorer(lang='en')# Calculate BERTScore for the summary 1 against the excerpt# P1, R1, F1_1 represent Precision, Recall, and F1 Score respectivelyP1, R1, F1_1 = scorer.score([eval_summary_1], [ref_summary])# Calculate BERTScore for summary 2 against the excerpt# P2, R2, F2_2 represent Precision, Recall, and F1 Score respectivelyP2, R2, F2_2 = scorer.score([eval_summary_2], [ref_summary])print('Summary 1 F1 Score:', F1_1.tolist()[0])print('Summary 2 F1 Score:', F2_2.tolist()[0])The close F1 Scores between the summaries indicate that they may perform similarly in capturing the key information.  However, this small difference should be interpreted with caution.  Since BERTScore may not fully grasp subtleties and high-level concepts that a human evaluator might understand, reliance solely on this metric could lead to misinterpreting the actual quality and nuances of the summary.  An integrated approach combining BERTScore with human judgment and other metrics could offer a more reliable evaluation.Evaluating using GPT-4 Here we implement an example reference-free text evaluator using gpt-4, inspired by the G-Eval framework which evaluates the quality of generated text using large language models.  Unlike metrics like ROUGE or BERTScore that rely on comparison to reference summaries, the gpt-4 based evaluator assesses the quality of generated content based solely on the input prompt and text, without any ground truth references.  This makes it applicable to new datasets and tasks where human references are sparse or unavailable.  Here's an overview of this method: We define four distinct criteria: Relevance: Evaluates if the summary includes only important information and excludes redundancies.  Coherence: Assesses the logical flow and organization of the summary


Setup

Advancing evaluation methods is crucial for driving progress in production applications .  References G-EVAL: NLG Evaluation Using GPT-4 with Better Human Alignment - Liu Y, Iter D, Xu Y, Wang S, Xu R, Zhu C.  Published May, 2023.  BERTScore: Evaluating Text Generation with BERT - Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y.  Published online February, 2020.  ROUGE: A Package for Automatic Evaluation of Summaries - Lin CY.  Published July, 2004.  SummEval: Re-evaluating Summarization Evaluation - Fabbri et al.  Published April, 2021.


Setup

Assign a score for consistency based on the Evaluation Criteria.'''# Metric 4: FluencyFLUENCY_SCORE_CRITERIA = '''Fluency(1-3): the quality of the summary in terms of grammar, spelling, punctuation, word choice, and sentence structure.1: Poor .  The summary has many errors that make it hard to understand or sound unnatural.2: Fair.  The summary has some errors that affect the clarity or smoothness of the text, but the main points are still comprehensible.3: Good.  The summary has few or no errors and is easy to read and follow.'''FLUENCY_SCORE_STEPS = '''Read the summary and evaluate its fluency based on the given criteria.  Assign a fluency score from 1 to 3.'''def get_geval_score( criteria: str, steps: str, document: str, summary: str, metric_name: str): prompt = EVALUATION_PROMPT_TEMPLATE.format( criteria=criteria, steps=steps, metric_name=metric_name, document=document, summary=summary, ) response = openai.ChatCompletion.create( model='gpt-4', messages=[{'role': 'user', 'content': prompt}], temperature=0, max_tokens=5, top_p=1, frequency_penalty=0, presence_penalty=0, ) return response.choices[0].message.contentevaluation_metrics = { 'Relevance': (RELEVANCY_SCORE_CRITERIA, RELEVANCY_SCORE_STEPS), 'Coherence': (COHERENCE_SCORE_CRITERIA, COHERENCE_SCORE_STEPS), 'Consistency': (CONSISTENCY_SCORE_CRITERIA, CONSISTENCY_SCORE_STEPS), 'Fluency': (FLUENCY_SCORE_CRITERIA, FLUENCY_SCORE_STEPS),}summaries = {'Summary 1': eval_summary_1, 'Summary 2': eval_summary_2}data = {'Evaluation Type': [], 'Summary Type': [], 'Score': []}for eval_type, (criteria, steps) in evaluation_metrics.items(): for summ_type, summary in summaries.items(): data['Evaluation Type'].append(eval_type) data['Summary Type'].append(summ_type) result = get_geval_score(criteria, steps, excerpt, summary, eval_type) score_num = int(result.strip()) data['Score'].append(score_num)pivot_df = pd.DataFrame(data, index=None).pivot( index='Evaluation Type', columns='Summary Type', values='Score')styled_pivot_df = pivot_df.style.apply(highlight_max, axis=1)display(styled_pivot_df) #T_94fab_row0_col0, #T_94fab_row1_col0, #T_94fab_row1_col1, #T_94fab_row2_col0, #T_94fab_row3_col0 { background-color: lightgreen; } #T_94fab_row0_col1, #T_94fab_row2_col1, #T_94fab_row3_col1 { background-color: white; } Summary Type Summary 1 Summary 2 Evaluation Type Coherence 5 3 Consistency 5 5 Fluency 3 2 Relevance 5 4 Overall, the Summary 1 appears to outperform Summary 2 in three of the four categories (Coherence, Relevance and Fluency).  Both summaries are found to be consistent with each other.  The result might suggest that Summary 1 is generally preferable based on the given evaluation criteria.  Limitations Note that LLM-based metrics could have a bias towards preferring LLM-generated texts over human-written texts.  Additionally LLM based metrics are sensitive to system messages/prompts.  We recommend experimenting with other techniques that can help improve performance and/or get consistent scores, striking the right balance between high-quality expensive evaluation and automated evaluations.  It is also worth noting that this scoring methodology is currently limited by gpt-4's context window.  Conclusion Evaluating abstractive summarization remains an open area for further improvement.  Traditional metrics like ROUGE, BLEU, and BERTScore provide useful automatic evaluation but have limitations in capturing semantic similarity and nuanced aspects of summarization quality.  Moreover, they require reference outputs which can be expensive to collect/label.  LLM-based metrics offer promise as a reference-free method of evaluating coherence, fluency, and relevance.  However, they too have potential biases favoring text generated by LLMs.  Ultimately, a combination of automatic metrics and human evaluation is ideal for reliably assessing abstractive summarization systems.  While human evaluation is indispensable for gaining a comprehensive understanding of summary quality, it should be complemented with automated evaluation to enable efficient, large-scale testing.  The field will continue to evolve more robust evaluation techniques, balancing quality, scalability, and fairness


Evaluating using BERTScore

ROUGE relies on the exact presence of words in both the predicted and reference texts, failing to interpret the underlying semantics. This is where BERTScore comes in and leverages the contextual embeddings from the BERT model, aiming to evaluate the similarity between a predicted and a reference sentence in the context of machine-generated text. By comparing embeddings from both sentences, BERTScore captures semantic similarities that might be missed by traditional n-gram based metrics.


Example task

For the purposes of this notebook we'll use the example summarization below. Notice that we provide two generated summaries to compare, and a reference human-written summary, which evaluation metrics like ROUGE and BERTScore require.Excerpt (excerpt): OpenAI's mission is to ensure that artificial general intelligence (AGI) benefits all of humanity. OpenAI will build safe and beneficial AGI directly, but will also consider its mission fulfilled if its work aids others to achieve this outcome. OpenAI follows several key principles for this purpose. First, broadly distributed benefits - any influence over AGI's deployment will be used for the benefit of all, and to avoid harmful uses or undue concentration of power. Second, long-term safety - OpenAI is committed to doing the research to make AGI safe, and to promote the adoption of such research across the AI community. Third, technical leadership - OpenAI aims to be at the forefront of AI capabilities. Fourth, a cooperative orientation - OpenAI actively cooperates with other research and policy institutions, and seeks to create a global community working together to address AGI's global challenges. Summaries:Take a moment to figure out which summary you'd personally prefer and the one that captures OpenAI's mission really well.


Evaluating using ROUGE

ROUGE, which stands for Recall-Oriented Understudy for Gisting Evaluation, primarily gauges the overlap of words between a generated output and a reference text. It's a prevalent metric for evaluating automatic summarization tasks. Among its variants, ROUGE-L offers insights into the longest contiguous match between system-generated and reference summaries, gauging how well the system retains the original summary's essence.


Limitations

Note that LLM-based metrics could have a bias towards preferring LLM-generated texts over human-written texts. Additionally LLM based metrics are sensitive to system messages/prompts. We recommend experimenting with other techniques that can help improve performance and/or get consistent scores, striking the right balance between high-quality expensive evaluation and automated evaluations. It is also worth noting that this scoring methodology is currently limited by gpt-4's context window.


Evaluating using GPT-4

Here we implement an example reference-free text evaluator using gpt-4, inspired by the G-Eval framework which evaluates the quality of generated text using large language models. Unlike metrics like ROUGE or BERTScore that rely on comparison to reference summaries, the gpt-4 based evaluator assesses the quality of generated content based solely on the input prompt and text, without any ground truth references. This makes it applicable to new datasets and tasks where human references are sparse or unavailable.Here's an overview of this method: We define four distinct criteria: Relevance: Evaluates if the summary includes only important information and excludes redundancies. Coherence: Assesses the logical flow and organization of the summary. Consistency: Checks if the summary aligns with the facts in the source document. Fluency: Rates the grammar and readability of the summary. We craft prompts for each of these criteria, taking the original document and the summary as inputs, and leveraging chain-of-thought generation and guiding the model to output a numeric score from 1-5 for each criteria. We generate scores from gpt-4 with the defined prompts, comparing them across summaries. In this demonstration, we're using a direct scoring function where gpt-4 generates a discrete score (1-5) for each metric. Normalizing the scores and taking a weighted sum could result in more robust, continuous scores that better reflect the quality and diversity of the summaries.


References

G-EVAL: NLG Evaluation Using GPT-4 with Better Human Alignment - Liu Y, Iter D, Xu Y, Wang S, Xu R, Zhu C. Published May, 2023. BERTScore: Evaluating Text Generation with BERT - Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y. Published online February, 2020. ROUGE: A Package for Automatic Evaluation of Summaries - Lin CY. Published July, 2004. SummEval: Re-evaluating Summarization Evaluation - Fabbri et al. Published April, 2021.


Conclusion

Evaluating abstractive summarization remains an open area for further improvement. Traditional metrics like ROUGE, BLEU, and BERTScore provide useful automatic evaluation but have limitations in capturing semantic similarity and nuanced aspects of summarization quality. Moreover, they require reference outputs which can be expensive to collect/label. LLM-based metrics offer promise as a reference-free method of evaluating coherence, fluency, and relevance. However, they too have potential biases favoring text generated by LLMs. Ultimately, a combination of automatic metrics and human evaluation is ideal for reliably assessing abstractive summarization systems. While human evaluation is indispensable for gaining a comprehensive understanding of summary quality, it should be complemented with automated evaluation to enable efficient, large-scale testing. The field will continue to evolve more robust evaluation techniques, balancing quality, scalability, and fairness. Advancing evaluation methods is crucial for driving progress in production applications.


Function-calling with an OpenAPI specification

Open in GithubMuch of the internet is powered by RESTful APIs. Giving GPT the ability to call them opens up a world of possibilities. This notebook demonstrates how GPTs can be used to intelligently call APIs. It leverages OpenAPI specifications and chained function calls. The OpenAPI Specification (OAS) is a universally accepted standard for describing the details of RESTful APIs in a format that machines can read and interpret. It enables both humans and computers to understand the capabilities of a service, and it can be leveraged to show GPT how to call APIs. This notebook is divided into two main sections: How to convert a sample OpenAPI specification into a list of function definitions for the chat completions API. How to use the chat completions API to intelligently invoke these functions based on user instructions. We recommend familiariazing yourself with function-calling before proceding.!pip install -q jsonref # for resolving $ref's in the OpenAPI spec!pip install -q openaiimport osimport jsonimport jsonrefimport openaiimport requestsfrom pprint import ppopenai.api_key = os.environ['OPENAI_API_KEY']


How to convert an OpenAPI specification into function definitions

The example OpenAPI spec we use here was created using gpt-4.  We will transform this sample spec into a set of function definitions that can be supplied to the chat completion API.  The model, based on the provided user instructions, generates a JSON object containing the necessary arguments to call these functions.  Before we proceed, let's inspect this generated spec.  OpenAPI specs include details about the API's endpoints, the operations they support, the parameters they accept, the requests they can handle, and the responses they return.  The spec is defined in JSON format.  The endpoints in the spec include operations for: Listing all events Creating a new event Retrieving an event by ID Deleting an event by ID Updating an event name by ID Each operation in the spec has an operationId, which we will use as the function name when we parse the spec into function specifications.  The spec also includes schemas that define the data types and structures of the parameters for each operation.  You can see the schema here:with open('./data/example_events_openapi.json', 'r') as f: openapi_spec = jsonref.loads(f.read()) # it's important to load with jsonref, as explained belowdisplay(openapi_spec)Now that we have a good understanding of the OpenAPI spec, we can proceed to parse it into function specifications.  We can write a simple openapi_to_functions function to generate a list of definitions, where each function is represented as a dictionary containing the following keys: name: This corresponds to the operation identifier of the API endpoint as defined in the OpenAPI specification.  description: This is a brief description or summary of the function, providing an overview of what the function does.  parameters: This is a schema that defines the expected input parameters for the function.  It provides information about the type of each parameter, whether it is required or optional, and other related details.  For each of the endpoints defined in the schema, we need to do the following: Resolve JSON references: In an OpenAPI specification, it's common to use JSON references (also known as $ref) to avoid duplication.  These references point to definitions that are used in multiple places.  For example, if multiple API endpoints return the same object structure, that structure can be defined once and then referenced wherever it's needed.  We need to resolve and replace these references with the content they point to.  Extract a name for the functions: We will simply use the operationId as the function name.  Alternatively, we could use the endpoint path and operation as the function name.  Extract a description and parameters: We will iterate through the description, summary, requestBody and parameters fields to populate the function's description and parameters.  Here's the implementation:def openapi_to_functions(openapi_spec): functions = [] for path, methods in openapi_spec['paths'].items(): for method, spec_with_ref in methods.items(): # 1.  Resolve JSON references.  spec = jsonref.replace_refs(spec_with_ref) # 2.  Extract a name for the functions.  function_name = spec.get('operationId') # 3


Extract a description and parameters .  desc = spec.get('description') or spec.get('summary', '') schema = {'type': 'object', 'properties': {}} req_body = ( spec.get('requestBody', {}) .get('content', {}) .get('application/json', {}) .get('schema') ) if req_body: schema['properties']['requestBody'] = req_body params = spec.get('parameters', []) if params: param_properties = { param['name']: param['schema'] for param in params if 'schema' in param } schema['properties']['parameters'] = { 'type': 'object', 'properties': param_properties, } functions.append( {'name': function_name, 'description': desc, 'parameters': schema} ) return functionsfunctions = openapi_to_functions(openapi_spec)for function in functions: pp(function) print()


How to call these functions with GPT

Now that we have these function definitions, we can leverage GPT to call them intelligently based on user inputs. It's important to note that the chat completions API does not execute the function; instead, it generates the JSON that you can use to call the function in your own code. For more information on function-calling, refer to our dedicated function-calling guide.SYSTEM_MESSAGE = '''You are a helpful assistant. Respond to the following prompt by using function_call and then summarize actions. Ask for clarification if a user request is ambiguous.'''# Maximum number of function calls allowed to prevent infinite or lengthy loopsMAX_CALLS = 5def get_openai_response(functions, messages): return openai.ChatCompletion.create( model='gpt-3.5-turbo-16k-0613', functions=functions, function_call='auto', # 'auto' means the model can pick between generating a message or calling a function. temperature=0, messages=messages, )def process_user_instruction(functions, instruction): num_calls = 0 messages = [ {'content': SYSTEM_MESSAGE, 'role': 'system'}, {'content': instruction, 'role': 'user'}, ] while num_calls < MAX_CALLS: response = get_openai_response(functions, messages) message = response['choices'][0]['message'] if message.get('function_call'): print(f'\n>> Function call #: {num_calls + 1}\n') pp(message['function_call']) messages.append(message) # For the sake of this example, we'll simply add a message to simulate success. # Normally, you'd want to call the function here, and append the results to messages. messages.append( { 'role': 'function', 'content': 'success', 'name': message['function_call']['name'], } ) num_calls += 1 else: print('\n>> Message:\n') print(message['content']) break if num_calls >= MAX_CALLS: print(f'Reached max chained function calls: {MAX_CALLS}')USER_INSTRUCTION = '''Instruction: Get all the events. Then create a new event named AGI Party.Then delete event with id 2456.'''process_user_instruction(functions, USER_INSTRUCTION)Conclusion We have demonstrated how to convert OpenAPI specs into function specifications that can be given to GPT for it to intelligently call them, and shown how these can be chained together to perform complex operations. Possible extensions of this system could include handling more complex user instructions that require conditional logic or looping, integrating with real APIs to perform actual operations, and improving error handling and validation to ensure the instructions are feasible and the function calls are successful.


Conclusion

We have demonstrated how to convert OpenAPI specs into function specifications that can be given to GPT for it to intelligently call them, and shown how these can be chained together to perform complex operations.Possible extensions of this system could include handling more complex user instructions that require conditional logic or looping, integrating with real APIs to perform actual operations, and improving error handling and validation to ensure the instructions are feasible and the function calls are successful.


Fine-tuning GPT with Weights & Biases

Open in Github Note: you will need an OpenAI API key to run this colab.If you use OpenAI's API to fine-tune ChatGPT-3.5, you can now use the W&B integration to track experiments, models, and datasets in your central dashboard. All it takes is one line: openai wandb sync See the OpenAI section in the Weights & Biases documentation for full details of the integration!pip install -Uq openai tiktoken datasets tenacity wandb# Remove once this PR is merged: https://github.com/openai/openai-python/pull/590 and openai release is made!pip uninstall -y openai -qq \&& pip install git+https://github.com/morganmcg1/openai-python.git@update_wandb_logger -qqqOptional: Fine-tune ChatGPT-3.5 It's always more fun to experiment with your own projects so if you have already used the openai API to fine-tune an OpenAI model, just skip this section. Otherwise let's fine-tune ChatGPT-3.5 on a legal dataset!


Optional: Fine-tune ChatGPT-3.5

It's always more fun to experiment with your own projects so if you have already used the openai API to fine-tune an OpenAI model, just skip this section.Otherwise let's fine-tune ChatGPT-3.5 on a legal dataset!


Imports and initial set-up

import openaiimport wandbimport osimport jsonimport randomimport tiktokenimport numpy as npimport pandas as pdfrom pathlib import Pathfrom tqdm.auto import tqdmfrom collections import defaultdictfrom tenacity import retry, stop_after_attempt, wait_fixedStart your Weigths & Biases run. If you don't have an account you can sign up for one for free at www.wandb.aiWANDB_PROJECT = 'OpenAI-Fine-Tune'


Set up your API key

# Token counting functions encoding = tiktoken.get_encoding('cl100k_base') # not exact! # simplified from https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb def num_tokens_from_messages(messages, tokens_per_message=3, tokens_per_name=1): num_tokens = 0 for message in messages: num_tokens += tokens_per_message for key, value in message.items(): num_tokens += len(encoding.encode(value)) if key == 'name': num_tokens += tokens_per_name num_tokens += 3 return num_tokens def num_assistant_tokens_from_messages(messages): num_tokens = 0 for message in messages: if message['role'] == 'assistant': num_tokens += len(encoding.encode(message['content'])) return num_tokens def print_distribution(values, name): print(f'\n#### Distribution of {name}:') print(f'min / max: {min(values)}, {max(values)}') print(f'mean / median: {np.mean(values)}, {np.median(values)}') print(f'p5 / p95: {np.quantile(values, 0.1)}, {np.quantile(values, 0.9)}') # Last, we can look at the results of the different formatting operations before proceeding with creating a fine-tuning job: # Warnings and tokens counts n_missing_system = 0 n_missing_user = 0 n_messages = [] convo_lens = [] assistant_message_lens = [] for ex in dataset: messages = ex['messages'] if not any(message['role'] == 'system' for message in messages): n_missing_system += 1 if not any(message['role'] == 'user' for message in messages): n_missing_user += 1 n_messages.append(len(messages)) convo_lens.append(num_tokens_from_messages(messages)) assistant_message_lens.append(num_assistant_tokens_from_messages(messages)) print('Num examples missing system message:', n_missing_system) print('Num examples missing user message:', n_missing_user) print_distribution(n_messages, 'num_messages_per_example') print_distribution(convo_lens, 'num_total_tokens_per_example') print_distribution(assistant_message_lens, 'num_assistant_tokens_per_example') n_too_long = sum(l > 4096 for l in convo_lens) print(f'\n{n_too_long} examples may be over the 4096 token limit, they will be truncated during fine-tuning') # Pricing and default n_epochs estimate MAX_TOKENS_PER_EXAMPLE = 4096 MIN_TARGET_EXAMPLES = 100 MAX_TARGET_EXAMPLES = 25000 TARGET_EPOCHS = 3 MIN_EPOCHS = 1 MAX_EPOCHS = 25 n_epochs = TARGET_EPOCHS n_train_examples = len(dataset) if n_train_examples * TARGET_EPOCHS < MIN_TARGET_EXAMPLES: n_epochs = min(MAX_EPOCHS, MIN_TARGET_EXAMPLES // n_train_examples) elif n_train_examples * TARGET_EPOCHS > MAX_TARGET_EXAMPLES: n_epochs = max(MIN_EPOCHS, MAX_TARGET_EXAMPLES // n_train_examples) n_billing_tokens_in_dataset = sum(min(MAX_TOKENS_PER_EXAMPLE, length) for length in convo_lens) print(f'Dataset has ~{n_billing_tokens_in_dataset} tokens that will be charged for during training') print(f'By default, you'll train for {n_epochs} epochs on this dataset') print(f'By default, you'll be charged for ~{n_epochs * n_billing_tokens_in_dataset} tokens') print('See pricing page to estimate total costs')Validate train dataopenai_validate_data(train_file_path)Log our data to Weigths & Biases Artifacts for storage and versioningwandb.init( project=WANDB_PROJECT, # entity='prompt-eng', job_type='log-data', config = {'n_train': n_train, 'n_valid': n_test})wandb.log_artifact(train_file_path, 'legalbench-contract_nli_explicit_identification-train', type='train-data')wandb.log_artifact(test_file_path, 'legalbench-contract_nli_explicit_identification-test', type='test-data')# keep entity (typically your wandb username) for reference of artifact later in this demoentity = wandb.run.entitywandb.finish()mild-surf-1Weights & Biasesdocshttps://wandb.ai/capecape/OpenAI-Fine-Tunehttps://wandb.ai/capecape/OpenAI-Fine-Tune/runs/ivu21mjl(success).mild-surf-1https://wandb.ai/capecape/OpenAI-Fine-Tune/runs/ivu21mjl


Set up your API key

# # Enter credentialsopenai_key = 'YOUR_API_KEY'openai.api_key = openai_keyDataset Preparation We download a dataset from LegalBench, a project to curate tasks for evaluating legal reasoning, specifically the Contract NLI Explicit Identification task.  This comprises of a total of 117 examples, from which we will create our own train and test datasetsfrom datasets import load_dataset# Download the data, merge into a single dataset and shuffledataset = load_dataset('nguha/legalbench', 'contract_nli_explicit_identification')data = []for d in dataset['train']: data.append(d)for d in dataset['test']: data.append(d)random.shuffle(data)for idx, d in enumerate(data): d['new_index'] = idxLet's look at a few samples.len(data), data[0:2]Format our Data for Chat Completion Models We modify the base_prompt from the LegalBench task to make it a zero-shot prompt, as we are training the model instead of using few-shot promptingbase_prompt_zero_shot = 'Identify if the clause provides that all Confidential Information shall be expressly identified by the Disclosing Party.  Answer with only `Yes` or `No`'We now split it into training/validation dataset, lets train on 30 samples and test on the remaindern_train = 30n_test = len(data) - n_traintrain_messages = []test_messages = []for d in data: prompts = [] prompts.append({'role': 'system', 'content': base_prompt_zero_shot}) prompts.append({'role': 'user', 'content': d['text']}) prompts.append({'role': 'assistant', 'content': d['answer']}) if int(d['new_index']) < n_train: train_messages.append({'messages': prompts}) else: test_messages.append({'messages': prompts})len(train_messages), len(test_messages), n_test, train_messages[5]Save the data to Weigths & Biases Save the data in a train and test file firsttrain_file_path = 'encoded_train_data.jsonl'with open(train_file_path, 'w') as file: for item in train_messages: line = json.dumps(item) file.write(line + '\n')test_file_path = 'encoded_test_data.jsonl'with open(test_file_path, 'w') as file: for item in test_messages: line = json.dumps(item) file.write(line + '\n')Next, we validate that our training data is in the correct format using a script from the OpenAI fine-tuning documentation# Next, we specify the data path and open the JSONL filedef openai_validate_data(dataset_path): data_path = dataset_path # Load dataset with open(data_path) as f: dataset = [json.loads(line) for line in f] # We can inspect the data quickly by checking the number of examples and the first item # Initial dataset stats print('Num examples:', len(dataset)) print('First example:') for message in dataset[0]['messages']: print(message) # Now that we have a sense of the data, we need to go through all the different examples and check to make sure the formatting is correct and matches the Chat completions message structure # Format error checks format_errors = defaultdict(int) for ex in dataset: if not isinstance(ex, dict): format_errors['data_type'] += 1 continue messages = ex.get('messages', None) if not messages: format_errors['missing_messages_list'] += 1 continue for message in messages: if 'role' not in message or 'content' not in message: format_errors['message_missing_key'] += 1 if any(k not in ('role', 'content', 'name') for k in message): format_errors['message_unrecognized_key'] += 1 if message.get('role', None) not in ('system', 'user', 'assistant'): format_errors['unrecognized_role'] += 1 content = message.get('content', None) if not content or not isinstance(content, str): format_errors['missing_content'] += 1 if not any(message.get('role', None) == 'assistant' for message in messages): format_errors['example_missing_assistant_message'] += 1 if format_errors: print('Found errors:') for k, v in format_errors.items(): print(f'{k}: {v}') else: print('No errors found') # Beyond the structure of the message, we also need to ensure that the length does not exceed the 4096 token limit


Save the data to Weigths & Biases

Save the data in a train and test file first


Format our Data for Chat Completion Models

We modify the base_prompt from the LegalBench task to make it a zero-shot prompt, as we are training the model instead of using few-shot prompting


Create a fine-tuned model

We'll now use OpenAI API to fine-tune ChatGPT-3.5 Let's first download our training & validation files and save them to a folder called my_data. We will retrieve the latest version of the artifact, but it could also be v0, v1 or any alias we associated with itwandb.init(project=WANDB_PROJECT, # entity='prompt-eng', job_type='finetune')artifact_train = wandb.use_artifact( f'{entity}/{WANDB_PROJECT}/legalbench-contract_nli_explicit_identification-train:latest', type='train-data')train_file = artifact_train.get_path(train_file_path).download('my_data')train_filejumping-water-2Weights & Biasesdocshttps://wandb.ai/capecape/OpenAI-Fine-Tunehttps://wandb.ai/capecape/OpenAI-Fine-Tune/runs/1ili9l51Then we upload the training data to OpenAI. OpenAi has to process the data, so this will take a few minutes depending on the size of your dataset.openai_train_file_info = openai.File.create( file=open(train_file, 'rb'), purpose='fine-tune')# you may need to wait a couple of minutes for OpenAI to process the fileopenai_train_file_info


Log OpenAI fine-tune jobs to Weights & Biases

We can log our fine-tunes with a simple command.!openai wandb sync --helpCalling openai wandb sync will log all un-synced fine-tuned jobs to W&B Below we are just logging 1 job, passing: our OpenAI key as an environment variable the id of the fine-tune job we'd like to log the W&B project of where to log it to See the OpenAI section in the Weights & Biases documentation for full details of the integration!OPENAI_API_KEY={openai_key} openai wandb sync --id {ft_job_id} --project {WANDB_PROJECT}wandb.finish()(success).jumping-water-2https://wandb.ai/capecape/OpenAI-Fine-Tune/runs/1ili9l51Our fine-tunes are now successfully synced to Weights & Biases. Anytime we have new fine-tunes, we can just call openai wandb sync to add them to our dashboard.


Time to train the model!

Let's define our ChatGPT-3.5 fine-tuning hyper-parameters.model = 'gpt-3.5-turbo'n_epochs = 3openai_ft_job_info = openai.FineTuningJob.create( training_file=openai_train_file_info['id'], model=model, hyperparameters={'n_epochs': n_epochs})ft_job_id = openai_ft_job_info['id']openai_ft_job_info this takes around 5 minutes to train, and you get an email from OpenAI when finished. Thats it! Now your model is training on OpenAI's machines. To get the current state of your fine-tuning job, run:state = openai.FineTuningJob.retrieve(ft_job_id)state['status'], state['trained_tokens'], state['finished_at'], state['fine_tuned_model']Show recent events for our fine-tuning jobopenai.FineTuningJob.list_events(id=ft_job_id, limit=5)We can run a few different fine-tunes with different parameters or even with different datasets.


Run evalution and log the results

The best way to evaluate a generative model is to explore sample predictions from your evaluation set. Let's generate a few inference samples and log them to W&B and see how the performance compares to a baseline ChatGPT-3.5 modelwandb.init(project=WANDB_PROJECT, job_type='eval')artifact_valid = wandb.use_artifact( f'{entity}/{WANDB_PROJECT}/legalbench-contract_nli_explicit_identification-test:latest', type='test-data')test_file = artifact_valid.get_path(test_file_path).download('my_data')with open(test_file) as f: test_dataset = [json.loads(line) for line in f]print(f'There are {len(test_dataset)} test examples')wandb.config.update({'num_test_samples':len(test_dataset)})ethereal-energy-4Weights & Biasesdocshttps://wandb.ai/capecape/OpenAI-Fine-Tunehttps://wandb.ai/capecape/OpenAI-Fine-Tune/runs/iepk19m2Run evaluation on the Fine-Tuned Model Set up OpenAI call with retries@retry(stop=stop_after_attempt(3), wait=wait_fixed(60))def call_openai(messages='', model='gpt-3.5-turbo'): return openai.ChatCompletion.create(model=model, messages=messages, max_tokens=10)Let's get our trained model idstate = openai.FineTuningJob.retrieve(ft_job_id)ft_model_id = state['fine_tuned_model']ft_model_idRun evaluation and log results to W&Bprediction_table = wandb.Table(columns=['messages', 'completion', 'target'])eval_data = []for row in tqdm(test_dataset): messages = row['messages'][:2] target = row['messages'][2] # res = call_openai(model=ft_model_id, messages=messages) res = openai.ChatCompletion.create(model=model, messages=messages, max_tokens=10) completion = res.choices[0].message.content eval_data.append([messages, completion, target]) prediction_table.add_data(messages[1]['content'], completion, target['content'])wandb.log({'predictions': prediction_table})Calculate the accuracy of the fine-tuned model and log to W&Bcorrect = 0for e in eval_data: if e[1].lower() == e[2]['content'].lower(): correct+=1accuracy = correct / len(eval_data)print(f'Accuracy is {accuracy}')wandb.log({'eval/accuracy': accuracy})wandb.summary['eval/accuracy'] = accuracyRun evaluation on a Baseline model for comparison Lets compare our model to the baseline model, gpt-3.5-turbobaseline_prediction_table = wandb.Table(columns=['messages', 'completion', 'target'])baseline_eval_data = []for row in tqdm(test_dataset): messages = row['messages'][:2] target = row['messages'][2] res = call_openai(model='gpt-3.5-turbo', messages=messages) completion = res.choices[0].message.content baseline_eval_data.append([messages, completion, target]) baseline_prediction_table.add_data(messages[1]['content'], completion, target['content'])wandb.log({'baseline_predictions': baseline_prediction_table})Calculate the accuracy of the fine-tuned model and log to W&Bbaseline_correct = 0for e in baseline_eval_data: if e[1].lower() == e[2]['content'].lower(): baseline_correct+=1baseline_accuracy = baseline_correct / len(baseline_eval_data)print(f'Baseline Accurcy is: {baseline_accuracy}')wandb.log({'eval/baseline_accuracy': baseline_accuracy})wandb.summary['eval/baseline_accuracy'] = baseline_accuracywandb.finish()(success). table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%} .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% } .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }


Run evaluation on a Baseline model for comparison

Lets compare our model to the baseline model, gpt-3.5-turbo


Resources

OpenAI Fine-Tuning Guide W&B Integration with OpenAI API Documentation W&B Report: GPT-3 exploration & fine-tuning tips


Run summary:

ethereal-energy-4https://wandb.ai/capecape/OpenAI-Fine-Tune/runs/iepk19m2And thats it! In this example we have prepared our data, logged it to Weights & Biases, fine-tuned an OpenAI model using that data, logged the results to Weights & Biases and then run evaluation on the fine-tuned model. From here you can start to train on larger or more complex tasks, or else explore other ways to modify ChatGPT-3.5 such as giving it a different tone and style or response.Resources OpenAI Fine-Tuning Guide W&B Integration with OpenAI API Documentation W&B Report: GPT-3 exploration & fine-tuning tips


Run evaluation on the Fine-Tuned Model

Set up OpenAI call with retries


Dataset Preparation

We download a dataset from LegalBench, a project to curate tasks for evaluating legal reasoning, specifically the Contract NLI Explicit Identification task.This comprises of a total of 117 examples, from which we will create our own train and test datasets


OpenAI API Monitoring with Weights & Biases Weave

To visualize your work in real-time as you iterate, you can: keep the Board open in a separate tab and refresh to view the latest data rename the Board for easier reference at any point and 'Publish' that version to share a link with others find previously saved Boards by navigating to the relevant W&B entity and W&B project name from weave.wandb.ai or open a new instance of a Board template to start fresh with all the data accumulated so far Next we'll illustrate a few ways you could track OpenAI API calls .  There are many more possibilities depending on your use case, and we can't wait to see what you create from these starter templates.  Examples Example 0: Log a prompt and its completion Monitor a ChatCompletion request and print the corresponding response, extracting only the text of the completion.response = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[ {'role': 'user', 'content': f'What is the meaning of life, the universe, and everything?'}, ])print(response['choices'][0]['message']['content'])Example 1: Track relevant parameters as attributes Factor out parameters of interest and track them as attributes on the logged record.  Here we track the 'system prompt' separately from the 'prompt template' and the 'equation' parameter.  This time we'll print the full structured response from the ChatCompletion call.system_prompt = 'you always write in bullet points'prompt_template = 'solve the following equation step by step: {equation}'params = {'equation': '4 * (3 - 1)'}openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[ {'role': 'system', 'content': system_prompt}, {'role': 'user', 'content': prompt_template.format(**params)}, ], # you can add additional attributes to the logged record # see the monitor_api notebook for more examples monitor_attributes={ 'system_prompt': system_prompt, 'prompt_template': prompt_template, 'params': params })Example 2: Log an ongoing stream of messages Monitor a stream of messages and log the result as a single record.  Note: tokens are not counted in this format.from weave.monitoring.openai import message_from_streamr = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[ {'role': 'system', 'content': 'You are a robot and only speak in robot, like beep bloop bop.'}, {'role': 'user', 'content': 'Tell me a 50-word story.'}, ], stream=True)for s in message_from_stream(r): print(s, end='')Example 3: Structure prompt engineering experiments Here we compare a few toy options for the system prompt, user question, and intended audience.  Try your own experiments and see if any interesting insights emerge as you explore in the Board and group by different parameters.def explain_math(system_prompt, prompt_template, params): openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[ {'role': 'system', 'content': system_prompt}, {'role': 'user', 'content': prompt_template.format(**params)}, ], # you can add additional attributes to the logged record # see the monitor_api notebook for more examples monitor_attributes={ 'system_prompt': system_prompt, 'prompt_template': prompt_template, 'params': params })# feel free to substitute your own prompts :)system_prompts = ['you're extremely flowery and poetic', 'you're very direct and precise', 'balance brevity with insight']prompt_template = 'explain the solution of the following to a {audience}: {equation}'equations = ['x^2 + 4x + 9 = 0', '15 * (2 - 6) / 4']audience = ['new student', 'math genius']for system_prompt in system_prompts: for equation in equations: for person in audience: params = {'equation' : equation, 'audience' : person} explain_math(system_prompt, prompt_template, params)


Open in GithubNote: you will need an OpenAI API key to run this colab.  Use the W&B OpenAI integration to monitor OpenAI API calls and understand how your projects and teams are leveraging LLMs.  In this example, we'll generate templated Weave Boards: LLM usage monitoring dashboards which you can explore and customize from the UI.  automatically track LLM usage and aggregate useful metrics like cost, latency and throughput across your projects/teams dynamically query and derive insights from the logs of all your OpenAI API calls iterate visually to slice, aggregate, and explore your data; customize panels to focus on interesting patterns; share progress more easily with your team through an interactive dashboard Play with a live version of this Weave Board → New to Weights & Biases? -> Sign up for an account here <- Step 0: Setup Install dependencies, login to W&B so you can save and share your work, and authenticate with OpenAI.# if not already installed!pip install -qqq weave openai tiktoken wandbimport wandbwandb.login()import weaveimport osWANDB_BASE_URL = 'https://api.wandb.ai'os.environ['WANDB_BASE_URL'] = WANDB_BASE_URL# authenticate with OpenAIfrom getpass import getpassif os.getenv('OPENAI_API_KEY') is None: os.environ['OPENAI_API_KEY'] = getpass('Paste your OpenAI key from: https://platform.openai.com/account/api-keys\n')assert os.getenv('OPENAI_API_KEY', '').startswith('sk-'), 'This doesn't look like a valid OpenAI API key'print('OpenAI API key configured')Step 1: Configure data streaming and storage in W&B Set WB_ENTITY to your wandb username or team name.  Log in to W&B and navigate to Home Page at wandb.ai/home to see valid options under your 'Profile' and 'Teams' in the left sidebar.WB_ENTITY = # set to your wandb username or team nameWB_PROJECT = 'weave' # top-level directory for this workSTREAM_NAME = 'openai_logs' # record table which stores the logs of OpenAI API calls as they stream inStep 2: Call init_monitor() To start monitoring OpenAI API usage, call init_monitor(<stream>), where <stream> has the form <wandb_team_or_user>/<wandb_project>/<stream_name>.  The stream records and stores all the OpenAI API calls.  Running this cell will print out a link to view the current project in the Weave UI.from weave.monitoring import openai, init_monitorm = init_monitor(f'{WB_ENTITY}/{WB_PROJECT}/{STREAM_NAME}')# specifying a single model for simplicityOPENAI_MODEL = 'gpt-3.5-turbo'# prefill with some sample logsr = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[{'role': 'user', 'content': 'hello world!'}])r = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[{'role': 'user', 'content': 'what is 2+2?'}])Step 3: Preview monitoring dashboard Click on the link above to preview the data stream, then click 'OpenAI Monitor Board' in the right sidebar to create a Weave Board for this data stream.  Step 4: Explore & understand your LLM usage To save your work, rename the board by clicking on the autogenerated name at the top of the page.  To share your board, click 'Publish' in the top right


Step 0: Setup

Install dependencies, login to W&B so you can save and share your work, and authenticate with OpenAI.


Step 1: Configure data streaming and storage in W&B

Set WB_ENTITY to your wandb username or team name. Log in to W&B and navigate to Home Page at wandb.ai/home to see valid options under your 'Profile' and 'Teams' in the left sidebar.


Step 2: Call init_monitor()

To start monitoring OpenAI API usage, call init_monitor(<stream>), where <stream> has the form <wandb_team_or_user>/<wandb_project>/<stream_name>. The stream records and stores all the OpenAI API calls.Running this cell will print out a link to view the current project in the Weave UI.


Step 3: Preview monitoring dashboard

Click on the link above to preview the data stream, then click 'OpenAI Monitor Board' in the right sidebar to create a Weave Board for this data stream.


Step 4: Explore & understand your LLM usage

To save your work, rename the board by clicking on the autogenerated name at the top of the page. To share your board, click 'Publish' in the top right.To visualize your work in real-time as you iterate, you can: keep the Board open in a separate tab and refresh to view the latest data rename the Board for easier reference at any point and 'Publish' that version to share a link with others find previously saved Boards by navigating to the relevant W&B entity and W&B project name from weave.wandb.ai or open a new instance of a Board template to start fresh with all the data accumulated so far Next we'll illustrate a few ways you could track OpenAI API calls. There are many more possibilities depending on your use case, and we can't wait to see what you create from these starter templates.


Example 0: Log a prompt and its completion

Monitor a ChatCompletion request and print the corresponding response, extracting only the text of the completion.


Example 2: Log an ongoing stream of messages

Monitor a stream of messages and log the result as a single record. Note: tokens are not counted in this format.


Example 1: Track relevant parameters as attributes

Factor out parameters of interest and track them as attributes on the logged record. Here we track the 'system prompt' separately from the 'prompt template' and the 'equation' parameter. This time we'll print the full structured response from the ChatCompletion call.