Exploring Large Language Models -Part 1

LLM Training and the Emergence of Understanding We can understand this topic in a more fun way by following the history of some famous AI/ML systems. Rule Engine/Tree Search We can start with IBM DeepBlue the chess-playing supercomputer. By using the power of a Super Computer and custom chips it defeated Gary Kasparov - the chess grandmaster in 1997. However, there was no AI or NeuralNet involved then, it was TreeSearch. You could abstract this a bit and say that it was a Rule-based engine. The training data was hand-coded domain expertise, distilled into a set of rules. The algorithm optimized the choice of the next move from a vast but computable result set based on the current state of its world. It was clear, however, that it was impossible to do this rule-based programming for broader domains. Supervised Training A decade later in 2011, IBM Watson -designed for Question Answering and specifically trained for Trivia QA played Jeopardy and won against a champion. There was a big hype around this as the next knowledge system that would revolutionise everything. The primary way the system was trained seems to be Supervised Training. That is data with labels on which the system was trained to pick the right answer or in this case, the right question. Supervised Learning works well. This is the bedrock of almost all machine learning used in production today. Given enough labeled data, AI/ML systems will learn to approximate any complex multivariate functions. They are excellent universal function approximators. The problem with Supervised Learning is that labelling huge amount of datasets for training huge models, need a lot of expensive and time-consuming human effort. The best example of one of the largest labeled datasets and its implication is Imagnet. It was the huge amount of labeled image data collected as part of the ImageNet project that helped AlexNet introduced in 2012 (Ilya Sutskever, Alex Krizhevsky and Geoffrey Hinton) revolutionize computer vision, though even by 1998 Yann LeCun and others and others had introduced LeNet-based convolutional neural network for handwriting recognition. Note Ilya Sutskever is also one of the founders of OpenAI and was instrumental in training the GPT models later. Back to the story. The hype surrounding IBM Watson died down over the years as the limitations of the system became apparent. A NY Times article gives insight into some of the challenges of why it could not generalize well to other fields like IBM hoped, the lack of properly labeled data being the primary. Reinforcement Learning In 2016, Google DeepMind AlphaGo became very popular by defeating the champion Go player. This game is a much wider domain /strategy than Chess (impossible for rule engine/tree search type algorithms). The key here was Reinforcement learning (RL). Here the training can be abstracted to making a random move, and if the move takes you closer to winning (though some loss calculation) make more of such moves and vice versa. And then they created agents and pitted the agents against each other, thereby playing probably thousands of years worth of games and getting good at the game. A more complicated game than Go is Dota, and in 2019, a small company (at that time) called OpenAI defeated the current reigning Dota 2 champions Here is an interesting snippet related to this event. The last word 'scale' is an interesting part that may suggest how they used this concept for their future work in GPT models DeepDive - Tough RL looks like it can be used for everything (for example, this is how a baby learns to walk or most organisms learn), outside of domains like games which have a very limited or controlled state space, it is very difficult to implement. For example, in a self-driving car, any small state change in some previous step can contribute later positively or negatively. It boils down to implementing a loss function that can store and work temporarily. (https://stanford.edu/~ashlearn/RLForFinanceBook/book.pdf, Why backpropagation approximation is needed for RL https://stats.stackexchange.com/a/340657/191675) So we covered supervised learning ( the bread and butter of AI/ML algorithms so far) and Reinforcement Learning - mostly in Video games and similar. Unsupervised Learning - The Holy Grail of all Learning? Before we go there to explore what he means by "Holy Grail", let's step back to make the context clear and explore what engineers usually mean when they say unsupervised learning and what is meant here. What he means here in short, is a higher-level learning abstraction and not the actual implementation. It is still a mystery how a network trained to predict the next tokens can generalize so much as the LLMs; that is, they learn to generalize in an unsupervised manner, though the training is supervised next token prediction. The usual take on Unsupervised Learning When we usually speak of unsupervised learning in ML, it is a few algorithms related to Clustering, for example, k-means clustering, Dimensionality Reduction, Principal Component Analysis, or Time series fitting. These are based on maths or matrix properties. These are all pretty complex but if you are good at maths or if you put enough effort, these can be clearly understood. In the case of Deep learning, the lower-level example is AutoEncoders. Autoencoders are more interesting in the context of LLMs in how they learn structures for internal compressed representation. In Autoencoders the target is the same as the input. That is, given complex data (say a highly detailed image) train a shallow network to give similar output. For this, the network needs to learn some pattern in the data to compress this sufficiently. See more http://ufldl.stanford.edu/tutorial/unsupervised/Autoencoders/ In this article, we are focussing not on the above classic ML or DL use-case but on unsupervised learning for LLMs. As such, there is nothing extraordinary here. We have seen the usual so-called unsupervised learning (in the context of LLM) via supervised training, the loss function, the training ( based on the usual gradient descent and backpropagation) Ilya Sutskever describes this https://youtu.be/AKMuA_TVz3A?t=490 So, from the humble supervised training, the mysterious un-supervised learning evolves The Emergence of Understanding The paper Language Models are Unsupervised Multitask Learners introduced GPT-2 to the world based on Transformer architecture. It did something very extraordinary. It proved empirically that a sufficiently large LLM, when trained with a sufficiently large dataset (CommomCrawl / WebText- cleaned up internet data) starts to "understand" the language structure. I am deliberately not using the less controversial term 'generalize' instead of 'understand' here to convey the meaning to all better. Here is a Google/DeepMind research paper that is easy to read on this topic https://arxiv.org/pdf/2206.07682.pdf. The same authors have discussed this with better visuals (below) There are other studies like this refuting this, and this debate has just started. The paper links to the inability of the LLM in maths, which is of course, not its forte. To really understand something and not just be a probabilistic distribution generator, one needs essentially a learned internal world model as well -according to Ilya Sutskever. This makes it completely different from other forms of ML and AI. To make this clear, here is a sample input and output from the ChatGpt4 model. The model has not been explicitly trained in automobile driving. But you can see that it has formed a world view or world model from its training data and can comprehensively reply. It is not clear how the emergence of understanding happens when training and model parameter scales, as what we simplify as a probability distribution function generator or dismiss as stochastic parrots may have more revelations in future research. The role of randomness, or its study/measure - probability, in complex interfacing systems, for example, as presented in The Blind Watchmaker, related to evolution and physics at the quantum level sometimes makes one reflect if there is any other way other than probability distribution functions to describe them. It needs more research and more structure to explain this more clearly. For example, I searched entropy and LLM and came to this paper Study of the possibility of phase transitions in LLMs and also a related talk by the founder of Wolfarm and scientist Stephen Wolfarm. So it is not far-fetched to think in these directions. Note that other signals that follow this property of language could also be effective candidates for LLM training, where instead of words, the signal could be tokenized. An example could be music https://ai.googleblog.com/2022/10/audiolm-language-modeling-approach-to.html. Similarly, structures that don't follow this model, like, say, DNA, could be a hard fit for these types of models. Coming back to the point - there is a race on how to tap this potential of LLMs in various fields. We will start with simpler ones and later with more expanded forms where the capability of reasoning of the model could be used in applications. We will cover this in Part 2 of the series.


Fine-Tuning Models using Prompt-Tuning with Hugging Face's PEFT Library

What is Prompt Tuning? It's an Additive Fine-Tuning technique for models. This means that we WILL NOT MODIFY ANY WEIGHTS OF THE ORIGINAL MODEL. You might be wondering, how are we going to perform fine-tuning then? Well, we will train additional layers that are added to the model. That's why it's called an Additive technique. Considering it's an Additive technique and its name is Prompt-Tuning, it seems clear that the layers we're going to add and train are related to the prompt. EXACTLY! That's correct. A prompt is nothing more and nothing less than the instructions we give to the model to perform an action. We write them in our language, i.e., natural language, but the model receives tokens in its language. In large language models, we often refer to them as embeddings, which are numerical representations of the text we send in the prompt. The embeddings on the right represent the phrase 'I am your Prompt.' To carry out the training, what we do is add some additional spaces to the input model's embeddings, and it's those embeddings that will have their weights modified through training. In other words, we are creating a kind of superprompt by allowing a model to supplement part of the prompt with what it has learned. However, that part of the prompt cannot be translated into natural language. It's as if we've learned to express ourselves in embeddings and create incredibly efficient prompts. All the weights of the pretrained model are locked and, therefore, cannot be modified during the training phase. In each training cycle, the only weights that can be modified to minimize the loss function are the ones incorporated into the prompt. The first consequence of this technique is that the number of parameters to train is really small. However, we run into a second, maybe more important consequence, which is that because we don't alter the weights of the pretrained model, it doesn't change its behavior or forget anything it has previously learned.


Our Notebook. The notebook is available on Github: And it is part of the Large Language Models Course: As I mentioned earlier, we're going to train two models. The first will be a model specialized in generating prompts for large language models. The second will be trained to create motivational phrases. To achieve this, we will use two datasets and a single pretrained model. Datasets used: https://huggingface.co/datasets/fka/awesome-chatgpt-promptshttps://huggingface.co/datasets/Abirate/english_quotes The Model: Any model from the Bloom family. In the notebook, I've used bigscience/bloomz-560m and bigscience/bloom-1b1. The first thing we're going to do is load some of the libraries to be used in the notebook. Please note that I loaded the libraries needed in Colab. If you prefer to work in your environment, you might already have some of them loaded, or you may need to install others.


Now we can load the model and the tokenizer. While I've conducted tests with two of the models from the Bloom family, we could have used any model that's compatible with Prompt Tuning for Casual Modeling Language in the PEFT library. You can check the compatible models in the library's GitHub repository: https://github.com/huggingface/peft#models-support-matrix The variable NUM_VIRTUAL_TOKENS contains the number of tokens we want to add to the prompt, which will be trainable. You can modify the value to test how it affects the result; more tokens means more parameters to be trained. Now, let's load the tokenizer and model. Note that with larger models, the download time may increase. I set the trust_remote variable to True so that the model can execute code for its installation, if necessary. The tokenizer is responsible for translating between natural language and tokens understood by the model, and vice versa. It translates the input prompt and the model's outputs. Loading the tokenizer and the model is straightforward using the classes provided by the Hugging Face Transformers library. With this, we already have the model in memory, and we can conduct an initial test without performing any fine-tuning. This way, we can observe how the results change after the two fine-tuning processes.


An initial test with the pretrained model. To execute the model call, I'm going to create a function that takes: the model, the model's input, and the maximum response length. Getting a response from the model will be as simple as calling this function and waiting for the model to provide us with the result. As I've chosen to train two different models, I'll conduct two tests. ["I want you to act as a motivational coach. Don't be afraid of being challenged."] ['There are two things that matter: the size and shape of a flower'] It can't be said that any of the answers are incorrect. The model complements the sentence we've given, but it doesn't know our intention or how we want it to behave. We'll attempt to address this in the fine-tuning phase, where we will train each model with specific sentences to make it respond with a different personality after each fine-tuning. Let's see if we can change the behavior of the pretrained model in the two fine-tuning processes.


Preparing the Datasets. We'll load each dataset separately. To improve performance, I'm only going to load a small number of rows from each of them. In this example, I'm removing the 'act' column from the dataset. It's a design decision; I truly believe it doesn't contribute significantly. However, you can test how it affects the results if we keep the column. It's as easy as commenting out the line that removes the column; the rest of the notebook continues to work. Let's see the first row of the Dataset: {'act': ['Linux Terminal'], 'prompt': ['I want you to act as a linux terminal. I will type commands and you will reply with what the terminal should show. I want you to only reply with the terminal output inside one unique code block, and nothing else. do not write explanations. do not type commands unless I instruct you to do so. when i need to tell you something in english, i will do so by putting text inside curly brackets {like this}. my first command is pwd']} I believe that by keeping only the information from the 'Prompt' column, we can achieve better results. I've tested it, and I think it is the best option. But feel free to run the test yourselves. For the second dataset, I've followed the exact same strategy, removing columns that I consider less relevant. Now we have the data in two datasets: train_sample_sentences and train_sample_prompt. Since we've already loaded the model and the tokenizer, we're ready to begin the fine-tuning process.


Fine-tuning with PEFT. The first step is to create an object with the training configuration. We'll be using the PromptTuningConfig method, but it offers various options, and we need to specify which ones we want to use. Using PromptTuningConfig, we create a variable that holds the configuration we should use in the call to get_peft_model. Let's examine the parameters we are passing to it: task_type: With the value CAUSAL_LM, we are indicating that we want to fine-tune a model for text generation.prompt_tuning_info: This allows us to specify an initial value for the added vectors that can be trained. In this case, I'm indicating that they should be initialized with random values. However, we could also specify that they all start with a specific value, providing a text. I haven't detected significant differences using one method or the other. To do so, we would need to specify the values: - prompt_tuning_info = PromptTuningInit.TEXT - prompt_tuning_init_text = "Initial text here"num_virtual_tokens: The number of tokens added to each prompt that can be trained. It should maintain a relationship with the length of the prompts we are going to send. The shorter the prompt, the fewer virtual tokens should be configured for training. Increasing them doesn't always lead to better model performance.tokenizer_name_or_path: The name of the model. To create the model to fine-tune, all we need to do is call get_peft_model and provide it with the pretrained model we have selected and the configuration we just created. We will create two models using the same configuration and pretrained model. trainable params: 4,096  all params: 559,218,688  trainable%: 0.0007324504863471229 None trainable params: 4,096  all params: 559,218,688  trainable%: 0.0007324504863471229 None What do you think? We're going to train a model by modifying only 0.0007% of its weights. This is spectacular! It's even more remarkable when you consider that in the paper "The Power of Scale for Parameter-Efficient Prompt Tuning" you can achieve results equivalent to full fine-tuning. Once we have the model, we need to create the training configuration. We achieve this with the TrainingArguments class, which I'm going to place within a function to be able to use different arguments for each of the models. In the first parameter, we see that we're specifying a directory. This directory will contain the fine-tuned model, and it's mandatory to pass it. The other parameters are already familiar. The learning_rate, which indicates the maximum weight change allowed in each step, and num_train_epochs, which specifies the number of epochs we want for the training. The following code only creates the directories that will hold the models in case they don't exist. Now we can create the training arguments for each model. Let's continue the fine-tuning process. I'm going to create a function that we will call for each model we want to fine-tune. We create the Trainer object, and to do so, we pass: The PEFT model is obtained from the call to get_peft_model.The arguments we created with TrainingArguments.The dataset we prepared at the beginning of the notebook.The result of calling DataCollatorForLanguageModeling, which prepares the dataset to be processed in blocks by the model. Now we have everything ready to perform the fine-tuning of the models: The model training process in Colab can take about 10 to 15 minutes. On my machine, a MacBook Pro with an M1 Pro chip, it takes around 3 minutes per model. As you can understand, the training conducted hasn't been very exhaustive. Only a small amount of data and a few epochs were used. Feel free, I would even say it's almost necessary, to adjust the parameters to perform fine-tuning with more data and epochs to see how the response of our models varies. I save the models because one of the most important features of this type of training is the minimal disk and memory space they occupy. It's true that they need to work in conjunction with the pretrained model, which needs to be loaded into memory. However, we could have n fine-tuned models that use the same pretrained model loaded into memory. With other fine-tuning techniques, we would need n copies of the pretrained model, since their weights would have been modified. On my hard drive, each of the models I just saved occupies 300KB. Please check how much they occupy on yours, but it should be something similar. Let's load each of the models and make the same call as to the pretrained model to see if their response is the same or has been modified. I want you to act as a motivational coach. You can use this method if you're not sure what your goals are. Let's compare the results: Pretrained Model: I want you to act as a motivational coach. Don't be afraid of being challenged.Fine Tuned Model: I want you to act as a motivational coach. You can use this method if you're not sure what your goals are. It's clear that the response is different, and therefore, we have influenced the model's response with our fine-tuning process. While it may not resemble the prompts we've provided, it does resemble instructions more than the phrase from the pretrained model. Let's take a look at the second model: Pretrained Model: There are two things that matter: the size and shape of a flowerFine Tuned Model: There are two things that matter: one is the weather and another, what you do. So, more or less the same as with the first model. The important thing is that we have in memory two models that behave differently due to their fine-tuning, and we only have one copy of the pretrained model in memory.


Conclusion and continue. The Prompt-Tuning technique is truly impressive. The results it yields are very promising, and the savings in both training and inference are remarkable. Please use the notebook to do your own experiments. You can change the Model, the Epochs, the learning_rate, the Datasets. Investigate by yourself, it is the best way to learn.


Resources. The full course about Large Language Models is available at Github. To stay updated on new articles, please consider following the repository or starring it. This way, you'll receive notifications whenever new content is added. If you're interested in delving deeper into the workings of Prompt-Tuning, take a look at this paper. It's quite easy to understand and provides you with an idea of when using prompt-tuning can be a good choice. This article is part of a series where we explore the practical applications of Large Language Models. You can find the rest of the articles in the following list: I write about Deep Learning and AI regularly. Consider following me on Medium to get updates about new articles. And, of course, You are welcome to connect with me on LinkedIn, and twitter.


Temporal Edge Regression with PyTorch Geometric

Graphs and Time Graphs are becoming one of the favorite tools of data scientists. Their inherent structure allows for efficient storage of complex information, such as the ongoing protein interactions in your body or the ever-evolving social network surrounding you and your friends. Additionally, graphs can be adapted to temporal scenarios. They can vary from the simple static form, where there is no notion of time, to a fluid spatiotemporal setup, where the topology is fixed but the features change at regular intervals, to the chaotic, fully continuous and time-dynamic mode, where everything can change at any time. [1] Graph Neural Networks (GNNs) are used to process graphs, and many have already demonstrated exceptional results for recommendation systems, supply chain optimization, and road traffic prediction. [2] Nonetheless, most GNNs operate on static graphs, limiting their use for temporal problems. What happens then when we have time series data stored within graphs for which we want to predict future values? It turns out that the Transformer could be a solution, as it has already shown remarkable performance in time series forecasting scenarios. [3]


Our Goal To study the effectiveness of combining GNNs and Transformers for time series forecasting on graphs, we will be solving an edge weight prediction problem. Using the data provided by Vesper, we will create a sequence of spatio-temporal graphs acting as snapshots of the global trade market of butter. Our goal is then to


Trade Data All countries publish their trade records on a monthly basis. Each record indicates the reporting country, the partner country in the transaction, the product, and its traded volume (our target variable). Since each trade is logged twice, we focus on exports. The data at our disposal ranges from January 2015 to April 2023, totaling more than 153,000 transactions. The dataset lists 242 "countries," including the exports of regions that aggregate multiple countries together, which means that there are 242*241 = 58,322 possible country pairs. Across all months, each pair is an individual time series that we will be forecasting. We also include other data series as features to support our model's learning. In particular, we define country-specific features, such as domestic butter production or GDP, and pair-specific attributes, such as the traded volume or the exchange rate.


Building Graphs with PyTorch Geometric We proceed by segmenting the data monthly. We designate two types of nodes: one for the exporter (exp_id) and another for the importer (imp_id). Undirected edges are then used to indicate the relationship between two countries. In this manner, we can assign distinct country-specific features to the various types of nodes and embed the pair-specific attributes on the edges. The resulting graph for each month is heterogeneous, fully connected, and bipartite. It can be visualized as follows. Altogether, we end up with a sequence of graphs, each serving as a static snapshot of the market's status at the end of a specific month. Assuming that the data series for the features are updated more rapidly compared to trade data, we can create graphs for the three most recent months we aim to predict, which we refer to as the target months. These graphs are identical to the historical months but lack trade information in the edges. In Python, we construct a heterogeneous graph using the HeteroData object from the PyTorch Geometric (PyG) library. [4] Note that trade volume data is stored in variable vol, and it is assigned to the edge weights, called edge_label in PyG, instead of the edge attributes. The resulting HeteroData object for January 2015 is: It contains the following variables: exp_id: a list of node features per exporting node.imp_id: a list of node features per importing node.edge_index: a list of node indices denoting the connections of the edges.edge_label: a list of the edge labels, where our target variable is stored.edge_label_index: a list mirroring edge_index, for the edge labels.edge_attr: a list of edge attributes per edge. With the graphs ready, we can proceed to define the individual components of our model.


The Model - Individual Components Taking inspiration from the time-and-graph approach, we employ an Encoder-Decoder architecture. [5] The three crucial units are: GNN Encoder: to generate static node embeddings.Transformer: to create temporal node embeddings.Edge Decoder: to extrapolate predictions.


GNN Encoder Similar to how Convolutional Neural Networks work with images, GNNs perform an optimizable transformation on all the features of a graph, preserving the data distributions within that graph. [6] They are used to convert a complex graph schema into vectors of numbers known as node embeddings. Naturally, some information loss is unavoidable. Therefore, the better the GNN, the lower the loss. For more information on GNNs, you can refer to this Medium article. In our case, we deploy the GATv2Conv operator, which introduces attention to rank neighboring nodes based on their significance in creating each node embedding. [7] In Python:


Transformer Transformers are a remarkable architecture capable of making sequence-to-sequence predictions and serve as the foundation of large language models like ChatGPT. [8] Through complex attention mechanisms, they can quantify the relationships of embeddings relative to input order. Then, they generate a relevant prediction (as an embedding) for the next element in the sequence. To understand Transformers on a deeper level, I recommend this YouTube video. We implement a transformer to extrapolate the temporal dynamics across the static monthly snapshots and generate a "temporal" node embeddings for the month under prediction. After applying positional encoding, as shown on PyTorch's website, we define:


Edge Decoder This unit receives the temporal node embeddings and makes inferences for the target variable. In practice, the embeddings are passed through two linear layers that reduce the dimensionality of a pair of node embeddings to a single number: the predicted trade volume between a specific pair of countries. For a refresher on linear layers, I invite you to read this Medium article.


The Model - All Together The full model is initialized as follows: To understand how the model is constructed and the interplay between the three units, I will guide you through the training process of one epoch. After setting the number of months to predict (3 in our case) and choosing the optimizer with its learning rate (Adam, 0.00005), we run: Here, monthly_snapshots_list is the sequence of monthly snapshots, and tot_num_months equals its length. To capture the temporal trends, we implicitly set in range(5, ...) a window with a horizon of 4. This setup ensures that historical always holds the four months leading up to the month under prediction, named m. During training, we slide this window along the sequence of monthly snapshots, updating m with each iteration. For the train and test functions, we follow a typical structure inspired by the "Training a Heterogeneous GNN" section of this Google Colab notebook. The functions send historical and m to the model, which is initialized in its complete form like this: The to_hetero function adapts the GATv2Conv operator to heterogeneous graphs. Next, we define the forward pass, summarised by the following diagram: First, each HeteroData object from the historical list is passed through the GNN Encoder. Next, the resulting static node embeddings are concatenated into the source tensor for the Transformer. The unit generates the temporal node embeddings for the month under prediction, which are subsequently fed to the Edge Decoder. The latter ultimately produces the predictions for month m. In Python: For each training step, we compute the Mean Squared Error by comparing the model's forecast to the actual trade volumes of the month under prediction. The loss is then back-propagated to each unit, with PyTorch ensuring that this occurs in a balanced manner. At the end of the training phase, we slide the window once more and allow the model to compute the predictions for the first month in the target list. When the window is moved again, the historical list includes the last three training months plus the first target month, which has just been enriched with the predicted trade volumes. Therefore, as we slide the window into the future, the model will increasingly base of its forecasts on previous predictions it made. After training our model for several epochs, we proceed to its evaluation.


The Results To assess the effectiveness of our model, we benchmark its performance against traditional forecasting techniques by running PyCaret forecasting experiments on a subset of trade routes. [9] The model achieves perfect predictions for when trade doesn't occur, and performs relatively well on pairs with extremely high volumes. Both scenarios involve trades of outlying amounts, making their predictions easier to make. In contrast, trade routes with average volumes are more challenging to distinguish, explaining the model's inconsistent performance in such cases. Future developments will focus on enhancing the model's ability to capture the complexity within the data by: expanding the set of features;optimizing the architecture and hyperparameters;exploring methods from the PyG Temporal library;increasing the size of the training and testing datasets. [10]


The Verdict In this article, we created a sequence of static heterogeneous graphs to represent the global trade market of butter. We then implemented a model that slides along that sequence and uses a Graph Neural Network and a Transformer to extrapolate relationships within and across snapshots. Finally, we tested the model's performance against traditional forecasting methods to quantify its relative improvement. Given the size of the problem and the various facets of improvement we discussed, it is premature to draw a conclusion on the effectiveness of combining GNNs with Transformers for making time series predictions on graphs. However, graphs have demonstrated their strength in representing temporal problems. Due to their holistic nature, they allowed us to learn relationships between pairs of nodes by drawing insights from other pairs, which becomes useful in the future if the graph structure changes. In contrast, traditional forecasting methods work on an individual series completely in isolation and lack this advantage. For those interested in the full story of the butter trade challenge, feel free to read this other Medium article that I wrote. If this article has inspired you and helped you to better understand time series forecasting on graphs, I invite you to connect with me on LinkedIn!


Acknowledgments Data provided by Vesper, the most comprehensive and user-friendly commodity intelligence platform. Project ideator: Ea Werner.


Bibliography Machine Learning on Dynamic Graphs and Temporal Graph Networks  MLSYS 2021. (n.d.). YouTube. https://youtube.com/watch?v=xzcYyIiFXcY&list=PLG2m1ulw_f2Xz4FIVR0Bcuew8sZ%20S9hm-z&index=1Zhou, J. (2018, December 20). Graph Neural Networks: A Review of Methods and Applications. arXiv.org. https://arxiv.org/abs/1812.08434Yes, Transformers are Effective for Time Series Forecasting (+ Autoformer). (n.d.). https://huggingface.co/blog/autoformerHome - PYG. (n.d.). https://pyg.org/Huang. (2023, January 19). Temporal Graph Learning in 2023 - Towards Data Science. Medium. https://towardsdatascience.com/temporal-graph-learning-in-2023-d28d1640dbf2#a31bLabonne, M. (n.d.). Hands-on graph Neural networks using Python. O'Reilly Online Learning. https://www.oreilly.com/library/view/hands-on-graph-neural/9781804617526/Brody, S. (2021, May 30). How Attentive are Graph Attention Networks? arXiv.org. https://arxiv.org/abs/2105.14491Vaswani, A. (2017, June 12). Attention is all you need. arXiv.org. https://arxiv.org/abs/1706.03762Home - PyCaret. (2023, June 28). PyCaret. https://pycaret.org/PyTorch Geometric Temporal Documentation - PyTorch Geometric Temporal documentation. (n.d.). https://pytorch-geometric-temporal.readthedocs.io/


Navigating the Mind of GPT: How to Elicit Clarity and Avoid AI Hallucinations

Introduction When working with cutting-edge language models like GPT, we occasionally find ourselves stumbling upon "hallucinations." A hallucination, in the context of a language model, is when the model generates information that isn't accurate, is unsubstantiated, or is simply made up. Although GPT is trained on vast amounts of text and is very proficient at generating human-like responses, it isn't infallible. A challenge users often encounter is how to reduce these hallucinations without having access to the model's inner workings or training methods. This blog post is here to help with that, offering a way to get more accurate and guarded responses from GPT solely through prompting.


Diving Into GPT's Imaginary World! Ever seen GPT spin a tale from thin air? Here is an example, GPT knows a lot about popular TV shows or other works of fiction but doesn't know the details (like all the scenes or episodes). But toss in a quirky detail like a "giant hat," and voila - you've got GPT daydreaming! In this episode of Community, the only hat Jeff wears is a graduate cap, no giant hats.


Understanding The Basics First, it's vital to grasp the data sources GPT relies upon. Take the C4 dataset, for instance. It's a colossal collection of text that GPT uses for training. But what exactly is the C4 dataset? In the example above, GPT generates a response that provides a general overview of the C4 dataset but admits uncertainty regarding specific details.


Crafting Careful Prompts The trick to getting more accurate responses lies in the prompt's phrasing. Here are some strategies to consider: 1. Be Specific: Asking vague questions will often lead to vague answers. Narrow down your inquiry.2. Request Confirmation: Ask GPT to provide sources or admit when it's unsure. 3. Limit the Scope: Instead of seeking a long answer, ask for brief responses and then follow up for clarity.


"I'm not sure" - Your New Safety Net Adding qualifiers like "if you're not sure, say 'I'm not sure'" can be an effective way to reduce hallucinations. It offers GPT a "way out" instead of pushing it to guess or fabricate an answer. For instance: The "I'm not sure" clause can act as a filter, signaling the model to be more cautious with its output.


Avoiding Overloading While the "I'm not sure" method is handy, it's essential not to overdo it. Using it too often might yield overly cautious responses, where the model constantly defaults to saying, "I'm not sure" instead of providing potentially useful information. Balance is the key.


Additional Strategies to Enhance Prompt Clarity and Reliability 1. Ask Direct Questions: Ambiguous queries can lead to more instances of hallucination. When you ask direct questions, you offer less room for the model to wander. 2. Encourage Fact-Checking: Ask GPT to cite its sources or state the confidence in its response. 3. Multi-step Queries: Breaking down your main question into smaller parts can be effective. This way, if the model is uncertain about one segment, it can express it without affecting the other parts. 4. Ask for Confirmation: After getting a response, you can ask the model to confirm its accuracy. 5. Utilize Follow-up Questions: Don't just rely on the initial response. Dive deeper with follow-up questions. If the model is consistent in its answers, it's a good sign. If it starts to waver or shows inconsistency, it's a hint that it might be uncertain.


Wrapping Up Hallucinations can be a hurdle when interacting with models like GPT. However, with careful prompting, one can substantially reduce these occurrences. The beauty of GPT lies in its adaptability to prompts, giving users the power to guide the kind of responses they receive. By making your prompts more specific, requesting confirmatory details, and providing the model with a "way out" when it's unsure, you can harness the best of what GPT has to offer while minimizing misinformation. Remember, technology is only as effective as how we choose to use it. With the right approach, GPT can be a reliable and insightful tool in your information-gathering arsenal.


LangChain 101: Part 2c. Fine-tuning LLMs with PEFT, LORA, and RL

Model Fine-tunning Model fine-tuning, also known as transfer learning, is a machinelearning technique used to improve the performance of a pre-existing model on a specific task by training it further on new datarelated to that task.Fine-tuning is commonly employed in scenarios where a pre-trainedthe model has learned valuable representations of a general domain(in our case, natural language) and is adapted to perform wellon a narrower, more specific task. I've heard a lot of questions: is it worth fine-tuning a large language model? Why not use prompts - they seem to be ok. Can I do the vectorstore instead? And other questions regarding the topic... Think of the following situation: you're up to go to a dentist. Who would you prefer: A person who acts as a doctor (prompt: "Imagine you're a dentist...")A person who has read all the literature regarding dental care (using vectorstores)A doctor who was trained to do dental operations (fine-tuned model) The audience whom I spoke to, definitely chose the last option (PyData and Data Science meetups)


PEFT - The Parameter-Efficient Fine-tuning PEFT, or Parameter-Efficient Fine-tuning, is a method that improves the performance of pre-trained language models without fine-tuning all of the model's parameters. This makes it a much more efficient and cost-effective way to fine-tune LLMs, especially for large models with hundreds of billions or trillions of parameters. PEFT works by freezing some of the layers of the pre-trained model and only fine-tuning the last few layers specific to the downstream task. This is based on the observation that the lower layers of LLMs tend to be more general-purpose and less task-specific, while the higher layers are more specialized for the task that the LLM was trained on. Classic transfer learning. Here is an example of how PEFT can be used to fine-tune an LLM for a text classification task: Freeze the first few layers of the pre-trained LLM.Fine-tune the unfrozen layers of the LLM on a smaller dataset of labeled text.The model is fine-tuned and can be tested on unseen data.


LoRa - The Low-Rank Adaptation Everyone Talks About LoRa, or Low-Rank Adaptation, is a fine-tuning technique that can adapt large language models (LLMs) to specific tasks or domains without training all of the model's parameters. LoRa doesn't modify the current transformer architecture in any fundamental way. LoRa freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture. LORA works by adapting the weights of a pre-trained transformer model to a specific task or domain by decomposing the difference between the original pre-trained weights and the desired fine-tuned weights into two smaller matrices of low rank. These two matrices are then fine-tuned instead of the complete set of model parameters. This can reduce the number of trainable parameters by 10000 times, still achieving comparable performance to full-parameter fine-tuning. During fine-tuning, LORA updates the weights of the low-rank embedding and projection layers, as usual for data science, minimizing the loss function. Now, what is the difference between PEFT and LoRa? PEFT is a method that employs various techniques, including LoRa, to fine-tune large language models efficiently. This approach has several advantages: It is more time-efficient than full-parameter fine-tuning, especially for large transformer models.It is more memory-efficient, making it possible to fine-tune models on devices with limited memory.It is more controlled fine-tuning, as the low-rank matrices encode specific knowledge or constraints.


Reinforcement Learning Reinforcement learning (RL) is another way you can fine-tune models. It requires two copies of the original model. One copy is the active model, trained to perform the desired task. The other copy is the reference model, which is used to generate logits (the unnormalized output of the model) that constrain the active model's training. The requirement of having two copies of the model can be a challenge, especially for large models GPU-wise. However, this constraint is necessary to prevent the active model from deviating too much from its original behavior, as RL algorithms can lead to models that generate unexpected or harmful outputs. Let's go through the fine-tuning process: The reference model initializes with the parameters of the pre-trained language model.The active model starts training using reinforcement learning.At each optimization step, the logits of both active and reference models are computed (log-probs).The loss function is then computed using the logits of both the active and reference models (KL metric).The parameters of the active model are then updated using the gradients of the loss function or proximal policy optimization.


Time to Code! The full code is available on GitHub. We'll start with importing peft and preparing it for fine-tuning. We'll set the LoraConfig parameters and use the get_peft_model method to create a PeftModel: r: The rank of the low-rank matrices represents the difference between the original pre-trained weights and the desired fine-tuned weights. A higher value of r allows LORA to learn more complex relationships between the parameters, but it will also be more computationally expensive.lora_alpha: A hyperparameter that controls the trade-off between the loss function for the downstream task and the loss function for preserving the original pre-trained weights. A higher value of lora_alpha will give more weight to the loss function for protecting the initial pre-trained weights.target_modules: A list of the names of the modules in the model that should be fine-tuned with LORA.lora_dropout: The dropout rate to apply to the low-rank matrices during training.bias: The type of bias to use in the low-rank matrices. Valid options are "none," "shared," and "individual."task_type: The type of task that the model is being fine-tuned for. Valid options are "CAUSAL_LM" and "CLASSIFICATION." Now it's time to set up the Trainer class: data_collator: A function that is used to collate the training data into batches.per_device_train_batch_size: The batch size per GPU.gradient_accumulation_steps: The number of steps to accumulate gradients before updating the model parameters. This can be used to reduce memory usage and improve training speed.warmup_ratio: The proportion of the training steps spent on a linear learning rate warmup.fp16: Whether to use floating-point 16 (FP16) precision training. This can improve training speed and reduce memory usage.logging_steps: The number of training steps between logging updates.output_dir: The directory where the trained model and other training artifacts will be saved.optim: The optimizer to use to train the model. Valid options are "adamw," "sgd," and "paged_adamw_8bit".lr_scheduler_type: The learning rate scheduler to use. Valid options are "cosine," "linear," and "constant." All we have left is to start the training: We can now either use the model right away with tokenized input_ids or save it for further use:


Reminder: The complete code is available on GitHub. This is the end of Part 2c. The next part (de) will be dedicated to fine-tuning models using human feedback. Clap and follow me, as this motivates me to write new articles and lets you know when the next part is published.


Notes on CLIP: Connecting Text and Images

Limitations with supervised learning Off-the-shelf features generated by image classification models have been used in other tasks like image retrieval. However, these features do not generalize very well, as the classification models were trained to recognize a fixed set of classes. Any new category added to this set of classes would require collecting additional annotated images for this new category and then retraining the model. This is a time-consuming and expensive process.


Main Contribution The authors propose a pre-training task (CLIP = Contrastive Language-Image Pre-training) of predicting which caption goes with which image in order to learn SOTA image representations from scratch. For this, they created a dataset of 400 million (image, text) pairs collected from the internet. This pre-trained model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset-specific training.


Background CLIP draws its inspiration from the field of supervised image captioning. Each image with a corresponding caption is used to train a model that predicts the exact words in the caption for the corresponding images. This is a difficult task as an image can be described in various ways and still convey the same meaning.


Contrastive Pre-training Consider a batch of N images and their corresponding N captions. With these, we can create N x N possible (image, text) pairings across a batch. Now, the task is to predict the N real pairs in the batch. To do so, CLIP learns a multi-modal embedding space by jointly training an image encoder and text encoder (see Figure 1). The image encoder produces a feature vector, I; similarly, the text encoder produces a feature vector, T. For the N real pairs, we want to maximize the cosine similarity between I and T.For N² - N incorrect pairings, we want to minimize the cosine similarity between I and T.


Zero-shot Prediction Consider the task of image classification (see Figure 2). At test time, for a single image, the image encoder will produce a feature vector I₁. To identify the class of the image, the text encoder embeds the class names of the target dataset to produce N feature vectors T₁, T₂ ..., and so on. N in the number of classes in the target dataset.


Model Details For the image encoder, the authors evaluate two different architectures: ResNet-50: They used the modified ResNet-D (refer paper) architecture with anti-aliased rect-2 blur pooling (refer paper). They also replace the global average pooling layer with a "transformer-style" attention pooling mechanism.Vision Transform (ViT): The authors use an additional layer normalization to the combined patch and position embeddings before the transformer and use a slightly different initialization scheme. For the text encoder, use a Transformer described in this paper with 63M parameters (12-layer 512-wide) and 8 attention heads.


Training The authors train 5 ResNets (ResNet-50, ResNet-101, and 3 EfficientNet-style ResNet models) and 3 Vision Transformers (ViT-B/32, a ViT-B/16, and a ViT-L/14). The models are trained for 32 epochs using Adam optimizer with decoupled weight decay regularization and decay the learning rate using a cosine schedule. They used a very large minibatch size of 32,768.


Effect of Prompt Engineering: Image classification datasets are annotated with label IDs that are mapped to class names. Since the CLIP model is trained with text being a full sentence, the authors found using the prompt template "A photo of a {label}." to be a good default for the text associated with the image. In Figure 3, we see the classification accuracy improve by 5 points using prompt engineering across 36 classification datasets.


Zero-shot CLIP versus Linear Probe The zero-shot CLIP classifier outperforms a supervised linear classifier fitted on ResNet-50 features on 16 out of 27 datasets (Figure 4). However, the performance of CLIP is still below that of state-of-the-art for most of these datasets.


Limitations CLIP does not perform well on tasks like counting objects in an image or finding the distance to the nearest object in an image.It performs very poorly on out-of-distribution datasets like MNIST. Its performance on digital OCR is good however, it fails (88% accuracy) in recognizing hand-written digits of MNIST.Using CLIP for few-shot learning leads to poor performance. There is a counter-intuitive drop in performance when going from zero-shot to few-shot learning.As CLIP is trained on the text-image pairs queried from the internet, it will learn many social biases.


Prompt Robustness: How to Measure and How to Enhance

Why Prompts Are Vulnerable When the generative LLMs were trained, they learned the patterns from the input and used the patterns to predict the next token. Then, they use the input string plus the newly generated output to predict the next token. They iterate the process until the end of the output. This structure is called an autoregressive because it always predicts the next output in a sequence based on the previous outputs in the sequence. This mechanism is very similar to the autoregressors that are often used in time series analysis, where they are used to predict future values of a time series variable based on its past values. Despite the fact that LLMs display impressive language understanding and generation capabilities, their learning is based on statistical patterns, not true knowledge extraction and reasoning as humans do. As such, the LLMs' behavior is highly impacted by the training data. The models are more familiar with the patterns presented in the training data. As a result, any data deviation in the prediction phase may cause a more or less significant performance drop. How do we call this type of problem? Overfitting! It has been reported that trashing low-quality training data can actually improve the model's performance. But that kind of selection of training data can also lead to selection bias in that the language patterns become even stronger. Potentially, the overfitting may get worse if it's not specially treated. Similarly, the computer-vision models also suffer from the same problem. They are sensitive to the unexpected details in training materials as well. For example, a model to detect cows may pay too much attention to the grass and clouds. One of the ways to solve the problem when training CV models was through data augmentation. That is, alter the training images a little bit to create a new data set by tilting, stretching, clipping, adding random noise, masking, etc. However, these tricks are not commonly used in LLM training. The main reason was that, compared to CV tasks, NLP tasks already have much larger amounts of human-generated training materials to train on. Computational resources are already a bottleneck for LLM training. Also, generating high-quality augmented data is not easy. That's why NLP data augmentation is not a priority today. The problem that LLMs do not always follow prompts accurately is just one of the symptoms of LLMs' imperfect training process. Because of this, we must have a way to evaluate the LLM's prompt robustness, i.e., how stable the LLM can perform with different variations of instructions. Making sure the LLMs strictly follow the prompts has been a widespread concern. People are looking at the problem from different angles. Some studies are centered around organizing prompts to make LLM understand better. For example, in the case of few-shot learning, researchers noticed that the LLMs put more weight on the last example, and the common terms in the examples had a higher impact on the model. They are researching how to design prompt templates and organize the examples in a way that makes the prompt work better. Some other research is about safety and security. That is, for instance, how to make sure the LLMs don't disclose sensitive information and don't produce unwanted outputs. This is also an interesting topic. It will be discussed separately. Last but not least: when the LLMs get imperfect prompts, how badly will the performance drop? This is a very practical problem we are facing. When we decide on an LLM to base it on, we need to know how the LLM works with not-ideal prompts. When we fine-tune a model, we need to know how the new LLM compares to the foundation model. Even further, we want to learn how to enhance our model training process to reduce the performance drop, and we would also like to enhance our prompts with the know-how we learned from the model comparison.


PromptBench Introduction The simplest way to evaluate LLM prompt robustness is to produce your own test prompt set manually and use that to run against the models. While this is a quick fix, you can never know if the test set is representative enough, and you can't get an objective indicator of the model's performance. PromptBench [2] offers a plausible solution on this track. Compared to the manual trick, PromptBench's solution is systemic and extensible. It can permute prompt attacks (prompt variations) based on four strategies: Character-level: PromptBench can use TextBugger and DeepWordBug to manipulate texts by introducing typos or errors to words, e.g., by adding, deleting, repeating, replacing, and permuting characters for certain words.Word-level: Use BertAttack and TextFooler to replace words with synonyms or contextually similar words to deceive LLMs.Sentence-level: Use StressTest and CheckList to append irrelevant or extraneous sentences to the end of prompts, intending to distract LLMs.Semantic-level: Simulate different linguistic expressions with the same meaning by translating from six common languages (Chinese, French, Arabic, Spanish, Japanese, and Korean) into English, introducing linguistic nuances and variations that could potentially impact LLMs. It was designed to process a combination of four types of prompts: zero-shot role-based, few-shot role-based, zero-shot task-based, and few-shot task-based prompts. More complicated prompt patterns like COT, ReAct, etc. are not supported. When we use PromptBench, we need to provide a labelled dataset, a task, and a model for the utility to run against. PromptBench already has a list of models and datasets supported, including out-of-box support for ChatGPT and ChatGPT 4. The users can easily extend the framework to their own models and datasets as well. One of the great things about PromptBench is that it provides an evaluation metric. As such, we can compare results on different tasks, using different datasets, against different models. The definition of the metric function is as follows, which, in plain English, is the average performance drop rate: The framework can be found in its Git repo [2]. At the moment, it can only be installed using the conda command. pip install is not supported yet. Running an attack is simple:


Not all LLMs are created equal Some models are more sensitive to prompt attacks than others. As summarized below, the UL2 model is more robust to prompt attacks, and T5 is also not too bad. ChatGPT comes third. Vicuna is the worst model in the benchmark. The difference in PDR between the LLMs is a good reason for us to apply promptBench testing before we decide on the LLMs we are going to use. It also indicated that we should investigate better methods for model training and fine-tuning. When UL2 performs much better than ChatGPT, the team probably must have done something right; while Vicuna performs much worse, there must be some lessons to learn in the training and data preparation processes.


Vulnerable to Lower-level Attacks The second interesting finding is that the models are more vulnerable to word- and character-level attacks. The word-level attacks simulate the impact of using synonyms, while the character-level attacks simulate the impact of typos and non-standard wording. As shown in the above table, word-level attacks result in a 33% to 35% performance drop on average. And character-level attacks caused a 20% to 23% performance drop on average. These two types of attacks are closer to typos and word misuse. The countermeasure is very simple: correct the typos and uncommon sentences in the prompts, and make sure the prompts are concise and clear.


Term Frequency Relevancy The authors also noticed that certain terms are more frequently related to unstable prompts, while others are more commonly used in robust prompts. As depicted below, prompts containing 'acting', 'answering', 'detection', and 'provided' cause less performance drop. In other words, the prompts with these words are more robust. While vulnerable prompts often have words like 'respond', 'following', and 'examine'. This finding can help us produce better prompts by using a more stable vocabulary.


Abra Cadebra Interestingly, the researchers noticed that adding meaningless strings to the prompt can have either a positive or negative impact on its performance. For instance, introducing a random sequence 'LKF0FZxMZ4' during a CheckList attack distracted the attention of the model and reduced focus on the critical segments. As such, it reduced the model's performance. On the other hand, by adding an irrelevant sequence such as 'and true is true', LLMs focus intensifies on the significant words, thus making a better prediction. My understanding is that this revealed the importance of domain-relevant fine-tuning. Many domain applications use specific vocabularies. This kind of information can only make sense to the models after fine-tuning. Before that, they were more like random strings in the LLMs' eyes. I haven't found much research on the impact of adding random sequences to the prompt yet, except for some effort to enhance prompt performance by permutation. I'm more concerned about the risks of this new finding. Because the fact that adding a random string can overturn the model prediction is so scary. The author's report didn't find that the added random strings completely changed the result. Actually, the average performance drop was not significant compared to character- and word-level attacks. But, who knows, probably this feature can be the zero-day bug of the LLM models.


Limitations One of the limitations of PromptBench is that it doesn't support more complicated prompt patterns, like COT, TOT, ReAct, etc. The reason was that it is difficult to apply the same attack strategies to those patterns. However, those advanced use cases are becoming more and more important. I hope it will not take very long before a similar framework is released when people find promptBench necessary. Generating high-quality NLP permutations is very challenging. One way to make sure the generated new sentences are not too ridiculous is by using a constraint function, as implemented in the underline TextFooler library. It discards the generated samples when the cosine distance between the sample and the original sentence is below a certain value. The other way to do quality assessment, as conducted in the PromptBench, is to do human-checking. The authors reported that they found that over 70% of character- and word-level attacks are human-acceptable. However, both the threshold and the standard of acceptable are subjective. Especially when character- and word-level attacks are the most impactful attack types. It's reasonable to question: has the generated prompt gone too far away? This is one of the things we need to be aware of when using the library.


Closing Words In a proper software project, a new piece of software must go through a set of tests before being rolled out into production. It needs to pass unit tests to make sure every logical turn aligns with the requirements. It needs to go through an integration test to make sure the new function doesn't break any contracts with other components. Finally, it needs to pass a regression test to make sure it is compatible with the old version and doesn't introduce any unwanted changes. ML projects do not strictly follow the same process because ML is more experimental than software projects. However, making things repeatable and measurable is essential to making ML widely applied. In this sense, PromptBench is a good example. It's an important step ahead in making generative AI development an engineering process. My suggestion for using PromptBench is: Conduct benchmark testing when you choose a LLM, before and after fine-tuning your modelDevelop and version control your own test datasetKeep records of your benchmark resultsApply the insights to your model's fine-tuning and prompt design.


This AI newsletter is all you need #67

What happened this week in AI by Louie With recent developments, it has become increasingly clear that Large Language Modes (LLMs) are now becoming far more than chatbots. We were interested to read renowned AI figure Andrej Karpathy's thoughts on this in a recent tweet where he sees LLMs becoming the kernel processes of a new operating system. In his view, thinking of LLMs as "just chatbots" is analogous to thinking of early computers as "just calculators". This distinction becomes even more apparent with LLMs' evolving capabilities, such as their capacity to operate in a multi-modal context, their ability to access various tools and the Internet, their aptitude for interpreting and executing code, and their versatile application as embedding databases. We find this analogy and the carry over of other computing concepts to LLMs intriguing and can help us form a better picture of how the technology could continue to develop. While closed source LLMs have been busy rolling out new functionality such as ChatGPT's new updates to accommodate multi-modality, we have also been noting several promising new open-source competitors in the field. Alibaba, for instance, has open-sourced their Qwen model, along with its chat variant with 7 billion parameters, in a move aimed at competing with Meta. This also signifies the very first instance of a prominent Chinese company publicly releasing a model. Also new to the open source LLM race, Mistral, a startup that secured over $100 million in funding just six months ago has now publicly released the Mistral-7B model under the Apache 2.0 license. This model demonstrates superior performance across various benchmarks when compared to both LLaMA 2 13B and LLaMA 1 34B. It achieves CodeLlama 7B-level performance in coding tasks while maintaining strong proficiency in English-related tasks. The model is freely available for download with no restrictions. - Louie Peters - Towards AI Co-founder and CEO


This AI newsletter is all you need #67

Announcing our Free Certification Course on Training & Fine-Tuning LLMs for Production We're excited to release Towards AI's second free certification course on Training and fine-tuning LLMs for Production in collaboration with Activeloop and Intel Disruptor Initiative. In this course, you will cover the intricacies of training, fine-tuning, and seamlessly integrating these models into AI products. This course will guide you in building the most cost-efficient software and hardware stack and state-of-the-art methods for preparing LLMs for production. It will also cover essential topics such as proprietary versus open-source models, various LLM training methodologies, and production deployment strategies. We also touch upon advanced fine-tuning techniques like LoRA, QLoRA, SFT, and RLHF or training custom models with Cohere. With the support of our partners at Cohere and Lambda, qualifying participants will receive compute credits to be able to run the examples themselves! The course will go live on 9th October 2023. Pre-register today!  Join the AI Revolution with our friends at Alep!  Alep is shaping the future role of professional investment analysts by removing repetitive tasks and amplifying research capabilities, all anchored on the vision that LLMs will revolutionise the analyst role. As a nascent startup, Alep's MVP is in motion and they are gearing up to bring on their first customers. While UK-rooted, Alep are open to remote flexibility with a touch of essential in-person collaboration. Alep AI is founded by Matty Cusden-Ross, who was previously the CEO & Founder of fintech Flux which raised £10mn+ and saw integration into Barclays, Tier 1 retailers and a user base of 1.3mn! Prior to Flux Matty was the 2nd employee at Revolut where he drove user growth from 0 to 100k and tapped a 400k+ growth channel. Alep are on the hunt for a visionary Technical Lead or CTO - could it be you? Apply here and shape the future of financial analysis!


This AI newsletter is all you need #67

Hottest News Mistral 7B The Mistral AI team released the Mistral 7B model, the most potent language model for its size to date. Powered by Grouped-query attention (GQA) and Sliding Window Attention (SWA), it outperforms other models in various domains while maintaining strong performance in English and coding tasks. 2. OpenAI's ChatGPT Now Searches the Web in Real Time - Again OpenAI has reintroduced web searching for ChatGPT, available to Plus and Enterprise users, allowing users to generate answers to prompts that search the web for the latest information. Some crucial updates include compliance with robots.txt rules and user agent identification, giving websites more control. 3. LLM Startup Embraces AMD GPUs, Says ROCm Has 'Parity' With Nvidia's CUDA Platform A startup called Lamini uses over 100 AMD Instinct MI200 GPUs and found that AMD's ROCm software platform "has achieved software parity" with Nvidia's dominant CUDA platform. They claim the computing costs are ten times cheaper than Amazon Web Services. 4. How General AI Will Eventually Reshape Everything The transition to Artificial General Intelligence (AGI) signifies more than a change in terminology; it represents a significant leap in capabilities. AGI tackles the core challenge posed by AI - machines that excel at tasks but lack generalization abilities. With AGI, we can unlock doors to understanding and problem-solving. 5. Chroma X Google for Building on PaLM Chroma is a leading vector database for storing embeddings for AI applications. Chroma has integrated with PaLM embeddings throughout the beta and is partnering with Google AI on the public release.


This AI newsletter is all you need #67

Five 5-minute reads/videos to keep you learning First Impressions With GPT-4V(ision) OpenAI has released GPT-4V for Plus users, showcasing its image processing skills, OCR capabilities, and performance in solving mathematical problems. This guide shares the first impressions of the GPT-4V image input feature with a series of experiments to test the functionality of GPT-4V, showing where the model performs well and where it struggles. 2. Non-Engineers Guide: Train a LLaMA 2 Chatbot This tutorial presents how anyone can build their open-source ChatGPT without writing a single line of code. With tools like AutoTrain, ChatUI, and Spaces, even non-ML specialists can create advanced ML models, fine-tune LLMs, and easily interact with open-source LLMs. 3. How To Prompt Like a Pro? This video tutorial focuses on why large language models react differently to the same prompts. It covers tokenizers, embeddings, inference parameter optimization, and some excellent prompting techniques. 4. Is AI a Platform Shift? Think of a platform shift as a change in the dominant layer that applications are built on. This article presents AI as a possible platform shift with different hypotheses to support it. 5. Student Use Cases for AI While Generative AI tools and LLMs empower students and educators with advanced technology, they also present challenges like the need for user verification and potential biases. This article examines student use cases such as AI as a feedback generator, personal tutor, team coach, and learner.


This AI newsletter is all you need #67

Papers & Repositories QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models QA-LoRA, a new method in quantization-aware training, outperforms QLoRA in terms of efficiency and accuracy. It balances the trade-off between quantization and adaptation, leading to minimal loss of accuracy. It is especially effective in low-bit quantization scenarios like INT2/INT3 without the need for post-training quantization, and it can be applied to various model sizes and tasks. 2. Small-Scale Proxies for Large-Scale Transformer Training Instabilities A study has discovered that instabilities in training large Transformer-based models can be detected in advance by analyzing activations and gradient norms. These instabilities, which occur in smaller and larger models with higher learning rates, can be mitigated using strategies employed in large-scale settings. 3. Language Models in Molecular Discovery Language models are being used in chemistry to accelerate the process of molecule discovery and show potential in early-stage drug research. These models assist in de novo drug design, property prediction, and reaction chemistry, offering a faster and more effective approach to the field. Moreover, open-source language modeling software enables scientists to easily access and advance scientific language modeling, facilitating quicker chemical discoveries. 4. Tabby: Self-Hosted AI Coding Assistant Tabby is a fast and efficient open-source AI coding assistant compatible with popular language models. It supports CPU and GPU for coding tasks and offers swift coding experiences. 5. The Reversal Curse: LLMs Trained on "A Is B" Fail To Learn "B Is A" Researchers have discovered a phenomenon called the "Reversal Curse" that affects the generalization abilities of auto-regressive LLMs. These models struggle to infer the reverse of a fact, hindering their ability to answer related questions accurately. Even larger models like GPT-3.5 and GPT-4 face challenges in addressing this issue, indicating a need for further advancements in language modeling. Enjoy these papers and news summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #67

Meme of the week! Meme shared by r1gell


This AI newsletter is all you need #67

Featured Community post from the Discord Bri built Galactic, a cleaning and curation tool for massive unstructured text datasets. It's designed to help you curate fine-tuning datasets, create document collections for retrieval-augmented generation (RAG), and even perform deduplication of web-scale datasets for LLM pre-training. Check it out on GitHub and support a fellow community member! Share your feedback and questions in the thread here.


This AI newsletter is all you need #67

AI poll of the week! Join the discussion on Discord.


This AI newsletter is all you need #67

Article of the week Temporal Edge Regression with PyTorch Geometric by Marco Lomele Graphs are becoming one of the favorite tools of data scientists. Graphs can be adapted to temporal scenarios. They can vary from the simple static form, where there is no notion of time, to a fluid spatiotemporal setup, where the topology is fixed. Still, the features change at regular intervals to the chaotic, fully continuous, and time-dynamic mode, where everything can change at any time. This article studies the effectiveness of combining GNNs and Transformers for time series forecasting on graphs.


This AI newsletter is all you need #67

Our must-read articles Smart Control of Traffic Lights Using AI by Mihir Gandhi Advanced Matplotlib: A Comprehensive Guide to Data Visualization by RaviTeja G If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.


This AI newsletter is all you need #67

Job offers Prompt Engineer - US - 280923 @Sirion Pte Ltd (Remote) Senior Machine Learning Engineer @Employment Hero (Remote) Senior Data Scientist @Tiger Analytics (US/Remote) Data Science & AI Applications Manager @BambooHR (Remote) Team Lead, Machine Learning @Clari (Remote) Research Engineer, AI @Foundry (London, UK) Junior Software Engineer @Scientific Systems Company, Inc. (Woburn, MA, USA) Interested in sharing a job opportunity here? Contact sponsors@towardsai.net. If you are preparing your next machine learning interview, don't hesitate to check out our leading interview preparation website, confetti!


Chat with Your BigQuery Data

Introducing Vanna AI Vanna AI is an open-source Python-based AI SQL agent trained on your schema that writes complex SQL in seconds.


Chat with Your BigQuery Data

How does it work? Vanna works in two easy steps: Step 1: Train a model on your data Provide Vanna with sample SQL queries, documentation, and DDL statements to teach Vanna about your database structure and terminology. The more high-quality training data you provide, Vanna will understand your data better. Step 2: Ask questions Once trained, you can ask Vanna plain English questions about your data, like "What are the top customers by revenue?". Vanna will analyze the question, search its training data to understand your schema and generate the appropriate SQL query to answer the question. Behind the scenes, Vanna utilizes GPT-4 to generate highly accurate SQL tailored to your database. As you continue to use Vanna, it learns from your questions and feedback, constantly improving over time. Read their whitepaper to learn how they maximize SQL generation accuracy.


Chat with Your BigQuery Data

What does training mean? Training here means providing the LLM context about your specific data and use case. Vanna supports automatic training for Snowflake, and it's in the works for BigQuery. Below are the options to manually train Vanna. Train Using Schema Train Using Documentation Train Using SQL Examples Enough talking; let's see what Vanna can do!


Chat with Your BigQuery Data

Vanna  BigQuery Here is the Colab notebook for you to follow along. Click here to get an account with Vanna. First, install Vanna for BigQuery. We're using Colab to authenticate with BigQuery; refer to the docs for other ways to authenticate. Now, get your Vanna API key. Provide your email, and you should receive a code. Then, provide Vanna with your project ID on GCP.


Chat with Your BigQuery Data

Medium Blog Titles The first dataset we'll test is a dataset of Medium blog post titles and claps I used for a past project for optimizing blog titles using GPT-3. We give a name for the model we're building for this dataset. Here, I call it medium_blog_titles. Follow the prompt, and choose BigQuery. Note: You can see all the models you've created using vn.get_models() I give it the DDL of the dataset, which I got using this query. You can see that Vanna now has this training data with vn.get_training_data() Let's start with a simple question. A successful run will return the SQL query, the results as a dataframe, a Plotly figure, and follow-up questions. What is happening under the hood? vn.askis a wrapper around vn.generate_sql, vn.run_sql, vn.generate_plotly_code, vn.get_plotly_figure, and vn.generate_followup_questions. Here's what it looks like in code: If you don't want Vanna to output everything, you can set print_results to False, and unpack the results into the respective values. Now, let's try another question. Note: For every question you ask, Vanna fetches context from the model's training data. You can see the related training data for a specific question using vn.get_related_training_data() as shown below: Let's ask Vanna. This is the SQL query. The interactive plot. The follow-up questions. Now for a statistics challenge. It did pretty well! Let's test Vanna on a BigQuery public dataset.


Chat with Your BigQuery Data

Chicago Crime data We'll be using the Chicago Crime dataset on BigQuery. After providing the DDL, you can have Vanna come up with questions for this dataset. Let's try the first question: That's a lot of crime! Let's see which block you should avoid in Chicago by asking where the highest number of reported crimes occurs. Let's see some trends in the crime. Is it increasing? It's decreasing! What about in terms of months? Are there months that are more popular for crimes? Looks like July is a hot time for crime. Last one, let's see if Vanna can do window functions. It is showing top crimes for each district, but I want it to be only one row for each district. Let's tell Vanna that That's the right result! It just needed a litle prompting. Let's test Vanna on a dataset with two tables.


Chat with Your BigQuery Data

Austin bike share There are two tables in the Austin Bikes dataset: bikeshare_trips - contains all the trip IDs, trip start and end, membership type and station names + IDbikeshare_station - a list of all the stations and their geographical location & station ID Let's ask a question that incorporates both tables. One more: Most of the stations are solar-powered!


Chat with Your BigQuery Data

Final thoughts After trying out Vanna on a couple of datasets, my initial impression is that it works great with straightforward queries. For more complex ones, you would have to phrase your question better. I tried providing SQL queries as "training data," but it still failed to get the right query with the same question provided. I have yet to test it on a more complex dataset with more tables, but so far, it works great. The queries don't take long, and the generated plots are a nice touch. So why use Vanna? Here's what the GitHub page says. As someone who's currently a student, I see this as a good learning tool. If I were given a dataset to explore, this tool would provide good questions to dig into the dataset and help me write the correct SQL queries. There's even a function in Vanna that takes SQL queries and turns them into English questions. I assume this would also be a good exploration tool for someone in the industry. Some other thoughts I have is whether more fine-tined models like SQLCoder would outperform GPT-4. As well as safety concerns and risks of using this as an interface with a data warehouse. But that's for another article. Thank you for reading, and check out some resources below if you want to explore Vanna further!


Chat with Your BigQuery Data

Resources You can try out Vanna in this Streamlit app (code) Full DocumentationWebsiteSlack channel for supportLinkedIn


Distributed training with 🤗 Accelerate

As models get bigger, parallelism has emerged as a strategy for training larger models on limited hardware and accelerating training speed by several orders of magnitude. At Hugging Face, we created the 🤗 Accelerate library to help users easily train a 🤗 Transformers model on any type of distributed setup, whether it is multiple GPU’s on one machine or multiple GPU’s across several machines. In this tutorial, learn how to customize your native PyTorch training loop to enable training in a distributed environment.


Setup

Get started by installing 🤗 Accelerate:
  Copied pip install accelerate
Then import and create an Accelerator object. The Accelerator will automatically detect your type of distributed setup and initialize all the necessary components for training. You don’t need to explicitly place your model on a device.
  Copied >>> from accelerate import Accelerator
>>> accelerator = Accelerator()


Prepare to accelerate

The next step is to pass all the relevant training objects to the prepare method. This includes your training and evaluation DataLoaders, a model and an optimizer:
  Copied >>> train_dataloader, eval_dataloader, model, optimizer = accelerator.prepare(
...     train_dataloader, eval_dataloader, model, optimizer
... )


Backward

The last addition is to replace the typical loss.backward() in your training loop with 🤗 Accelerate’s backwardmethod:
  Copied >>> for epoch in range(num_epochs):
...     for batch in train_dataloader:
...         outputs = model(**batch)
...         loss = outputs.loss
...         accelerator.backward(loss)
...         optimizer.step()
...         lr_scheduler.step()
...         optimizer.zero_grad()
...         progress_bar.update(1)
As you can see in the following code, you only need to add four additional lines of code to your training loop to enable distributed training!
  Copied + from accelerate import Accelerator
  from transformers import AdamW, AutoModelForSequenceClassification, get_scheduler
+ accelerator = Accelerator()
  model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)
  optimizer = AdamW(model.parameters(), lr=3e-5)
- device = torch.d


Backward

evice("cuda") if torch.cuda.is_available() else torch.device("cpu")
- model.to(device)
+ train_dataloader, eval_dataloader, model, optimizer = accelerator.prepare(
+     train_dataloader, eval_dataloader, model, optimizer
+ )
  num_epochs = 3
  num_training_steps = num_epochs * len(train_dataloader)
  lr_scheduler = get_scheduler(
      "linear",
      optimizer=optimizer,
      num_warmup_steps=0,
      num_training_steps=num_training_steps
  )
  progress_bar = tqdm(range(num_training_steps))
  model.train()
  for epoch in range(num_epochs):
      for batch in train_dataloader:
-         batch = {k: v.to(device) for k, v in batch.items()}
          outputs = model(**batch)
          loss = outputs.loss
-         loss.backward()
+         accelerator.backward(loss)
          optimizer.step()
          lr_scheduler.step()
          optimizer.zero_grad()
          progress_bar.update(1)


Train

Once you’ve added the relevant lines of code, launch your training in a script or a notebook like Colaboratory.


Train with a script

If you are running your training from a script, run the following command to create and save a configuration file:
  Copied accelerate config
Then launch your training with:
  Copied accelerate launch train.py


Train with a notebook

🤗 Accelerate can also run in a notebook if you’re planning on using Colaboratory’s TPUs. Wrap all the code responsible for training in a function, and pass it to notebook_launcher:
  Copied >>> from accelerate import notebook_launcher
>>> notebook_launcher(training_function)
For more information about 🤗 Accelerate and its rich features, refer to the documentation.


Text generation strategies

Text generation is essential to many NLP tasks, such as open-ended text generation, summarization, translation, and
more. It also plays a role in a variety of mixed-modality applications that have text as an output like speech-to-text
and vision-to-text. Some of the models that can generate text include
GPT2, XLNet, OpenAI GPT, CTRL, TransformerXL, XLM, Bart, T5, GIT, Whisper.
Check out a few examples that use generate() method to produce
text outputs for different tasks:
Text summarization Image captioning Audio transcription
Note that the inputs to the generate method depend on the model’s modality. They are returned by the model’s preprocessor
class, such as AutoTokenizer or AutoProcessor. If a model’s preprocessor creates more than o


Text generation strategies

ne kind of input, pass all
the inputs to generate(). You can learn more about the individual model’s preprocessor in the corresponding model’s documentation.
The process of selecting output tokens to generate text is known as decoding, and you can customize the decoding strategy
that the generate() method will use. Modifying a decoding strategy does not change the values of any trainable parameters.
However, it can have a noticeable impact on the quality of the generated output. It can help reduce repetition in the text
and make it more coherent.
This guide describes:
default generation configuration common decoding strategies and their main parameters saving and sharing custom generation configurations with your fine-tuned model on 🤗 Hu


Default text generation configuration

A decoding strategy for a model is defined in its generation configuration. When using pre-trained models for inference
within a pipeline(), the models call the PreTrainedModel.generate() method that applies a default generation
configuration under the hood. The default configuration is also used when no custom configuration has been saved with
the model.
When you load a model explicitly, you can inspect the generation configuration that comes with it through
model.generation_config:
  Copied >>> from transformers import AutoModelForCausalLM
>>> model = AutoModelForCausalLM.from_pretrained("distilgpt2")
>>> model.generation_config
GenerationConfig {
    "bos_token_id": 50256,


Default text generation configuration

    "eos_token_id": 50256,
}
Printing out the model.generation_config reveals only the values that are different from the default generation
configuration, and does not list any of the default values.
The default generation configuration limits the size of the output combined with the input prompt to a maximum of 20
tokens to avoid running into resource limitations. The default decoding strategy is greedy search, which is the simplest decoding strategy that picks a token with the highest probability as the next token. For many tasks
and small output sizes this works well. However, when used to generate longer outputs, greedy search can start
producing highly repetitive results


Customize text generation

You can override any generation_config by passing the parameters and their values directly to the generate method:
  Copied >>> my_model.generate(**inputs, num_beams=4, do_sample=True)
Even if the default decoding strategy mostly works for your task, you can still tweak a few things. Some of the
commonly adjusted parameters include:
max_new_tokens: the maximum number of tokens to generate. In other words, the size of the output sequence, not
including the tokens in the prompt. As an alternative to using the output’s length as a stopping criteria, you can choose
to stop generation whenever the full generation exceeds some amount of time. To learn more, check StoppingCriteria. num_beams: by specifying a number of beams higher than 1, you are effectively switching from greedy search to
beam search. This strategy evaluates several hypotheses at each time step and eventually choo


Customize text generation

ses the hypothesis that
has the overall highest probability for the entire sequence. This has the advantage of identifying high-probability
sequences that start with a lower probability initial tokens and would’ve been ignored by the greedy search. do_sample: if set to True, this parameter enables decoding strategies such as multinomial sampling, beam-search
multinomial sampling, Top-K sampling and Top-p sampling. All these strategies select the next token from the probability
distribution over the entire vocabulary with various strategy-specific adjustments. num_return_sequences: the number of sequence candidates to return for each input. This option is only available for
the decoding strategies that support multiple sequence candidates, e.g. variations of beam search and sampling. Decoding
strategies like greedy search and contrastive search return a single output sequence


Save a custom decoding strategy with your model

If you would like to share your fine-tuned model with a specific generation configuration, you can:
Create a GenerationConfig class instance Specify the decoding strategy parameters Save your generation configuration with GenerationConfig.save_pretrained(), making sure to leave its config_file_name argument empty Set push_to_hub to True to upload your config to the model’s repo
  Copied >>> from transformers import AutoModelForCausalLM, GenerationConfig
>>> model = AutoModelForCausalLM.from_pretrained("my_account/my_model")
>>> generation_config = GenerationConfig(
...     max_new_tokens=50, do_sample=True, top_k=50, eos_token_id=model.config.eos_token_id
... )
>>> generation_config.save_pretrained("my_account/my_model", push_to_hub=True)
You can also store several generat


ion configurations in a single directory, making use of the config_file_name
argument in GenerationConfig.save_pretrained(). You can later instantiate them with GenerationConfig.from_pretrained(). This is useful if you want to
store several generation configurations for a single model (e.g. one for creative text generation with sampling, and
one for summarization with beam search). You must have the right Hub permissions to add configuration files to a model.
  Copied >>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, GenerationConfig
>>> tokenizer = AutoTokenizer.from_pretrained("t5-small")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")
>>> translation_generation_config = GenerationConfig(
...     num_beams=4,
...     early_stopping=True,
..


.     decoder_start_token_id=0,
...     eos_token_id=model.config.eos_token_id,
...     pad_token=model.config.pad_token_id,
... )
>>> # Tip: add `push_to_hub=True` to push to the Hub
>>> translation_generation_config.save_pretrained("/tmp", "translation_generation_config.json")
>>> # You could then use the named generation config file to parameterize generation
>>> generation_config = GenerationConfig.from_pretrained("/tmp", "translation_generation_config.json")
>>> inputs = tokenizer("translate English to French: Configuration files are easy to use!", return_tensors="pt")
>>> outputs = model.generate(**inputs, generation_config=generation_config)
>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['Les fichiers de configuration sont faciles à utiliser!'


Streaming

The generate() supports streaming, through its streamer input. The streamer input is compatible with any instance
from a class that has the following methods: put() and end(). Internally, put() is used to push new tokens and
end() is used to flag the end of text generation.
The API for the streamer classes is still under development and may change in the future.
In practice, you can craft your own streaming class for all sorts of purposes! We also have basic streaming classes
ready for you to use. For example, you can use the TextStreamer class to stream the output of generate() into
y


Streaming

our screen, one word at a time:
  Copied >>> from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
>>> tok = AutoTokenizer.from_pretrained("gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> inputs = tok(["An increasing sequence: one,"], return_tensors="pt")
>>> streamer = TextStreamer(tok)
>>> # Despite returning the usual output, the streamer will also print the generated text to stdout.
>>> _ = model.generate(**inputs, streamer=streamer, max_new_tokens=20)
An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,


Decoding strategies

Certain combinations of the generate() parameters, and ultimately generation_config, can be used to enable specific
decoding strategies. If you are new to this concept, we recommend reading this blog post that illustrates how common decoding strategies work.
Here, we’ll show some of the parameters that control the decoding strategies and illustrate how you can use them.


Greedy Search

generate uses greedy search decoding by default so you don’t have to pass any parameters to enable it. This means the parameters num_beams is set to 1 and do_sample=False.
  Copied >>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> prompt = "I look forward to"
>>> checkpoint = "distilgpt2"
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")
>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)
>>> outputs = model.generate(**inputs)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['I look forward to seeing you all again!\n\n\n\n\n\n\n\n\n\n\n']


Contrastive search

The contrastive search decoding strategy was proposed in the 2022 paper A Contrastive Framework for Neural Text Generation.
It demonstrates superior results for generating non-repetitive yet coherent long outputs. To learn how contrastive search
works, check out this blog post.
The two main parameters that enable and control the behavior of contrastive search are penalty_alpha and top_k:
  Copied >>> from transformers import AutoTokenizer, AutoModelForCausalLM
>>> checkpoint = "gpt2-large"
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> model = AutoModelF


Contrastive search

orCausalLM.from_pretrained(checkpoint)
>>> prompt = "Hugging Face Company is"
>>> inputs = tokenizer(prompt, return_tensors="pt")
>>> outputs = model.generate(**inputs, penalty_alpha=0.6, top_k=4, max_new_tokens=100)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Hugging Face Company is a family owned and operated business. We pride ourselves on being the best
in the business and our customer service is second to none.\n\nIf you have any questions about our
products or services, feel free to contact us at any time. We look forward to hearing from you!'


Multinomial sampling

As opposed to greedy search that always chooses a token with the highest probability as the
next token, multinomial sampling (also called ancestral sampling) randomly selects the next token based on the probability distribution over the entire
vocabulary given by the model. Every token with a non-zero probability has a chance of being selected, thus reducing the
risk of repetition.
To enable multinomial sampling set do_sample=True and num_beams=1.
  Copied >>> from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed
>>> set_seed(0)  # For


Multinomial sampling

 reproducibility
>>> checkpoint = "gpt2-large"
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)
>>> prompt = "Today was an amazing day because"
>>> inputs = tokenizer(prompt, return_tensors="pt")
>>> outputs = model.generate(**inputs, do_sample=True, num_beams=1, max_new_tokens=100)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Today was an amazing day because when you go to the World Cup and you don\'t, or when you don\'t get invited,
that\'s a terrible feeling."']


Beam-search decoding

Unlike greedy search, beam-search decoding keeps several hypotheses at each time step and eventually chooses
the hypothesis that has the overall highest probability for the entire sequence. This has the advantage of identifying high-probability
sequences that start with lower probability initial tokens and would’ve been ignored by the greedy search.
To enable this decoding strategy, specify the num_beams (aka number of hypotheses to keep track of) that is greater than 1.
  Copied >>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> prompt = "It is astonis


Beam-search decoding

hing how one can"
>>> checkpoint = "gpt2-medium"
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")
>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)
>>> outputs = model.generate(**inputs, num_beams=5, max_new_tokens=50)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['It is astonishing how one can have such a profound impact on the lives of so many people in such a short period of
time."\n\nHe added: "I am very proud of the work I have been able to do in the last few years.\n\n"I have']


Beam-search multinomial sampling

As the name implies, this decoding strategy combines beam search with multinomial sampling. You need to specify
the num_beams greater than 1, and set do_sample=True to use this decoding strategy.
  Copied >>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, set_seed
>>> set_seed(0)  # For reproducibility
>>> prompt = "translate English to German: The house is wonderful."
>>> checkpoint = "t5-small"
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")
>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)
>>> outputs = model.generate(**inputs, num_beams=5, do_sample=True)
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'Das Haus ist wunderbar.'


Diverse beam search decoding

The diverse beam search decoding strategy is an extension of the beam search strategy that allows for generating a more diverse
set of beam sequences to choose from. To learn how it works, refer to Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models.
This approach has three main parameters: num_beams, num_beam_groups, and diversity_penalty.
The diversity penalty ensures the outputs are distinct across groups, and beam search is used within each group.
  Copied >>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
>>> checkpoint = "google/pegasus-xsum"
>>> prompt = (
...     "The Permaculture Design Principles are a set of universal design principles "
...     "that can be applied to any location, climate and culture, and they allow us to design "
...     "the most efficient and sustainable human habitation and food production systems. "
...     "P