Trends in AI - July 2022

9. ARF: Artistic Radiance Fields  Project page  Code By Kai Zhang, Nick Kolkin, Sai Bi, Fujun Luan, Zexiang Xu, Eli Shechtman, Noah Snavely.  Why -> Sort of like style transfer but for 3D renders.  Key insights -> Neural Radiance Fields (NeRF [9]) are neural networks that can synthesize novel photorealistic views of a given scene. This work presents a NeRF that enables generating views of a scene with arbitrary styles. To do so, they add a styling term to the loss function called Nearest Neighbor Feature Matching (NNFM) which steers the model to produce an output with similar image representations (i.e. VGG features). It's not as simple as it sounds though, a lot of optimization tricks are required to make this work such as deferred back-propagation, which enables training on full resolution images instead of only a subset of image pixels. The authors explain that existing 3D stylization approaches that operate on point clouds or meshes are sensitive to geometric reconstruction errors, which is not the case for the method proposed in this paper, being one of its core strengths. It might be a bit gimmicky but there's definitely artistic potential behind this, especially in the context of Virtual Reality and videogames. The results are too cool to not include them here! You must their videos on their project page to get a better sense of how styled 3D scenes look like.


Trends in AI - July 2022

10. Retrieval Augmented Classification for Long-Tail Visual Recognition By Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, Anton van den Hengel.  Why -> Learning from data in the long tail of a distribution is hard: a model needs to learn from very few instances of such data. Augmenting ML with explicit retrieval modules is a trend we've highlighted in the past which has already improved areas such as text generation. Now it's also applied to image classification.  Key insights -> An input image is routed in parallel through two systems: a regular base image encoder and a retrieval module. The retrieval module encodes the image into an embedding and finds its nearest neighbours in a database of images that are paired with text snippets, which can be labels or longer captions. The list of text snippets from the images is then concatenated and fed into a BERT-like text encoder that produces one text embedding. The embedding from the base image encoder and that from the text encoder are added together after normalization, providing the classification layer of the network with extra information about the image being classified, which is useful in the low data regime. Given that this process is end-to-end differentiable, it can be trained jointly end to end. One of the most fascinating findings is that with an adequate balanced learning objective, the retrieval module learns to focus on tail classes as the base encoder doesn't have enough labeled samples to learn from properly, as can be seen in the figure below (see how the retrieval module-only per class classification accuracy increases for classes with low sample counts).


Multimodal AI -> Combining Text With Images

Overview In this article, we will look at how you can combine the text generation capabilities of GPT-3 with the creative image generation part of DALL.E to produce a piece of art that would have required days if not months, with the conventional setup  Without further ado, let's write a poem on unstructured data in the style of Shakespear using GPT3TextGeneration Executor and generate the illustrations for the same using DALL.E-Flow. We will use the following colab notebook to access the GPT-3 Executor as it will keep all the computing on the cloud, so you don't have to worry about the dependencies  We will take the poem generated by GPT-3 and send it as input to DALL.E Flow to generate the artistic illustrations for our poem. We will use the following notebook to do that


Graphical Illustrations  Following is a line-by-line graphical illustration of the above-generated poem using DALL.E Flow  Line 1 -> "Unstructured data is like a wildflower in a field" Line 2 -> "It's pretty and free" Line 3 -> "But it can be hard to control" Line 4 -> "And it can be tough to find when you're looking for something specific!"


What is GPT-3? GPT-3 is the first-ever generalized language model in the history of natural language processing that can perform equally well on an array of NLP tasks. GPT-3 stands for "Generative Pre-trained Transformer," and it's OpenAI's third iteration of the model. Let us break down these three terms: Generative: Generative models are a type of statistical model that are used to generate new data points. These models learn the underlying relationships between variables in a dataset in order to generate new data points similar to those in the dataset.Pre-trained: Pre-trained models are models that have already been trained on a large dataset. This allows them to be used for tasks where it would be difficult to train a model from scratch. A pre-trained model may not be 100% accurate, but it saves you from reinventing the wheel, saving time, and improving performance.Transformer: A transformer model is a famous artificial neural network invented in 2017. It is a deep learning model that is designed to handle sequential data, such as text. Transformer models are often used for tasks such as machine translation and text classification. GPT-3 is considered the first step by some in the quest for Artificial General Intelligence. To understand how it is revolutionizing the field of AI, check out the most updated Primer on GPT-3!


What is DALL.E Flow? DALL·E Flow is an interactive workflow for generating high-definition images from a text prompt. First, it leverages DALL·E-Mega to generate image candidates and then calls CLIP-as-service to rank the candidates w.r.t. the prompt.


Why Human-in-the-loop? Generative art is a creative process. While recent advances of DALL·E unleash people's creativity, having a single-prompt-single-output UX/UI locks the imagination to a single possibility, which is bad no matter how satisfactory this single result is. DALL·E Flow is an alternative to the one-liner by formalizing the generative art as an iterative procedure. To know more about how DALL.E Flow works, check out the following GitHub Repository.


Prompt Engineering: The Secret Sauce If you have made this far reading the article, you might be thinking about some of these questions  The answer to all your questions lies in a simple term -> Prompt Engineering The Secret to writing good prompts is understanding what these AI models know about the world and how to get the models to use that information for generating useful results. To learn about Prompt Engineering in detail, check out the following resources  Chapter 2 of O'Reilly Book on GPT-3An article on Prompt Engineering by CoherePrompt Engineering: The Career of Future More about Prompt Engineering here  GPT-3 Prompts WikiA Guide to Writing Prompts for Text-to-image AI


Conclusion The future of creative AI is looking very bright. By combining text with images, we can create some truly amazing and unique creations. This is just the beginning of what you can do with this technology, and you can only imagine what the future holds for us!


How to Design a Pre-training Model (TSFormer) For Time Series?

TSFormer It is an unsupervised pre-training model for Time Series based on TransFormer blocks (TSFormer) with the well-implemented Mask AutoEncoder (MAE) strategy.This model is able to capture very long dependencies in our data.


NLP and Time Series: To some extent, NLP information and Time Series data are the same. They both are sequential data and locally sensitive, which means to be in relation to its next/previous data points. BTW, there are some differences which I'm going to say in the following: There are two facts (in fact, these are differences) we should consider while proposing our pre-trained model as we have done for NLP tasks: The density is much lower in Time Series data than in Natural LanguageWe need a longer sequence length of Time Series data than NLP data


TSFormer From ZERO Its process, like all other models, is just like a journey (Nothing new, but a good perspective). As I told you about the strategy of MAE, the main architecture is going through an encoder and then processed into a decoder, finally just to reconstruct the target. You can see its architecture in Figure 1: The process: 1. Masking 2. Encoding 3. Decoding 4. Reconstructing Target


1. Masking This is the first step to providing feed to the next step (Encoder). We can see that the input sequence (Sᶦ) has been distributed into P patches with the length L. Consequently, the length of the sliding window which is used to forecast the next time step is P x L. The making ratio (r) is 75% (relatively high, isn't it?); it is just about making a self-supervised task and makes the Encoder more productive. The main reasons for doing this (I mean patching input sequence) are: Segments (i.e. patches) are better than separate points.It makes it simpler to utilize downstream models (STGNN takes a unit segment as input)By this, we can decrease the size of input for the Encoder.


2. Encoding As you can see, it is an order of Input Embedding, Positional Encoding, and Transformer Blocks. The Encoder can perform just on unmasked patches. Wait!! What was that?? Input Embedding Q) The previous step was about masking and now I said we need unmasked ones??!! A) Input Embedding. Q) How? A) It is actually a linear projection that converts the unmasked to latent space. Its formula can be seen below: W and b are learnable parameters and U are the model input vectors in d hidden dimension. Positional Encoding Simple Positional Encoding layers are used to append new sequential information. The term "Learnable" is added, which helped to show better performance than sinusoidal. Also, learnable positional embedding shows good results for time series. Transformer Blocks The researchers used 4 layers of transformers, just lower than the common amount in computer vision and NLP tasks. The illustration of the type of transformer used here is the most used transformer architecture. You can read it thoroughly in "Attention is all you Need" published in 2017. BTW, I'm going to give you a summary of it(This short illustration comes from one of my previous articles, 'Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting'): Transformers are new deep learning models which are presented at a rising rate. They adopted the mechanism of self-attention and showed a significant increase in model performance on challenging tasks in NLP and Computer Vision. The Transformer architecture can be envisaged into two parts known as encoder and decoder, as it is illustrated in Figure 4, below: The main point of transformers is their independence from localities. In contrast to other popular models like CNNs, transformers are not limited by localization. Also, we do not propose any CNN architecture in transformers; instead, we use Attention-based structures in transformers, which allows us to accomplish better results.Attention Architecture can be summarized in Figure 5: The function of Scaled Dot-Product Attention is Eq. 2 Q (Query), K (Key), and V(Vector) are the inputs of our attention. For a complete fundamental realization of transformers, look at "Attention is all you need." It gives you a great understanding of attention and transformers; in fact, for the first time, I completely understood this important model through this paper.


3. Decoding The decoder includes a series of Transformer Blocks. It works with all sets of patches (masked tokens, etc.). In contrast, MAEs (Masked AutoEncoders), there are no positional embeddings due to patches has already positional information. The number of layers is just one. After that, simple MLPs (Multi-Layer Perceptions) were used (I'd like to be sure that there is no need to illustrate MLPs), which makes the output length equal to each patch.


4. Reconstructing Target Parallelly, the computation over masked patches is for every data point (i). Also, mae (Mean-Absolute-Error) is chosen as the loss function of the main sequence and reconstruction sequence.


How Minerva, a New NLP Model, Performs Better Than GPT-3

The chain of thought processing model proposed by Meier, Cheney, and Seyfarth (1990) suggests that when an animal is faced with a problem, it will first search its memory for a potential solution. For instance: 1. If no solution is found, the animal will then generate a new solution by modifying an existing one. 2. The final step in this process is to evaluate the new solution to see if it is successful. If the new solution is not successful, the animal will go back to the beginning and start again. 3. This cycle continues until a successful solution is found. This model has been supported by research on both animals and humans. For example, studies with rats have shown that when they are faced with a new problem, they will first search their memory for a potential solution. If no solution is found, they will then generate a new solution by modifying an existing one. The final step in this process is to evaluate the new solution to see if it is successful. If the new solution is not successful, the rat will go back to the beginning and start again. This cycle continues until a successful solution is found (Meier, Cheney, & Seyfarth, 1990).


Human examples of a chain of thought processing can be seen in everyday life. For instance: 1. When someone is trying to remember where they left their keys, they will first search their memory for a potential solution. 2. If no solution is found, they will then generate a new solution by modifying an existing one. 3. The final step in this process is to evaluate the new solution to see if it is successful. 4. If the new solution is not successful, the person will go back to the beginning and start again. 5. This cycle continues until a successful solution is found. The chain of thought processing model is a useful tool for understanding how animals and humans solve problems. It can help to explain why some people are better at problem-solving than others, and it can also help to identify potential strategies for improving problem-solving skills.


Scratchpad prompting A language model can be used to generate text that is like a given prompt by predicting the next word in the sequence based on the previous words. Such a model can be used to create paragraphs of text similar to the prompt or generate questions based on the prompt. For example, if the prompt is "The quick brown fox jumped over the lazy dog," the language model might predict the next word as "and," "then," or "." If the prompt is "How are you today?" the language model might predict the next word as "I," "am," or "fine."


Also, please kindly consider subscribing to my weekly newsletter: Sources: [1]: https://ai.googleblog.com/2022/06/minerva-solving-quantitative-reasoning.html [2]: Research paper on Minerva: https://arxiv.org/abs/2206.14858 [3]: Chain of Thought explanation by Google: https://ai.googleblog.com/2022/05/language-models-perform-reasoning-via.html [4]: Chain of Thought research: https://arxiv.org/abs/2201.11903 [5]: Meier, R. D., Cheney, D. L., & Seyfarth, R. M. (1990). Do animals remember the past and plan for the future? Behavioral and Brain Sciences, 13(1), 116.


A New BLOOM in AI? Why the BLOOM Model Can Be a Gamechanger

if you have found it interesting: You can look for my other articles, you can also subscribe to get notified when I publish articles, and you can also connect or reach me on LinkedIn. Thanks for your support! Here, is the link to my Github repository where I am planning to collect code, and many resources related to machine learning, artificial intelligence, and more.


Modern Spam Detection with DistilBERT on NVIDIA Triton

1. Introduction Unwanted and unsolicited bulk digital communication ("spam") is responsible for substantial direct and indirect economic damage every year. Traditional ways to identify spam by detecting certain keywords, manually reviewing text records, or even running Natural Language Processing ("NLP") pipelines are no longer sufficient. This article describes the architecture of a state-of-the-art Spam Detection Engine for social network posts with URL links consisting of multiple inter-dependent distinct classifiers delivering real-time, high-performance, superior accuracy with the minimum required manual review. This article presents a 3-part spam detection system that combines the best available technology approaches and focuses on the DistilBERT model for the NLP classifier. The application of NVIDIA Triton Inference Server for this use case provides the inference throughput 2.4 times higher than AWS TorchScript Inference Server with 52.9 times lower model latency. The article's GitHub Repository is https://github.com/jiripik/nvidia-triton-c-suite-labs-spam-detection-engine.


2. Single-Factor Approaches do NOT work Over time we realized that there is no reliable public API taking into multiple factors to generate an overall spam score. Partial, single-factor solutions simply do not work: Block lists do not work - solutions that block known abusive domains, keywords, IP addresses, or usernames are amazingly easy to bypassText Analysis Approaches do not work - solutions analyzing text in comments, such as OOPSpam, fail to address the text hidden in the preview image or in the destination pageDomain-Based Approaches do not work - solutions aiming at determining if the URL links are safe by studying the reputation of the hosting domain, such as Google Web Risk API, do not work - anybody can upload a video containing spammy text to YouTube, and these services would consider it safe since it's hosted on a trusted domain (YouTube).


3. A Blueprint for a Modern Spam Detection Engine We approach the problem from three distinct angles: User, Message, and Link. Each social network's record comes from a source, i.e., the user, from a device with an IP address following a certain sequence of UI steps. The record's content has a message and may contain a link. Our joint design consists of an ensemble of models: IP & URL Models Poster's IP address - The IP addresses of the source are effective spam indicators, and blocking them over time can significantly reduce spam posts. Our engine identifies the IP addresses consistently posting spam.Link URL Domain - Spam URLs follow similar patterns. Hence if the URL pattern matches previously reported URLs, it is predictive of spam. The key features are:+ url_age: time for which the link has been active on the web+ url_wot_score: Web of Trust score of the URL+ url_google_score: the Google Web Risk score for the URL+ url_domain: the domain of the URL+ url_tld: the network location top-level domain of the URL+ url_subdomain_count: the number of subdomains in the URL+ url_token: the concatenated path tokens extracted from URL textLink URL Context - We run data wrangling techniques on the historical data of the platform and generate such features that correlate to the spam:+ url_post_count: the number of times the URL has been posted+ post_share_count: the number of times the message with the URL is shared+ post_comment_count: the number of responses responses/traction does the post gets+ domain_report_count: the number of times the URL's domain has been reported+ text_content: the content of the message posted with the URL 2. Social Behavior Models Poster Profile - We study features like (1) friends count or (2) the length of time on our platforms, or (3) the user's past activities on the social networking site. If the user profile is similar to other accounts previously reported as spam, there is a high probability it is a spammer user profile. Also, the history of the user might also be indicative of a spam account. Combining these, the profile features that have a correlation to spam are:+ user_age: the time for which the user has an account on the platform+ user_email_domain: the domain of the user's email address which they are registered+ user_friends: the number of friends of the user+ user_followers: the count of followers of the user+ user_verified: a boolean to determine if the user is influential and verified by the platform+ user_posts: the number of posts the user has published on the platform+ user_spam_posts: the number of times the user's post has been reported as spam+ user_spam_report: a boolean to determine if the user has been reported as spam+ user_link_count: the number of times the user has shared this link+ user_post_count: the number of times the user has shared this postPoster Journey - We analyze the sequence of steps the user takes to post the social network's new record. For example, spammers or bots tend to utilize highly efficient workflows.Users' Feedback - We analyze the patterns of other users' complaints about each social network record 3. Content Models NLP analysis - We analyze the user post and the contents of the linked web page, including OCR analysis of any image on the page and the transcript of excerpts of all videos. The model is based upon supervised or semi-supervised learning models, such as BERT and GPT-3. We apply neural network and machine learning-based approaches to develop a series of models for each of these sub-systems. Their outputs are then fed into the primary machine learning model, a decision tree, that outputs the final classification result: The architecture leads to the following insights: If the URL domain is already safely banned, we can reach the immediate conclusion that the new social network's record is spam and stop.Social Behavior Analysis then provides an indication if the user's new post is spam - for example, past complaints, if present, and the poster journey are an extraordinarily strong indicators (about 90% accuracy) that the new post is spam.Finally, we run the NLP analysis on the textual content of the user post and the textual representation of the associated images or videos.


4.1 BERT Models Google's Bidirectional Encoder Representations from Transformers (BERT) base uncased models are the current state-of-the-art models for text classification delivering accuracy of over 97% depending on the quality of the training set. The BERT architecture is based on A Transformer Encoder model (a neural network that takes an input sentence and tokenizes each word, the output is a vector numerical representation of each token). It provides better context understanding than LSTM or RNN since it processes sentences simultaneously. It builds context for a word using the inputs before and after it, while LSTM or RNN only accounts for inputs before it -> BERT has better performance. For example, the word "apple" in "I need the apple" and "I need the apple product" would have the same vector value in LSTM or RNN but are different in BERT.Transfer Learning (train a model for a general task and reuse that to fine-tune BERT to a new task). BERT has been trained on BookCorpus (800M words, 11038 books - same as GPT-1) and the English Wikipedia (2.5B words). It took 4 days on 64 TPUs. Pretraining is slower, but fine-tuning is faster. Sometimes it can be done on a single GPU. BERT has two tasks: Task #1: Masked Language Model (MLM) was originally released by Devlin et al in 2018 at Google. Predicts a random 15% of (sub)word tokens. To regularize, it: Replaces input words with a mask 80% of the time. Replaces the input word with a random token 10% of the time. It leaves it unchanged 10% of the time (but still predicts it)It uses a transformer encoder to do thatThe pretrained inputs to BERT are two separate contiguous sequences Task #2: Next Sentence Prediction (NSP). The MLM does not support the concept of the relationship between sentences. In this task, the model is taught relationships between sentences. A study by Sergio Rojas-Galeano highlighted The BoW (Bag of Words), TFIDF (Term Frequency Inverse Document Frequency), and BERT encoders can extract effective functions to identify spam using widely used classification algorithms, but BERT performs slightly better. This confirms their quoted previous literature.Empirical evidence is that BERT can resist Mad-lib attacks while BoW or TFiDF are vulnerable. It uses an adversarial automatic procedure to generate the attacks. The study by Andrew McCarren and Jennifer Foster shows that BERT outperforms the neural models based on FFNN, CNN, and LSTM (RNNs).


4.2 DistilBERT Models For our Spam Detection engine, we chose its DistilBERT variant, which, according to its authors, is "60% of the size of a BERT model while retaining 97% of its language understanding capabilities and being 60% faster", implying it's also more cost-effective to apply. We used its HuggingFace implementation.


4.3 Training Set Our training set was constructed as a combination of these datasets: Professionally labeled spam datasetsManually curated spam datasetRule-based labeled dataset.


4.4 Training Code The simplified implementation of our DistilBERT code is below and in the article's GitHub repository:


5. Introduction to NVIDIA TensorRT and NVIDIA Triton Inference Server NVIDIA TensorRT™ is an SDK for high-performance deep learning inference, which includes a deep learning inference optimizer and runtime that delivers low latencyhigh throughput for inference applications. TensorRT can be understood as a Deep Learning Compiler that produces an NVIDIA GPU-optimized binary executable ("TensorRT engine"). NVIDIA Triton™ Inference Server is an open-source inference server that helps standardize model deployment and execution and delivers fast and scalable AI in production, supporting most machine learning frameworks, as well as custom C++ and Python code. Triton's Backend: TensorRT is the recommended backend with Triton for GPU optimal inference.The best way to convert your TensorFlow or PyTorch model to TensorRT is to convert it to an ONNX model and then convert the ONNX model to TensorRT. During the ONNX to TensorRT conversion step, the TensorRT Optimizer step runs several optimizations, such as Layer Fusion, which yields a highly GPU-optimized inference model.It is possible to choose another Triton-supported backend, such as TensorFlow, Torchscript, ONNX, etc. The principal reason you should use NVIDIA Triton Inference Server for the deployment of machine learning models is the increased throughput and higher utilization of the hardware compared to the performance provided by other model-serving solutions.


6. AWS Setup We prefer AWS to other public clouds for its reliability, cost-efficiency, and ease of use. We prefer NVIDIA GPUs for training deep learning models for their performance.


6.1 AWS Inference Environments AWS offers two solutions for training and deploying machine learning models: AWS SageMaker Notebook Instances for training and AWS SageMaker Inference Endpoints for inferenceAWS EC2 instances for training and AWS ECS for inference Each solution has its advantages and disadvantages: In addition, we strongly advise against using outputs from different versions of PyTorch / TensorFlow as inputs to another version of PyTorch / TensorFlow, for example, trying to import one from an EC2 Instance into a SageMaker Notebook. In most such cases, you will receive an error message only at the end of the ML pipeline - NVIDIA Triton's SageMaker Endpoint will fail to initialize.


6.2 Deployment of the model to NVIDIA Triton Inference Server We present below the deployment of the described model using AWS SageMaker Notebook Instance to AWS SageMaker Inference Endpoint running the Docker image of NVIDIA Triton Inference Server. We describe the steps outlined in the solution's Jupyter Notebook along with the expected output.


6.2.1 Step 0 - Setting up the AWS SageMaker Notebook Instance AWS offers multiple compute instance types suitable for deep learning. Check this guide to decide which one is the most suitable for your use case. We used this instance: Notice: We increased the required disk space to a safe value of 1 TB.We do not need to use Elastic Inference - it is used to add a fractional GPU for accelerating inference at cheaper cost. In our case, we are already using the GPU instance of ml.p3.2xlarge which comes with a whole V100 GPU -> we don't need the elastic inference.We used the platform identifier of Amazon Linux 2, Jupyter Lab 1. Using a different value may cause version conflicts.The Jupyter Lab's kernel to be used for the solution notebook is "conda_amazonei_pytorch_latest_p37". Using a different kernel may cause version conflicts.


6.2.2 Step 1 - Installation of the model required libraries Notice: We have upgraded the PyTorch library to the latest version. Without this step, we would not be able to save to model the ONNX file and deploy to NVIDIA Triton Inference Server. This upgrade causes minor library version conflicts - we do not depend on them:


6.2.3 Step 2 - Train the model The code has been described in section 4.4. Usually, we would add a step of evaluating the model on a larger test dataset. We do not share the dataset with the solution, so we omit.


6.2.4 Step 3 - Generate the ONNX file The ONNX file is generated by the script in workspace-trt/generate_models which needs to be present in the solution. trtexec is TensorRT's command line tool for building a .plan optimized TensorRT model file from an onnx file. Its parameter saveEngine (here model_bs16.plan) is used to specify the output engine's name. You can learn more by doing trtexec -help inside the PyTorch NGC container. Notice the parameters of the TensorRT's trtexec command specifying the shape of the inputs and outputs: Batch size When we specified minShapes as 1×128, optShapes as 1×128 and maxShapes as 1×128, we were defined as a fixed batch size meaning the model can only accept batch sizes 1.However, TensorRT supports building optimized engines with dynamic shapes, so we can also, for example, specify minShapes as 1×128, optShapes as 16×128 and maxShapes as 128×128, which means the model can accept batch sizes between 1 and 128 and is optimized for batch size 16. 2. Sequence length It is represented by the second dimension in these shape profiles. In this example, it's fixed at 128 (but you can change it to whatever sequence length is best for your model). WARNING: Only the ONNX to TensorRT conversion step needs to happen on the same GPU as the deployment GPU. This means if you are deploying on an endpoint with g4dn.xlarge (which has the T4 GPU), then you need to make sure that ONNX to TensorRT conversion step (which we also refer to as building the TensorRT engine) needs to be on the T4 GPU (g4dn instance). Everything else, from training the model to exporting the model to ONNX can happen on the training GPU, which can be a p3 instance, p4 instance or even g4dn instance. The supported parameters of the trtexec command depend on its version: The PyTorch 21.08 NGC container has TensorRT 8.0.1.6, which cannot support dynamic shapes for DistilBERT model. This limitation was fixed in recent TensorRT version 8.2.4.2 available in PyTorch 22.04 NGC container.Using the most recent PyTorch NGC container in the current SageMaker Inference Endpoint version is, however, not possible. To run the TensorRT model successfully in Triton we also need to update to newer Triton version like 22.05, which has the TensorRT version 8.2.4.2 or later. Currently, the latest Triton container version that SageMaker supports is 21.08, which has the old TensorRT version 8.0.1.6, but we need TensorRT v8.2.4.2 or later, which is available in recent NGC Triton 22.5 container. Unfortunately, the SageMaker team doesn't yet have a concrete release date for when they will officially add the new Triton 22.05 container in SageMaker.


6.2.5 Step 4 - Create SageMaker model package and upload it to SageMaker This script depends on the existence of the model config file triton-serve-trt/bert/config.pbtxt defining the model for NVIDIA Triton Inference Server, which needs to be present in the solution.


6.2.6 Step 5 - Create SageMaker Inference endpoint The code finds the NVIDIA Triton Inference Server's closest location in AWS ECR and deploys it to the AWS SageMaker Inference Endpoint. The deployment should take less than 10 mins. Notice: In the endpoint configuration, we define the endpoint instance type - it must be identical to the instance used for creating the TensorRT model. Creating the TensorRT model (engine) from ONNX model i.e. ONNX to TensorRT conversion step, needs to happen on the same GPU as the target deployment GPU. So if you want to deploy TensorRT model on T4 GPU, which is in g4dn instance then you build the TensorRT engine on g4dn instance. Similarly, if you are deploying the model on V100 GPU which is found in p3 instance, then you need to build the TensorRT engine from onnx model on p3 instance. Currently, we can't relax this hard requirement of TensorRTThe deployment log is available from the AWS SageMaker Inference Endpoint (for us-east-1). If the creation takes more than 10 mins, review them. If it contains an error, the deployment fails after more than 30 mins and then you can delete the endpoint.


6.2.7 Step 6 - Test Triton SageMaker Inference Endpoint While Triton does support batch inference for all its supported backends, for this model we were restricted to the batch size of one because the current SageMaker Triton container's available version of TensorRT doesn't support dynamic shapes for the DistilBERT model. This limitation of TensorRT has been fixed in its newer releases and should be available in the SageMaker Triton container soon.


6.2.8 Step 7 - Delete the SageMaker Inference Endpoint After we finish the testing of the solution, we can delete the SageMaker inference endpoint, endpoint configuration and the model.


7. Performance Comparison of NVIDIA Triton-based and TorchScript-based AWS SageMaker Inference For illustrative purposes, we have implemented identical code using AWS SageMaker Triton Inference endpoint and AWS SageMaker TorchScript Inference endpoint: We have repeated an inference for four test texts one thousand times using ThreadPoolExecutorWhile the results are approximate only, they should be robust enough: NVIDIA Triton Inference Server delivers 2.4 times higher throughput with 52.9 times lower model latency than TorchScript Inference Server. Inference TypeThroughputModel LatencyNVIDIA Triton Inference Server339.35 texts/sec1.48 msTorchScript Inference Server 140.4 texts/sec78.3 ms Compare the AWS CloudWatch's ModelLatency for both inference servers: The main speed boost comes from the TensorTRT acceleration compared to the framework performance as well as the efficient kserve v2 protocol in Triton. It is possible to increase the performance even more by trying out these optimizations provided natively by Triton.


Scalable and Blazing Fast Similarity Search With Milvus Vector Database

Introduction In this article, we will cover some interesting aspects relevant to vector databases and similarity search at scale. In today's rapidly evolving world, we see new technology, new businesses, new data sources and consequently we will need to keep using new ways to store, manage and leverage this data for insights. Structured, tabular data has been stored in relational databases for decades, and Business Intelligence thrives on analyzing and extracting insights from such data. However, considering the current data landscape, organizations have been leveraging the power of machine learning and deep learning to try and extract insights from such data as traditional query-based methods may not be enough or even possible. There is a huge, untapped potential to extract valuable insights from such data and we are only getting started! Unstructured data, as the name suggests, does not have an implicit structure, like a table of rows and columns (hence called tabular or structured data). Unlike structured data, there is no easy way to store the contents of unstructured data within a relational database. There are three main challenges with leveraging unstructured data for insights: Storage: Regular relational databases are good for holding structured data. While you can use NoSQL databases to store such data, it becomes an additional overhead to process such data to extract the right representations to power AI applications at a scaleRepresentation: Computers don't understand text or images as we do. They only understand numbers and we need to convert unstructured data into some useful numeric representation, typically vectors or embeddings.Querying: You can't query unstructured data directly based on definite conditional statements like SQL for structured data. Imagine, a simple example of you trying to search for similar shoes given a photo of your favorite pair of shoes! You can't use raw pixel values for search, nor can you represent structured features like shoe shape, size, style, color, and more. Now imagine having to do this for millions of shoes! Hence, in order for computers to understand, process, and represent unstructured data, we typically convert them into dense vectors, often called embeddings. There exists a variety of methodologies especially leveraging deep learning, including convolutional neural networks (CNNs) for visual data like images and Transformers for text data which can be used to transform such unstructured data into embeddings. Zilliz has an excellent article covering different embedding techiques! Now storing these embedding vectors is not enough. One also needs to be able to query and find out similar vectors. Why do you ask? A majority of real-world applications are powered by vector similarity search for AI-based solutions. This includes visual (image) search in Google, recommendations systems in Netflix or Amazon, text search engines in Google, multi-modal search, data de-duplication, and many more! Storing, managing, and querying vectors at scale is not a simple task. You need specialized tools for this and vector databases are the most effective tool for the job! In this article we will cover the following aspects: Vectors and Vector Similarity SearchWhat is a Vector Database?Milvus - The World's Most Advanced Vector DatabasePerforming visual image search with Milvus - A use-case blueprint Let's get started!


Vectors and Vector Similarity Search Earlier, we established the necessity of representing unstructured data like images and text as vectors, since computers can only understand numbers. We typically leverage AI models, to be more specific deep learning models to convert unstructured data into numeric vectors which can be read by machines. Typically these vectors are basically a list of floating-point numbers which collectively represent the underlying item (image, text, etc.).


Understanding Vectors Considering the field of natural language processing (NLP) we have many word embedding models like Word2Vec, GloVe, and FastText which can help represent words as numeric vectors. With advancements over time, we have seen the rise of Transformer models like BERT which can be leveraged to learn contextual embedding vectors and better representations for entire sentences and paragraphs. Similarly, in the field of computer vision, we have models like Convolutional Neural Networks (CNNs) which can help in learning representations from visual data such as images and videos. With the rise of Transformers, we also have Vision Transformers which can perform better than regular CNNs. The advantage with such vectors is that we can leverage them for solving real-world problems such as visual search, where you typically upload a photo and get search results including visually similar images. Google has this as a very popular feature in their search engine as depicted in the following example. Such applications are powered by data vectors and vector similarity search. If you consider two points in an X-Y cartesian coordinate space. The distance between two points can be computed as a simple euclidean distance depicted by the following equation. Now imagine each data point is a vector having D-dimensions, you could still use euclidean distance or even other distance metrics like hamming or cosine distance to find out how close the two data points are to each other. This can help build a notion of closeness or similarity which could be used as a quantifiable metric to find similar items given a reference item using their vectors.


Understanding Vector Similarity Search Vector similarity search, often known as nearest neighbor (NN) search, is basically the process of computing pairwise similarity (or distances) between a reference item (for which we want to find similar items) and a collection of existing items (typically in a database) and returning the top 'k' nearest neighbors which are the top 'k' most similar items. The key component to compute this similarity is the similarity metric which can be Euclidean distance, inner product, cosine distance, Hamming distance, etc. The smaller the distance, the more similar are the vectors. The challenge with the exact nearest neighbor (NN) search is scalability. You need to compute N-distances (assuming N existing items) every time to get similar items. This can be super slow especially if you don't store and index the data somewhere (like a vector database!). To speed up computation, we typically leverage approximate nearest neighbor search which is often called ANN search which ends up storing the vectors into an index. The index helps in storing these vectors in an intelligent way to enable quick retrieval of 'approximately' similar neighbors for a reference query item. Typical ANN indexing methodologies include: Vector Transformations: This includes adding additional transformations to the vectors like dimension reduction (e.g PCA \ t-SNE), rotation, and so onVector Encoding: This includes applying techniques based on data structures like Locality Sensitive Hashing (LSH), Quantization, Trees, etc. which can help in faster retrieval of similar itemsNon-Exhaustive Search Methods: This is mostly used to prevent exhaustive search and includes methods like neighborhood graphs, inverted indices, etc. This establishes the case that to build any vector similarity search application, you need a database that can help you with efficient storing, indexing, and querying (search) at scale. Enter vector databases!


What is a Vector Database? Given that we now understand how vectors can be used to represent unstructured data and how vector search works, we can combine the two concepts together to build a vector database. Vector databases are scalable data platforms to store, index, and query across embedding vectors that are generated from unstructured data (images, text, etc.) using deep learning models. Handling a massive number of vectors for similarity search (even with indices) can be super expensive. Despite this, the best and most advanced vector databases should allow you to insert, index, and search across millions or billions of target vectors, in addition to specifying an indexing algorithm and similarity metric of your choice. Vector databases mainly should satisfy the following key requirements considering a robust database management system to be used in the enterprise: Scalable: Vector databases should be able to index and run an approximate nearest neighbor search for billions of embedding vectorsReliable: Vector databases should be able to handle internal faults without data loss and with minimal operational impact, i.e be fault-tolerantFast: Query and write speeds are important for vector databases. For platforms such as Snapchat and Instagram, which can have hundreds or thousands of new images uploaded per second, speed becomes an incredibly important factor. Vector databases don't just store data vectors. They are also responsible for using efficient data structures to index these vectors for fast retrieval and supporting CRUD (create, read, update and delete) operations. Vector databases should also ideally support attribute filtering which is filtering based on metadata fields which are usually scalar fields. A simple example would be retrieving similar shoes based on the image vectors for a specific brand. Here brand would be the attribute based on which filtering would be done. The figure above showcases how Milvus, the vector database we will talk about shortly, uses attribute filtering. Milvus introduces the concept of a bitmask to the filtering mechanism to keep similar vectors with a bitmask of 1 based on satisfying specific attribute filters. More details on this are here.


Milvus - The World's Most Advanced Vector Database Milvus is an open-source vector database management platform built specifically for massive-scale vector data and streamlining machine learning operations (MLOps). Zilliz, is the organization behind building Milvus, the world's most advanced vector database, to accelerate the development of next-generation data fabric. Milvus is currently a graduation project at the LF AI & Data Foundation and focuses on managing massive unstructured datasets for storage and search. The platform's efficiency and reliability simplify the process of deploying AI models and MLOps at scale. Milvus has broad applications spanning drug discovery, computer vision, recommendation systems, chatbots, and much more.


Key Features of Milvus Milvus is packed with useful features and capabilities, such as: Blazing search speeds on a trillion vector datasets: Average latency of vector search and retrieval has been measured in milliseconds on a trillion vector datasets.Simplified unstructured data management: Milvus has rich APIs designed for data science workflows.Reliable, always on vector database: Milvus' built-in replication and failover/failback features ensure data and applications can maintain business continuity always.Highly scalable and elastic: Component-level scalability makes it possible to scale up and down on demand.Hybrid search: In addition to vectors, Milvus supports data types such as Boolean, String, integers, floating-point numbers, and more. Milvus pairs scalar filtering with a powerful vector similarity search (as seen in the shoe similarity example earlier).Unified Lambda structure: Milvus combines stream and batch processing for data storage to balance timeliness and efficiency.Time Travel: Milvus maintains a timeline for all data insert and delete operations. It allows users to specify timestamps in a search to retrieve a data view at a specified point in time.Community-supported & Industry recognized: With over 1,000 enterprise users, 10.5K+ stars on GitHub, and an active open-source community, you're not alone when you use Milvus. As a graduate project under the LF AI & Data Foundation, Milvus has institutional support.


Existing Approaches to Vector Data Management and Search A common way to build an AI system powered by vector similarity search is to pair algorithms like Approximate Nearest Neighbor Search (ANNS) with open-source libraries such as: Facebook AI Similarity Search (FAISS): This framework enables efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It supports indexing capabilities like inverted multi-index and product quantizationSpotify's Annoy (Approximate Nearest Neighbors Oh Yeah): This framework uses random projections and builds up a tree to enable ANNS at scale for dense vectorsGoogle's ScaNN (Scalable Nearest Neighbors): This framework performs an efficient vector similarity search at scale. Consists of implementations, which include search space pruning and quantization for Maximum Inner Product Search (MIPS) While each of these libraries is useful in its own way, due to several limitations, these algorithm-library combinations are not equivalent to a full-fledged vector data management system like Milvus. We will discuss some of these limitations now.


Limitations of Existing Approaches Existing approaches used for managing vector data as discussed in the previous section have the following limitations: Flexibility: Existing systems typically store all data in main memory, hence they cannot be run in distributed mode across multiple machines easily and are not well-suited for handling massive datasetsDynamic data handling: Data is often assumed to be static once fed into existing systems, complicating processing for dynamic data and making near real-time search impossibleAdvanced query processing: Most tools do not support advanced query processing (e.g., attribute filtering, hybrid search, and multi-vector queries), which is essential for building real-world similarity search engines supporting advanced filtering.Heterogeneous computing optimizations: Few platforms offer optimizations for heterogeneous system architectures on both CPUs and GPUs (excluding FAISS), leading to efficiency losses. Milvus attempts to overcome all of these limitations and we will discuss this in detail in the next section.


The Milvus Advantage -Understanding Knowhere Milvus tries to tackle and successfully solve the limitations of existing systems build on top of inefficient vector data management and similarity search algorithms in the following ways: It enhances flexibility by offering support for a variety of application interfaces (including SDKs in Python, Java, Go, C++, and RESTful APIs)It supports multiple vector index types (e.g., quantization-based indexes and graph-based indexes), and advanced query processingMilvus handles dynamic vector data using a log-structured merge tree (LSM tree), keeping data insertions and deletions efficient and searches humming along in real-timeMilvus also provides optimizations for heterogeneous computing architectures on modern CPUs and GPUs, allowing developers to adjust systems for specific scenarios, datasets, and application environments Knowhere, the vector execution engine of Milvus, is an operator interface for accessing services in the upper layers of the system and vector similarity search libraries like Faiss, Hnswlib, and Annoy in the lower layers of the system. In addition, Knowhere is also in charge of heterogeneous computing. Knowhere controls on which hardware (eg. CPU or GPU) to execute index building and search requests. This is how Knowhere gets its name - knowing where to execute the operations. More types of hardware including DPU and TPU will be supported in future releases. Computation in Milvus mainly involves vector and scalar operations. Knowhere only handles the operations on vectors in Milvus. The figure above illustrates the Knowhere architecture in Milvus. The bottom-most layer is the system hardware. The third-party index libraries are on top of the hardware. Then Knowhere interacts with the index node and query node on the top via CGO. Knowhere not only further extends the functions of Faiss but also optimizes the performance and has several advantages including support for BitsetView, support for more similar metrics, support for AVX512 instruction set, automatic SIMD instruction selection, and other performance optimizations. Details can be found here.


Milvus Architecture The following figure showcases the overall architecture of the Milvus platform. Milvus separates data flow from control flow and is divided into four layers that are independent in terms of scalability and disaster recovery. Access layer: The access layer is composed of a group of stateless proxies and serves as the front layer of the system and endpoint to users.Coordinator service: The coordinator service is responsible for cluster topology node management, load balancing, timestamp generation, data declaration, and data managementWorker nodes: The worker, or execution, node executes instructions issued by the coordinator service and the data manipulation language (DML) commands initiated by the proxy. A worker node in Milvus is similar to a data node in Hadoop, or a region server in HBaseStorage: This is the cornerstone of Milvus, responsible for data persistence. The storage layer is comprised of meta store, log broker and object storage Do check out more details about the architecture here!


Performing visual image search with Milvus - A use-case blueprint Open-source vector databases like Milvus make it possible for any business to create their own visual image search system with a minimum number of steps. Developers can use pre-trained AI models to convert their own image datasets into vectors, and then leverage Milvus to enable searching for similar products by image. Let's look at the following blueprint of how to design and build such a system. In this workflow, we can use an open-source framework like towhee to leverage a pre-trained model like ResNet-50 and extract vectors from images, store and index these vectors with ease in Milvus and also store a mapping of image IDs to the actual pictures in a MySQL database. Once the data is indexed we can upload any new image with ease and perform an image search at scale using Milvus. The following figure shows a sample visual image search.


Conclusion We've covered a fair amount of ground in this article. We started with challenges in representing unstructured data, leveraging vectors, and vector similarity search at scale with Milvus, an open-source vector database. We discussed details on how Milvus is structured and the key components powering it and a blueprint of how to solve a real-world problem, visual image search with Milvus. Do give it a try and start solving your own real-world problems with Milvus!


How To Train a Seq2Seq Summarization Model Using "BERT" as Both Encoder and Decoder!! (BERT2BERT)

Results We can see the BERT-to-BERT model performance that is fine-tuned on the CNN/DM dataset. I used the available checkpoint on the dataset hub using the Beam Search decoding method. The results are calculated using the ROUGE score metric. The BART model is the SOTA model in text summarization, and the BERT seq2seq network holds up pretty well! There is only a 1% difference that usually will not translate to a huge change in sentence quality.


Final Words, The mix and match approach can result in exciting experiments. For example, it is possible to connect BERT to GPT-2 to use the BERT's power to create a powerful representation of the texts and the GPT's ability to generate quality sentences. It is a good practice to use different networks for your custom datasets before choosing the SOTA model for all problems. The main difference between using BERT (compared to BART) is the 512 tokens input sequence length limitation (compared to 1024). So, it makes the BERT-to-BERT model a good choice if your dataset's input sequences are smaller. It would be more efficient to train smaller models and require fewer resources such as data and GPU memory.


ArgMiner: End-to-End Argument Mining

ArgMiner in a Nutshell The main features of ArgMiner are summarised below: Provides processors to extract SOTA datasets from the source without writing any extra line of codeProcessors can generate the following labeling methods {io, bio, bieo, bixo} at both word and subtoken levels with no extra line of codeProcessors have an inbuilt feature that enables custom augmentations without a change to the data processing pipelineProvides a PyTorch Dataset for argument mining fine-tuning with any HuggingFace TokenClassification modelProvides a pipeline for training and inference that is efficient The image below shows ArgMiner working end-to-end:


ArgMiner: End-to-End Argument Mining

Datasets Argument Annotated Essays (AAE): this is a collection of 402 essays found on the online essay feedback forum, essayforum. It has 3 argument components: Claim, MajorClaim, Premise. The dataset can be found in the TUDarmstadt datastore; the original paper is on the ACL, the subsequent paper is on the MIT Press Direct.PERSUADE: this is a collection of around 15000 essays written by U.S. school students in grades 612. It has 7 argument components: Lead, Position, Claim, Counterclaim, Rebuttal, Evidence, and Concluding Statement. The dataset can be accessed through the Kaggle Feedback Prize competition.ARG2020: this is a collection of 145 middle school student essays. It has 2 argument components: Claim and Premise. The dataset is publicly available on GitHub, and the paper for this work is on ArXiv. These datasets are stored and processed differently. For example, the AAE and ARG2020 datasets have .ann files that contain two types of information: 1) argument components, which are of the form: [argument_component_id, argument_component_text, span_start, span_end, text] and 2) argument relations, which are of the form: [relation_id, relation_type, argument_component_1, argument_component_2]. Argument Relations are not required for argument mining so these need to be removed. Further, the actual essay_ids are the names of the files themselves, so the code to parse this needs to take it into account. The data comes with .txt files for the raw essay texts as well. Unlike ARG2020, the AAE dataset comes with train-test IDs to use for splitting the data. On the other hand, the PERSUADE dataset has a more complicated directory structure, with train and test directories that include the raw .txt essays. The actual information about the argument components is contained in train.csv, which has the following form: [essay_id, discourse_id, span_start, span_end, discourse_text, discourse_type, discourse_type_num, predictionstring]. None of the datasets actually indicate parts of the essay that are not argument components, so-called "Other" classes. However, NER problems typically require this (else you are selectively viewing information from the essay and not the whole essay). So the pipeline needs to extract these segments from the essays themselves. In order to process these widely varying raw texts in a standardized format, ArgMiner adopts 3 phases: preprocess, process, and post-process. These are described in greater detail in the next sections.


ArgMiner: End-to-End Argument Mining

Preprocessing: Extracting Data From Source This step takes the data in the raw format (for each dataset) and uses the span_start and span_end features and the raw texts to generate a DataFrame with the following structure: [essay_id, text, argument_component]. This enables a standard method of moving forward to generate the NER labels, or augmenting data. The processors are all based on a base DataProcessor class that has inbuilt features for saving and applying train-test-split, so new processors can be created from it with ease.


ArgMiner: End-to-End Argument Mining

Processing: Generating Labels and (optional) Augmenting Data Now that the data is in a standard format, the user can request to generate the NER style labels for the data. By the end of this step, the dataset will look like this: [essay_id, text, argument_component, NER_labels]. However, sometimes one may be interested in augmenting the data, either for adversarial training or testing robustness on adversarial examples. In this case, the user can simply provide any function that takes a piece of text and return an augmented piece of text. This means that other NLP augmentation libraries such as textattack and nlpaug can be used with ease. When an augmenter is used, the data is first augmented before the


ArgMiner: End-to-End Argument Mining

Postprocessing: Aggregating Sequences To Documents The final step is very simple now that the labels have been created. Here, the task is to simply join the segments by their doc_ids. The resulting output from this stage is a DataFrame of the form: [essay_id, full_essay_text, NER_labels]. Doing train-test splits is also very easy using the inbuilt train and test features.


ArgMiner: End-to-End Argument Mining

PyTorch Dataset I won't go into detail about how the code of this works but will highlight why this is great: The PyTorch dataset is designed to take the inputs from the .postprocess() stage. At this point, the variable strategy_level can determine if the labeling strategy should apply to a word or token level (see the end of section).The dataset expands the class labels to the subtokens. This is a huge improvement from the examples on Kaggle because it is vectorized and can effectively use the GPU.The dataset creates a mapping to reduce the expanded labels to their core labels for inference (e.g. 'B-Claim, I-Claim, E-Claim' all get reduced to claim for the purpose of inference, if unsure, I would highly recommend my tutorial on NER inference) It is also very simple to use, and since it is PyTorch based you can easily integrate it for training. As an example:


ArgMiner: End-to-End Argument Mining

Inference ArgMiner also provides functions for training models and running inference. However, you can also write your own training loops. The defining feature of ArgMiner is that the inference functions are written to be efficient (where they make use of the GPU and vectorization where possible) and batched (thus ideal for low memory settings), meaning that the inference function can be used during training against validation data as well. The inference loop can be very helpful for training data debugging purposes as well because it allows you to add other metrics that you want to measure. These are then applied on tensors and on tensors after reduction from subtokens to words, which can give you an indication as to where exactly are you losing performance. Another feature that sets this inference loop apart from others that I've seen is the ability to easily choose the level of aggregation when mapping back from tokens to words. For example, given two tokens "Unit" and "ed" with probabilities for each class, you can aggregate them into "United" using just the best probability for the word "Unit", the best mean probability, or the best max probability. Overall, this inference scheme offers several advantages compared with the ones used in the Feedback Prize competition.


ArgMiner: End-to-End Argument Mining

Web App ArgMiner comes with a web app for seeing what outputs your models give (or any model from HuggingFace), but also for evaluating the performance of models on custom datasets that you write yourself. This is a useful (informal) way of probing a model on specific examples to see what it is doing.


ArgMiner: End-to-End Argument Mining

Concluding Remarks For a long time, Argument Mining literature has been very thin on datasets, but this has changed with the releases of PERSUADE and ARG2020Much research needs to be done on the transfer of knowledge in argument mining, as well as robustness. This is typically difficult from a data processing side of things because of different source data formats, multiple methods of representing data, and efficiency issues due to working with non-equal segments for representation and inferenceArgMiner is a package in Early Release Access that standardizes processing, augmenting, training, and performing inference on SOTA Argument Mining datasets


ArgMiner: End-to-End Argument Mining

Some Limitations and Ongoing Work While the core of the package is ready, there are some bits and pieces that need to be ironed out, and some missing features. These are described below: A DataProcessor for the ARG2020 datasetThe augmentation feature at the moment can only have the text as input. Further improvements will look to standardizing other features as well, for example, if augmentations are to be done on specific classes only. This will be of great use for model understanding, specifically to determine how specific transformations can affect the model outputExtending the DataProcessor class to allow for stratified train-test splits, and splits based on other features, e.g. length for curriculum learning, a class for zero-shot learning


ArgMiner: End-to-End Argument Mining

Call for NLP Enthusiasts If you are interested in Argument Mining and NLP, and you found this article interesting, I'd be happy to chat and add you as a collaborator on the project. I'm hoping to take this project much further, beyond the Early Access Release and refactor some of the codebase into more generic NLP util libraries, as well as NLP diagnosing and data robustness libraries/apps (I'm interested in incorporating the following projects: Data Cartography and Information-Theoretic Dataset Difficulty). These are long-term plans that I'm very excited about, but cannot do as a sole developer. If you are interested, please reach out to me via LinkedIn.


A Journey in Building Data-Centric Autonomous AI

Introduction   The vision for Vic.ai is ambitious and far-reaching: building an autonomous AI that will entirely replace the manual accounting workflow for processing invoices. This reminded me of the initial reason for inventing computers: replacing the work of human "computers" whose occupation was to meticulously calculate and correct numbers. Like the human computers a century ago, accountants today increasingly work through volumes of invoices, instead of tasks requiring creative problem solving and human interaction. To an ML scientist, this problem had a real-world impact, and raised questions: how to build the AI, and if this type of task could be made autonomous, what other types of tedious work could be automated using autonomous AI? The role of the initial "Head of Machine Learning" in early-growth startups is considerably different than in larger companies and involves wearing many hats. During my tenure, I managed all of the data science projects, designed the ML algorithms Vic uses today, and hired the current ML and backend engineers. Until the last year, I did all of the analysis to drive the AI development forward, the technology choices for building the AI stack, trained the deployed models and did a fair share of the analysis to resolve prediction errors. This workload sounds exhausting, but in the early phase each of these takes limited time, and improved software has greatly reduced these times. As someone who's worked in AI research, startups, and consulting for the past two decades, Vic.ai struck me as unique in many ways. The usual pitfalls of AI projects were absent, and the work could focus on the core challenges of building an AI product. Many of these remaining challenges weren't fully recognized by AI research or industry. I've decided to share the learnings gained along this journey, as these challenges are common to most organizations building AI, and this experience can be of interest to others.


CRISP-DM Meets DevOps Culture Before Vic, my commercial AI experience was as an external consultant, and as an in-house ML developer for several startups. It hadn't taken me long to notice the mismatch between managing AI and software development projects. The majority of AI development projects were conducted with the CRISP-DM framework originating in data mining consulting, resulting in a high success rate and fast turnaround for clients. Software development was done with Scrum-influenced practices, with less than half of the projects succeeding. The two solved different problems and didn't fit together. CRISP-DM is non-linear, with undetermined outcomes. Scrum is linear, with planned outcomes. Most commonly, AI projects were prototyped and delivered for production use, but the organizational support failed to turn these into successful products. Like most of the AI community, I got increasingly pessimistic about commercial AI due to these organizational problems, and the complete mismatch between CRISP-DM and Scrum. Encountering Vic changed this, as the company had an exceptionally professional early developer team. Many of them had CTO-level experience or continued to a CTO role from Vic. Engineering and the rest of the company practiced DevOps culture and unlike Scrum applied the Agile #1 value "people before processes". After close encounters with Scrum, this changed my perception of software engineering. The ML efforts had the complete support of the company, and I was fortunate that the CTO had a passion for ML, having led ML projects for the past few years. There had been some temporary contractors who had done some initial AI work for Vic. Some of the organizational planning was solved before I started, and there was implicitly already a type of "Agile CRISP-DM" framework in practice. My team consisted of 24 experienced backend and ML developers for the first three years, and this was the ideal team size for building the foundation of the AI codebase. Later, I expanded this into an 8-person MLOps team with more specialized roles. The work was asynchronous and fully remote, with typically weekly or biweekly meetings over a Kanban board. Areas of responsibility were clearly defined as per Conway's Law, with the main splits being between backend/ML and dev/ops. The team worked on a DevOps basis, both developing and maintaining the code in production with regular release cycles. If there were requests from other teams, these were addressed first, so that their work was unblocked. Our requests to other teams were handled likewise, usually within a day or two, so that company-wide goals progressed rapidly. This prevented silo effects from blocking projects and allowed us to consider new projects since we could trust to have rapid support and prioritization from other teams. Typically around half of any member's work time went to development, and the other half to responding to issues and requests. The goals and milestones for the team were negotiated at the company level, and projects were handled autonomously by the team. This meant that tasks were often rapidly added, changed, and reprioritized between sync calls. Early termination, parallel tasks, task halting, and even failures were all normal parts of the process. In the Scrum context, these would be unthinkable, but for developing AI this is just the normal routine. What mattered was reaching the long-term goals, not the outcome of individual tasks. Another aspect of this framework was fully data-driven development for AI. Instead of executives mandating "use deep learning to solve X", every problem went through the same process. The best solution for the problem could be string matching completed within a day, a complex AI solution taking several months to develop, or reporting that the problem would need to be revised. All of these were common outcomes. Using this approach, the best solutions for problems were achieved at a fast rate of completion. The overall outcome was an AI system that combined techniques from traditional software and AI, selected by data-driven analysis for each case.


Data-centric AI Some time after developing my first models, I got feedback about one. Reported accuracy from initial testing was around 30%. This was stunning, as we trusted the development metrics, which gave good results for the model. Looking into the details, it seemed there were multiple data quality issues for any type of AI trained from this data. I understood that this was a technical problem and we could still produce high-accuracy models, by treating the data preprocessing and model development as a single problem. I started working on solutions for this type of AI, like many others who discovered the limits of model-centric AI, and around this time the field was named "data-centric AI". Developing labeling processes through "label engineering" was a critical type of data-centric AI for us. There are various data quality problems in accounting platforms, but the most severe for AI is that label information is often uncertain, incomplete, incorrect, outdated, or available as indirectly matchable meta-data. As a thumb rule, 20% of labels are missing or have wrong values. Supervised ML trained on the data requires correct label information, or it reproduces the errors seen in training. Some of the techniques I used to address this included noisy regex matching, score-based heuristics, decision trees, information theory-based matching, and model-based pseudo-labeling. In some cases, these were best applied as an external process. In other cases, it was useful, even mandatory, that the ML model had an internal labeling process. For all of the problems I encountered, various techniques could be combined to yield high-quality training data, although at the cost of engineering and analyzing the labeling process for each model. The labeling processes were the most crucial part to get completed, but data-centric AI presents a whole host of challenges on top of normal DevOps for ML. As more models were deployed and more customers were onboarded, more different types of data issues would arrive. These included data conversion issues, OCR errors, platform bugs, etc. Data issues would be detected from a variety of sources: release QA, incident management, client feedback, bug reports, monitoring, data validation, metrics, etc. Many more went undetected until exploratory data analysis (EDA) was done, which took time. I essentially needed to automate EDA for data quality and found a dedicated developer to work on this as my first hire to the team. Detecting the issues was crucial, but soon the majority of our time went to maintaining the models and acceptable accuracy. It was obvious that we'd need better tooling and processes in the long run. Some of the ops toolings we built early on were metrics and monitoring dashboards, with support from the engineering ops team. On the model side, we built held-out testing of all models as part of training, as well as model-testing with a worker process at each pull request, and a QA procedure following each release. The model-testing was a game-changer in preventing any production issues from models. We had close to no model failures and had to roll back models around once a year, due to having these tests in place. One additional layer of prevention I did was adding variable typing and validation to all of the models so that data incoming to the models was always typed and validated.


Autonomous AI Once I had tackled the main data quality problems in building ML models in this domain, I started developing the autonomous AI solution for invoice processing. This was a challenging task, as there was no existing research on the topic, and it wasn't known how well this could be done with the current technology. But, I saw no problems from an ML design point of view. I had access to massive amounts of data, solid support from both engineering and management, and relaxed time-frames to build the solution. I had zero doubts about accepting the challenge. The autonomous AI problem in accounting has some simplifying properties, compared to the commonly known problem of autonomous driving. Most importantly, we don't need to automate all documents. With autonomous driving, the AI can't choose to stop controlling the vehicle in a difficult situation. In our case, it is acceptable or even desirable that an anomalous document will get sent for human review instead of automation. Another simplification is that automation error rates don't need to be arbitrarily close to 0. With autonomous driving, any type of error has a potential cost of a life. In our case, the worst types of errors can be easily prevented, many types of errors are inconsequential or average out, and many important types require only an error rate matching humans. A key advantage was that I was designing the system on top of the platform we controlled, which at this stage was processing invoices for hundreds of companies, with an exponentially growing stream of documents. This meant I could develop the system to work on exactly the stream of data that it was deployed on, eliminating any bias coming from transferring an ML model from an external source. Also, I could fix any ML components causing issues, and request changes when needed to the backend, and the increasing data volume would enable better modeling later. One complication in our case was the number and diversity of required predictions to automate. To automate the accounting workflow, the AI needed to predict all of the outcomes produced by an accountant, for invoices in different languages and legislations. Fortunately, AI toolkits were going through constant evolution, providing better tools to address each modeling problem. When I started, Vowpal Wabbit and CPU-based gradient boosting were industry-standard solutions for large-scale problems. Available ML tools have since gone through numerous developments, such as GPU-based gradient boosting, Pytorch for deep learning, and transformer-based AI. Transformers in particular is a true revolution in applied ML and could become for AI what transistors were for electronics. This expanding toolkit allowed the design of solutions that were inconceivable just a few years before. It took me a couple of months of focused work, and a third major iteration of the modeling approach to find a solution that worked well. I proposed the name "Accounting Autopilot" for the product, which got shortened to Autopilot. The solution was conceptually simple but would require considerable work getting the details right. Any solution to this problem would require high accuracy from all of the ML components, consistency of the platform, and detailed information from the platform at the time of each prediction. This meant I had to double down on the data-centric AI efforts, but at the same time, we were growing exponentially, so we had to upscale our AI stack and platform to match the growth.


Scaling AI on the Cloud Since the beginning, we've had a partnership with one of the largest accounting technology companies in the Nordics, providing access to hundreds of millions of invoices. As we expanded across US and EU, I had data on a similar scale arriving from multiple regions from our clients. It was clear the solution needed to scale massively, regularly training and predicting on millions of documents. The AI had to be scalable from the beginning, and portable to distributed cloud-based processing. Effectively this is composed of two parts: writing efficient, maintainable, and distributable code, and setting up distributed processing for both training and prediction. The choice of Python tooling and best practices was a simple way to improve scalability. I discarded Jupyter Notebooks from any prototyping work. Instead, we deployed the same Python scripts that were prototyped. Soon after I came across Joel Grus's famous "I Don't Like Notebooks" presentation, which said everything I had been thinking about notebooks. Deploying scripts instead of notebooks removed the time required for production conversions, and meant that production issues could be debugged using the same exact code offline. The scripts could be optimized much further as well, starting with multiprocessing and serialization. This was done as far as reasonable with the Python data stack, so the base cost of operating the models was minimized. This included caching dataframes, Cython extensions for critical model code, use of GPU where possible, model pruning and compression, and regular analysis and optimization of processing bottlenecks. A while before Vic, I had worked with Spark for a couple of years. The functional MapReduce paradigm it offered was exciting, but the PySpark integration had fundamental shortcomings. Inspired by Spark, I developed my own Python library Wordbatch, which could do MapReduce processing of any Python method or function, executed by a chosen scheduler backend, such as a serial process, multiprocessing, Dask, or Spark. I could develop code on a single node, and scale it on the cloud using Wordbatch, with nearly linear speedups in the number of cores. On a single 128-thread workstation, Wordbatch speeded up most bottlenecks by 100x, enabling fast development iteration and model training. Soon after developing the first models, Ray was announced. I started using Ray as a Wordbatch backend extensively, for both model development and training. It simplified using nested parallelism and allowed production training of models at a massive scale at a low cost. Today, we train and operate hundreds of thousands of models simultaneously in production. On model serving, I considered the alternatives, as our initial Celery-based workers were not scalable or reliable. A couple of the senior developers in the team thought about the FastAPI-based model serving. I considered FastAPI or Ray Serve, and even discussed with the FastAPI creator Sebastián Ramírez about collaborating with us. Ultimately, the Ops team was determined on AWS serverless processing and wanted us to use Lambda and other AWS serverless tools. The choice was up to the backend developers, and the model serving side was scaled with AWS serverless tooling. The third type of cloud computing was the use of external AI microservices. When I started these were not available, but over the past four years, the cloud giants (Amazon, Google, Microsoft) started to offer an increasingly comprehensive selection of AI APIs, many usable for the type of document processing we were doing. None of these directly competed with our AI but offered useful basic building blocks for building a solution, including OCR, machine translation, table processing, document AI, etc. I organized systematic evaluations on many of these, and we started using external APIs for some AI workloads. This had several benefits: the external services were scalable and reliable, the costs of each process were known, and the AI results were high accuracy in many cases. I could jumpstart projects knowing that the external API could be deployed to production within a couple of weeks, and the executive threshold for approving projects was reduced, compared to launching internal R&D efforts with uncertain costs and outcomes.


Building an MLOps Team Building and maintaining a commercial AI such as ours eventually required a larger team, due to the high requirements for accuracy and automation. I recruited the teams for Vic.ai in two stages, first as the 24 person ML team to build the initial solution, then expanding this to an 8-person MLOps team once the workloads became unsustainable for the small team. I received the full support of the execs to proceed with this, and personally planned each role and conducted the hiring processes. One obstacle during this time was the "Great Resignation" which was starting to affect hiring in AI. Especially US-based candidates had a wide selection of companies to choose from, high expectations for the role, and a low threshold to move on. Rather than compete for the talent pool in the US, engineering had the idea early on to hire EU-based developers and started an engineering office in Oslo before covid. Many of these early developers were great hires and became valuable contributors. I took this idea further and decided to hire in regions with time zones close to New York and Oslo, but with better talent pools for hiring: especially developers with advanced engineering degrees in data and ML, but with limited local AI companies to offer them roles. For Oslo, countries such as Poland, Spain, Portugal, Italy, and France had good engineering colleges and universities, but limited AI startup scenes. For New York, Brazil and Canada were countries with similar qualities. Thanks to remote work, developers in the southern hemisphere were just as close to collaboration. Only the time difference to their team mattered, and even this not much. I planned each role according to current and future requirements we'd have for operating the AI, roughly corresponding to Google's "Data science steps for ML". For the small team, the hires were for data analysis, OCR, and backend engineering. With the larger team, I divided the team into data, ML, and ops, with at least 2-3 people operating each area. This meant that for each area of the codebase there would be at least two people who knew that area in detail, in case of the proverbial bus incident, as well as to have multiple developers for pair coding on each area. The overall goal of the larger team would be building towards full MLOps maturity. All candidates were required to have a minimum of a CS-related degree, some years of commercial software development experience, and specialization skills for their role. The rest of our process was fairly standard. For all roles, we had multiple interviews, with the full team given a chance to interview and veto a candidate. All our advertised positions received hundreds of candidates, some thousands, so we needed to efficiently screen the suitable hires if any were available. From the beginning, I conducted an initial screening interview, followed by a technical interview with the team, a coding test, and a final interview with one of the execs present. For the larger team, we had support from a technical recruiter, including an additional HR screening round. The process created a funnel that selected suitable candidates, but the time committed by the team was still considerable. Overall this was worth it. We got amazing coworkers who started contributing from their first weeks.


Epilogue The most important lesson from these years is that the organizational foundation is a key requirement for building an AI solution. This includes several matters outside the work of AI developers, including the company vision, culture, executives, product, and industry connections. The executives and culture are required for AI development to have sufficient support and autonomy to succeed, while other matters external to AI development determine if the developed solution will lead to success in the market. The success with Vic has shown that autonomous small remote teams are highly effective in building commercial AI. During the first 4 years, I designed, developed, and trained all of the models deployed in production, while the 13 other developers in the ML team handled the required ops, data engineering, and backend workloads. Moreover, the work was conducted almost entirely remotely, due to the pandemic. The effectiveness of small teams is well established in software engineering, and AI is not an exception. The crucial difference is in task management, where AI and data science require a different process at the project level. Building a commercial AI solution is altogether separate from what is learned in ML degrees, research, or competitions, even if the theory and tools learned from these provide the required technical foundation. Commercial AI solutions are built in a cutting-edge environment, where the product vision strives for solutions not possible with last-generation technologies, and perhaps not with current ones. At the same time, the data-centric and scalability considerations introduce additional layers of complexity to the design problem. Finally, the work integrates with software engineering and product management, in a dynamic environment where various events determine the planning and long-term goals. We're currently in a golden age for AI, with the amounts of research, data, computation, software, developers, applications, companies, and funding all growing at exponential rates for the past decade. This is a period similar to what the 90s were for PCs when every year brought unexpected ways to use increasingly better computers. At the same time, there's increasingly heavy competition from both startups and the cloud giants. If one wants to build a product in AI, one strategy is to target a new AI application and establish a leadership position, much like Vic has done with autonomous accounting. Most of all, this is a time to be excited about the numerous uses for AI that are coming available.


How To Train Your BERT Model 5X Faster Than In Colab

Example notebook For this article, we will use a notebook called TransferLearning, created using code from a post by Alvin Chen. We will build a sentiment classifier with a pre-trained BERT model using the Hugging Face transformers library. Here are the files needed to download and run the notebook yourself: NotebookDataset is included (downloaded in a cell)


Transfer learning refresher Transfer learning is a method that can be used to accelerate the development of new machine learning models. It works by leveraging the knowledge gained from training a model on one dataset and applying it to another related dataset. For example, if a data scientist wants to build a model to identify dogs in pictures, they could use data from a previous project that was used to train a model to identify cats. This can be especially useful when data is limited or when there is a need to build a model quickly.


BERT refresher BERT is a Transformer-based model for natural language processing that was proposed in 2018 and open-sourced by Google. The model is trained on a large corpus of text, such as Wikipedia, and can be used for various tasks such as question answering and text classification. BERT can be easily fine-tuned on a variety of tasks such as text classification and named entity recognition.


Baseline performance Since fine-tuning the BERT model is by far the most computationally intense part of this example, we will only focus on the training cell (cell #18). In our last article, we used a laptop with 8 CPU cores and 32 GB of RAM to get a baseline performance. This experiment would likely run without fail on the same laptop but it would take impractically long, so instead, we will use Google Colab. Colab offers free GPU access and is a common workspace for deep learning. However, one thing to remember is that you are unable to predict or select resources - the type of GPU and amount of RAM will vary for each session. When running this experiment we happened to be given a T4 GPU (not bad!) with 16 GB of GPU RAM and 12 GB of regular RAM. The BERT training cell took 17 minutes and 17 seconds.


Our enhancements Use mixed-precision Mixed-precision is a technique used to improve the performance of machine learning models. It involves using both lower-precision data types (such as 16-bit floats) and higher-precision data types (such as 32-bit floats) in training and inference. The benefits of mixed-precision include reduced training time, reduced memory usage, and increased accuracy. Because we are using a newer T4 (uncommon as Colab typically allocates older GPUs), we can take advantage of mixed-precision. To do so, we added the following lines of code: We ran the notebook again and it was completed in ~8 minutes. 2. Leverage a newer GPU   This time, the BERT training cell was completed in 3 minutes and 5 seconds - more than a 5x speedup from the original experiment, and it only cost 18 cents!


Key takeaways So, what should be remembered going forward? Mixed-precision has many compelling benefits, such as reduced processing time and reduced memory usage, but can only be utilized if you have modern GPU hardware. Keep this in mind when using cloud services that offer free compute resources - you might only have access to low-end GPUs. There are tradeoffs!Using powerful resources for short periods of time can be cheap. The cognitive friction of a paid vs. free service might be more than the actual dollars you'll spend. You might be able to unlock significant performance gains for under a dollar.Examples like this one often stick to a small scale to facilitate reproducibility, but the techniques to improve performance are often generalizable. Try them on larger-scale projects!


If you found this article helpful, we have written others like it! You might enjoy: How To Speed Up Your Grid Search 60xScale ML experiments from JupyterLab to the cloud Please also feel free to leave claps below so other people on Medium see it and so we know to produce more


About the author I mostly write about practical data science and machine learning workflow challenges and walk through ways to solve them. You can follow me on Medium or LinkedIn.


Trends in AI - June 2022

Research Every month we analyze the most recent research literature and select a varied set of 10 papers you should know of. This month we're covering topics such as multimodality, Reinforcement Learning (RL), Diffusion Models, Information Retrieval, Prompting, and more.


Trends in AI - June 2022

1. A Generalist Agent  Blog post By Scott Reed, Konrad Żołna, Emilio Parisotto, et al. Why -> We've been highlighting multimodality as one of the key pillars for the future of AI, exemplified by the recent generalist visual language model from Deepmind Flamingo⁴. This work, also from Deepmind, takes multimodality to the next level: one single model trained on vision, text,, and control (e.g. RL) data from hundreds of varied datasets. Key insights -> El Gato applies the usual recipe of a decoder-only Transformer (1.2B parameters) trained to predict the next token autoregressively. But now instead of language-only tokens, they serialize generic inputs into embeddings which can represent: Text, tokenized via SentencePiece with 32000-word vocabulary size.Image patches of size 16x16. These are embedded using a single ResNet, instead of using discretized image tokens like DALL·E 1.Discrete values such as Atari button presses are tokenized with values between 0 and 1023.Continuous values for control (e.g. joystick position in a videogame) are also discretized into tokens with values between 0 and 1023. Many intricate choices are made to make sure all tasks can be represented in a sequence of embeddings (e.g. observation/action/separators for control policies) but the gist of it is relatively simple. Also, unlike text-only Language Models, Gato is mostly trained in a supervised setting where labeled samples from hundreds of datasets are drawn and fed to the model step by step to predict the next token. This means that for control problems such as Robotics or Atari games, Gato learns through behavior cloning, instead of actively interacting with an environment as it's commonly done in RL settings. The results convey that this is a feasible and successful approach, but not yet a SOTA-shattering for now. The resulting model can do things such as captioning images, keeping up with a conversation, and playing Atari games among others. Their analysis on scaling shows that their model is nowhere near maximal performance, so we expect similar approaches scaling up further to come out in the upcoming months!


Trends in AI - June 2022

2. Planning with Diffusion for Flexible Behavior Synthesis  Project page  Code  Colab Notebook By Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Why -> Diffusion models, which have proven to be incredibly useful for image synthesis, are finding new avenues to shine, such as model-based RL. Key insights -> Existing model-based RL approaches often learn environment dynamics and then use a classical gradient-free trajectory optimizer for decision-making. Given that the learned environment dynamics are only an approximation, the trajectory optimizer might exploit the approximate dynamic weaknesses and come up with adversarial-esque trajectories which don't translate well to the original environment, hurting performance. This work leverages diffusion models to include the trajectory optimization as part of the learning process, mitigating the pitfalls of current trajectory optimizers, and aiming to achieve: Long-horizon scalability: training based on the overall trajectory accuracy instead of a single-step error, avoiding myopic failure modes.Task compositionality: a simple way to add auxiliary loss functions to condition the sampling of plans in the diffusion process.Temporal compositionality: achieving global trajectory consistency by imposing local consistency at each diffusion step.Effective non-greedy planning: model and planner being jointly optimized aims at a synergic interaction between the components instead of an adversarial one. In broad strokes, consider a trajectory which is a causal sequence of {state, action} pairs. A diffusion process is applied to this sequence that gradually adds noise to each {state, action} pair, and then the model learns to reverse this process: starting from a trajectory that's only noise, gradually refine it until it's a good trajectory. In their experiments, the proposed diffuser performs competitively with existing methods; but perhaps more interestingly, the trajectories that the model comes up with seem to satisfy the 4 goals mentioned above: long-term horizon planning, task, and temporal compositionality and non-greedy planning. You can check some video examples on their project page, check their code, or even play with a colab notebook. This method represents an important conceptual departure from existing planners and has the potential to become a fertile ground for new research.


Trends in AI - June 2022

3. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding By Chitwan Saharia, William Chan et al. Why -> DALL·E 2 from OpenAI² was arguably the biggest highlight from April, as it pushed the boundaries of how complex and realistic image generation can be, sparking an intense debate around the capabilities and limitations of such models. Now Google has published a model that rivals DALL·E 2: Imagen. While nobody rivals the OpenAI marketing genius, this is also a work with paying attention to. Key insights -> This approach called Imagen is similar to DALL·E 2 in that it uses a Diffusion Model for image generation, but it's different from it in how it represents prompts and how it achieves high-resolution images. The text representation comes from a text-only frozen language model (T5), and the image generation is performed by a diffusion model (as usual at this point) with two super-resolution upsampling steps, up to 1024x2014. It uses training techniques such as classifier-free guidance which lets it learn both conditional and unconditional generation. Another technique introduced is dynamic thresholding, which prevents the diffusion process from saturating in certain regions of the image, which is a phenomenon that hurts image fidelity especially when the weight on the text conditional generation is high. In terms of results, they achieve state-of-the-art on the COCO dataset with a zero-shot FID score of 7.27, even better than DALL·E 2. Still, objectively evaluating image generation is challenging so take these numbers with a grain of salt. Generally speaking, the generation seems on par with that of DALL·E 2, but we would need to interact at length with both systems to get a better sense of how they compare, which is not currently possible. One of the key findings from this paper is the fact that such representations are surprisingly effective despite not being trained in a multimodal way like CLIP [10]. Moreover, they find that increasing the size of the pre-trained language model improves image generation more than increasing the size of the diffusion model that generates images.