Up-Sampling (Decoding) The up-sampling process is pretty straightforward (without any fancy techniques). Firstly, the input from the CNN-Transformer encoder is run by a 3x3 convolution layer with ReLU activation, upsampled, and then concatenated with the output of the third-level CNN feature extractor.The resultant feature maps are then run through the 3x3 convolution with the ReLU activation layer again. The output is then concatenated with the output from the second-level CNN feature extractor.The step is repeated again. Now, the output is a mask in shape (C, H, W), with C=number of objective classes, H=image height, and W=image width. The authors also noted that more intensive incorporation of low-level features generally leads to better segmentation accuracy.


TransUNet - Revolutionize Traditional Image Segmentation

Results The model is run on the Synapse Multi-Organ Segmentation dataset. The final scores are evaluated by the Dice Similarity Coefficient and Hausdorff distance. As you can see, TransUNet performed better than most of the existing SOTA architectures, such as V-Net, ResNet U-Net, ResNet Attention U-Net, and Vision Transformers, which suggests that the Transformer-based architecture is better at leveraging self-attention than other self-attention based CNN U-Nets.


Implementations Official TransUNet Implementation


Reference TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation


This AI newsletter is all you need #10

What happened this week in AI This week, our attention is still turned to Stable Diffusion. We had to cover it once more with even more fantastic news as an important step for the AI community. Stable diffusion is a fantastic initiative brought to us thanks to an amazing human being named Emad (learn more about Emad in Yannic's video!) and his company, Stability.Ai. The idea behind Stability AI is to create and make publicly accessible important AI efforts from the community in direct competition with big companies. Diffusion models recently achieved state-of-the-art results for most image tasks, including text-to-image with DALLE but many other image generation-related tasks too, like image inpainting, style transfer, or image super-resolution. Stable diffusion is based on the same principle, but within the latent space, which makes all the difference in computing needed for us regular people. If you still do not know what Stable Diffusion is or how latent diffusion in general works, we recommend this article for a better understanding of the model.


This AI newsletter is all you need #10

Hottest News A GitHub repository full of pre-trained diffuser models!Thanks to HuggingFace,  Diffusers provides pretrained diffusion models across multiple modalities, such as vision and audio, and serves as a modular toolbox for inference and training of diffusion models.Stable Diffusion API is out and you can use it!The Stable Diffusion model, a collaboration between Stability, LAION, and EleutherAI, has been released open source and is available on HuggingFace! Try it now :)A free AI writing assistant built on top of OpenAI's GPT-3The tool, by ParagraphAI, is able to generate articles, essays, and other original texts and is available for free on the Google chrome application store.


This AI newsletter is all you need #10

Most interesting papers of the week DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation"With just a few images (typically 35) of a subject (left), DreamBooth - our AI-powered photo booth - can generate a myriad of images of the subject in different contexts (right), using the guidance of a text prompt."Z-Code++: A Pre-trained Language Model Optimized for Abstractive SummarizationAs the title says, Z-code++ is "a new pre-trained language model optimized for abstractive text summarization."Wonder what is abstractive text summarization?"Abstractive summarizers paraphrase the idea of the source documents in a new form, and have a potential of generating more concise and coherent summaries."Playing for 3D Human RecoveryThey introduce GTA-Human, "a large-scale 3D human dataset generated with the GTA-V game engine, featuring a highly diverse set of subjects, actions, and scenarios. More importantly, [they] study the use of game-playing data and obtain five major insights...". Get the code. Enjoy these papers and news summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #10

We will be at ECCV 2022 - come meet us! Thanks to our friends at Qwak and their help, our head of community, Louis Bouchard, will be at ECCV 2022 in Tel Aviv (October 2327). Feel free to reach out if you think you will be attending and would like to have a coffee or chat there!


This AI newsletter is all you need #10

Featured Community post from the Discord altryne#7376 created an awesome list of resources for image synthesis (aiArt). Altryne regroups the different text-to-image models, prompt engineering resources, interesting communities and many useful tools for anyone interested in doing AI arts. Give it a look!


This AI newsletter is all you need #10

Article of the week Why Should Adam Optimizer Not Be the Default Learning Algorithm? An increasing share of deep learning practitioners are training their models with adaptive gradient methods due to their rapid training time. Adam, in particular, has become the default algorithm used across many deep learning frameworks. These adaptive optimization methods are known to generalize poorly. The author comprehends the findings of empirical studies carried out to mull over the marginal value of adaptive gradient methods. If you are interested in writing for us at Towards AI, please sign up here and we will publish your blog to our network if it meets our editorial policies and standards. https://contribute.towardsai.net/


This AI newsletter is all you need #10

Lauren's Ethical Take on Twitter's Whistleblower The news that I want to spotlight this week regards Twitter whistleblower Peiter "Mudge" Zeiko's recent allegations over serious data mismanagement, security weaknesses, poor tracking of bots, vulnerability to foreign intelligence, and possible FTC violations. We've seen big tech whistleblowers before, and some people have become desensitized or resigned to the nature of the beast due to the news cycle or a cynical outlook on how tech should operate in society. The news cycle's initial spike, talk of big things happening and the eventual simmering down of the hype don't help the feeling of impermanence. Yet something tells me this story will be here to stay. The fact that all of these allegations are so egregious if true, and the gravity of the situation may prove that resulting penalties can have serious do-or-die consequences beyond mere warnings. Regulation might move slowly with respect to tech, but it can be powerful. The exposure of multiple foundational flaws that are brittle and have grown weaker over time shows the costs of building fast and growing so large that no one knows what's happening anymore. The fate of Twitter may serve as a very important lesson for the perils of building an online community unsustainably. More than just what allegations prove true or untrue is how they will be dealt with by both regulators and users. This will be an important story to follow in the coming months, and I look forward to the insights we learn from this experience.


This AI newsletter is all you need #10

Featured jobs this week Computer Vision Scientist - Percipient AI (Santa Clara, CA) Research Scientist - Machine Learning @ DeepMind (London, United Kingdom) Senior Data Scientist @ EvolutionIQ (Remote) Senior ML Engineer - Algolia AI @ Algolia (Hybrid remote) Senior ML Engineer - Semantic Search @ Algolia (Hybrid remote)


Easy Object Detection with Transformers: Simple Implementation of Pix2Seq Model in PyTorch

Introduction Object detection does not have to be a difficult task! I clearly remember the first time I implemented YOLO from scratch, and it was a pain to understand how it works under the hood. For beginners in computer vision applications, I believe that object detection is the hardest one to understand among classification, segmentation, etc. Once I first heard about the paper "Pix2seq: A Language Modeling Framework for Object Detection" from ICLR 2022, I got pretty excited, and I was sure my next blog post would be about it; so, here I am writing this post and hoping that you'll like it and find the pix2seq model easy to understand and implement. At the end of this tutorial, you'll learn to implement a simple model for object detection which produces the following results: I have made all of my code available as Google Colab Notebook and a Kaggle Notebook. I've also put the whole project and codes on my GitHub.


What's interesting about this paper The idea is pretty simple: Reframe the object detection problem as a task of text (token) generation! We want the model to "tell us" what objects exist in the image and also the (x, y) coordinates of their bounding boxes (bboxes), all in a specific format in the generated sequence, just like text generation! As you see, the object detection task is transformed into an image-captioning-ish task: describe the image in the text (sequence) but this time tell us exactly where the objects are.


Needed Modules The closest task to what Pix2Seq does is image-captioning. So, we are going to need an image encoder to convert an image into vectors of hidden representation and then a decoder to take the image representations and those of the previously generated tokens and predict the next token. We also need a tokenizer to convert object classes and coordinates into tokens that form their special vocabulary, just like the words in a natural language.


My Simple Implementation of Pix2Seq You can see the high-level pipeline of this project in the picture above. As you see, we need a dataset of images and their bboxes for which we will use Pascal VOC 2012 dataset. Next, we will write our own tokenizer from scratch to convert the bbox classes and coordinates into a sequence of tokens. Then, we will use DeiT (from this paper) as our image encoder and feed the image embeddings to a vanilla Transformer Decoder (from this paper). The decoder's task is to predict the next token given the previous ones. The outputs of the decoder are given to the language modeling loss function. The codes of this tutorial are available in the following links:- Google Colab Notebook- Kaggle Notebook- My GitHub repo


Dataset As I mentioned earlier, we will use VOC 2012 dataset with images and their corresponding objects from 20 classes. The paper uses the COCO dataset, which is an order of magnitude larger than VOC, and they also pre-train the models on a much larger dataset before training on COCO. But, to stay simple, I'm going to use this rather small VOC dataset. We need a PyTorch dataset class that gives us an image and its bbox coordinates and classes in the form of a sequence. As you see, most of the code here is what you expect from a simple dataset for classification, but there are small differences too. We need a Tokenizer to convert our labels and bbox coordinates (x and y) to a sequence so that we can perform train our model for the language modeling task (predicting the next tokens conditioned on the previously seen tokens).


Tokenizer How are we going to convert this information into a sequence? Well, it's not that difficult. To represent an object in an image, we need 5 numbers: 4 coordinate numbers and 1 to indicate which class it belongs to. You actually need to know the coordinates of 2 points of a bounding box to be able to draw it in an image; in pascal format, we use the top left point and the bottom right point of the bbox as those 2 critical points, and each point is represented by its x and y values -> so, we will need 4 numbers overall to draw a bounding box. You can see alternative formats to represent a bounding box down below. Also, look at where the start of x and y axis is (the 0, 0 point). As you see in the dataset's code, we give the bbox coordinates and labels to our tokenizer and get a simple list of tokens out. The tokenizer needs to do the following tasks: mark the start and end of the sequence w/e special tokens (BOS and EOS tokens).quantize the continuous value of coordinates (we can have x=34.7 as the coordinate of a point, but we need discrete values like 34 as our tokens because we are finally doing classification on a finite set of tokens)encode the label of the objects into their corresponding tokensrandomize the order of objects in the final sequence (more on this below) If you are familiar with NLP applications, these steps might sound familiar to you as they are also done when we are dealing with words in a natural language; we need to tokenize them and assign each word to its own discrete token, mark the start and end of the sequence, etc. Regarding the number 4 in this list, this is what the paper does, and there is an extensive ablation study on whether it is a good idea. What it says is that each time that we show the same image to the model (in different epochs), we randomize the order in which the objects appear in the corresponding sequence, which we feed to the model (with one token shifted) and our loss function. For example, if there is a "person", a "car", and a "cat" in an image, the tokenizer and dataset will put these objects in random order in the sequence: BOS, car_xmin, car_ymin, car_xmax, car_ymax, car_label, person_xmin, person_ymin, person_xmax, person_ymax, person_label, cat_xmin, cat_ymin, cat_xmax, cat_ymax, cat_label, EOSBOS, person_xmin, person_ymin, person_xmax, person_ymax, person_label, car_xmin, car_ymin, car_xmax, car_ymax, car_label, cat_xmin, cat_ymin, cat_xmax, cat_ymax, cat_label, EOS... Another note on how to quantize the continuous values of coordinates: imagine that the image size is 224. You can have a bbox with these 4 coordinates (12.2, 35.8, 68.1, 120.5). You will need at least 224 tokens (num_bins) to be able to tokenize (quantize) these 4 numbers with a precision of 1 pixel (you will lose information below 1 pixel). As you see in the tokenizer code, to convert these bbox coordinates to their tokenized version, we need to do the following: normalize the coordinates (make them between 0 and 1 by dividing them by the max value = 224)do this: int(x * (num_bins-1)) so, the converted version will be: (12, 35, 67, 119). Remember that int() function in Python does not round the number to the closest integer, but it will keep only the integer part of the number. As you see, we have lost some information on the exact position of the bbox but it is still a very good approximation. We can use a larger number of tokens (num of bins, as stated in the paper), and we will have a more precise location. Our tokenizer also has decode() function, which we will use to convert sequences into bbox coordinates and labels.


Collate Function Here, we will implement a custom collate_function to give to our PyTorch data loader. This function will take care of Padding for us: to make all the sequences the same length by adding PAD_IDX to the shorter ones in order to be able to build a batch with them. We are going to pad the sequence to a fixed max length of 300 tokens.


Encoder I finally arrived to the coolest part for every deep learning lover: The Model  Let's take a second look at the first image of this tutorial. First, we will need an encoder to take the input image and give us some embeddings (representations). The paper uses a ResNet50 (and also in other experiments uses ViT), but I decided to use DeiT. As the name suggests, this is a data-efficient vision transformer, and I thought it would be a good fit for our small dataset. Like ViT, it splits the image into patches and processes them like words in a sentence which again could be great for our task, as we will have a separate embedding for each of these patches, and we can give them to our decoder in the next section to predict the target sequence (see it like translation from English to French, where our image is like a sentence in English and our target sequence containing the coordinates and labels of bboxes is like the equivalent sentence in French). I will use timm library to implement a pre-trained DeiT model. The bottleneck layer is to reduce the number of features of these embeddings to that of the decoder. The paper used a decoder dim of 256, and that's the reason why I am reducing it here using Average Pooling. Also, the first token in this model relates to the CLS token, and I am skipping it in the forward method (features[:, 1:]).


Decoder Our decoder takes the patch embeddings of the input image and learns to predict the sequence containing bboxes. Here I am using PyTorch nn.TransformerDecoder module to implement a 6-layer decoder with a feature dimension of 256. We also need to add positional embeddings to the embeddings so that the model knows about each token's position in the sequence (I am adding positional embedding for both encoder tokens and decoder tokens. While we have to do this for the decoder, we might not need to add them to the encoder tokens as the DeiT model knows about the order of patches itself). I am doing this by those nn.Parameter modules will learn 1 parameter per token position. Finally, we will use a nn.Linear layer to predict the next token from our vocabulary. The create_mask function gives us two masks needed for training the decoder: one to tell the model to ignore the PAD tokens and not incorporate them in its attention modules and another to mask the future tokens in order to make the decoder predict tokens only by looking at the current token and the previous ones.


Putting Them Together This is a simple class encapsulating the encoder and decoder. It also has a predict function which calls the predict function of Decoder (not shown above, we will see it later) to detect objects in an image.


Training Now let's see how we can train this model. Most of the following code is just standard PyTorch training boilerplate, but there is a simple but important point in it. As mentioned earlier, we train the model like a language model (GPT for e.g.), and it works like this -> the model needs to predict the next token only by seeing the previous ones (tokens to the left). At the start, it only sees the BOS sentence, and it needs to predict the next token, and so on and so forth. And this is achieved simply by this part: y_input = y[:, :-1]y_expected = y[:, 1:]preds = model(x, y_input) I trained this model on Kaggle using a single GPU and only for 25 epochs in 6 hours. This is not much, but it was enough to get a decent performance on the evaluation metrics. The most common metric for object detection is Average Precision (AP) which you can read more about it here. The paper gets an AP of 43 w/ ResNet50 backbone after training on a whole lot of data for many hours of training. I could get an AP of 26.4 on my validation set with this small model and short training time, which was cool as this is a tutorial on how to implement this paper easily, and I didn't aim to beat the SOTA with this!


Inference Now let's take a look at how we can generate a detection sequence with this model for a test image. This is the prediction method of the Decoder's class. It takes the previously generated tokens, pads them to the max_length and predicts the next token for each sequence in the batch, and returns those new tokens. The following generate() function shows a simplified version of the whole sequence generation pipeline -> First, we will create a batch with shape (batch_size, 1) containing only a BOS token for each image in the batch. The model takes the images and these BOS tokens and then predicts the next token for each image. We take the model's predictions, perform softmax and argmax on it to get the predicted token, and concatenate this newly predicted token with the previous batch_preds tensor, which had BOS tokens. We then repeat this loop for max_len number of times. We finally decode the predicted tokens by our tokenizer's decode() method. You can take a look at the exact implementation on my GitHub, the Colab Notebook, or the Kaggle Notebook.


Results Finally, let's take a look at some of the results of our model so far. The model is not always this good (these are somewhat cherry-picked!), but these results show that the implementation is good enough, and w/ a bigger dataset and model and longer training times, you can easily get those perfect as stated in the paper. Our model works best when there are 1 to 2 bboxes in the image, and its performance degrades when there are lots of objects in the image. The next image shows some of the failure cases of our model:


Final Words I hope you've enjoyed this tutorial and learned something new. As always, I will be glad to hear your comments on this tutorial or answer any questions you might have regarding the paper and model. Have a nice day! I have made all of my code available as Google Colab Notebook and a Kaggle Notebook. I've also put the whole project and codes on my GitHub.


Fake News Detection with Model Selection and Hyperparameter Optimization in Python (>97% acc.)

Introduction This article aims at describing the model selection and hyperparameter tuning process to perform fake-news detection. My previous article, in which I explained how the model selection is executed within the field of machine learning on an Amazon Kindle reviews dataset, has been a success among readers. The project presented a limit, though, given the size of the dataset containing roughly 1.000.000 reviews, we couldn't deploy the algorithm on a good portion of unseen data. It was done on 50.000 records, approximately 5%, because of computing limitations. In this article, the goal is to identify and tune the most performing model. Training and testing will be done on all the records this time and the resulting model will be deployed on a different dataset to test its accuracy and generalization ability.


The data   title - news headlinetext - news corpussubject - describes what the news headline is about (news, politics...)date - date on which the news headline was released True.csv contains approximately 21.000 records whereas Fake.csv presents 18.000 of them.   uuid - unique identifier alphanumeric string for the newsord_in_thread - how many times has the same article been scrapedauthor -the author of the newspublished-date and time of publishingtitle -news headlinetext -news corpuslanguage - the language of the newscrawled - date and time it was scraped from the websitesite_url - the website's post URL linkcountry - where the news was posted from For the classification problem, for the three datasets, we only need text and information on whether the news is truthful or not correct. Below are two samples from the text column of the two datasets: The first corpus presents truthful news, you can tell from the objective tone of it focused on telling just the facts. The second one shows a piece of fake news. The information reported expresses unjustified hate, it doesn't limit itself to merely describing the facts. The piece wants to raise awareness among the public and negatively target it against a specific group of people. Of course, such information can be extremely dangerous, especially if not true. This second sample shows that particular attention is needed for the cleaning portion. Aside from stopwords, there are HTML text tags that need to be filtered out (e.g. "\n", "/") along with special characters (e.g., "...", "!").


Models For the project, I considered 7 common classification models, including parametric and non-parametric ones. All of them are supervised, learning models: Linear Discriminant Analysis (LDA)k-Nearest-Neighbours (kNN)Gaussian Naive Bayes (GaussianNB)Logistic regression (LR)Decision tree classifier (CART)Support Vector Machines (SVM) For a more in-depth explanation of the basic principle behind each of them, I suggest you read my previous article. The way we reach it is by performing cross-validation for each model through an ML pipeline. The two best models in terms of accuracy and efficiency will be further evaluated after being optimized thanks to hyperparameter tuning.


Pre-processing As previously mentioned, the first link contains two datasets, one with only truthful news and one with only fake ones. We now need to put them together if we want to perform training and testing. Let's start by importing pandasWe transform the two csvs into pandas dataframes and name themA classification column needs to be added to identify whether a record is fake news or not. With the command fake['classification'] = 1we create a new column called classification, and all its rows are set equal to 1. Our classifier will identify fake news and assign them the value of 1.the two dataframes are then concatenated, and their index count is reset by setting the parameter ignore_index equal to True


Data Exploration Data exploration focuses on visualizing data to lead a more in-depth analysis, discover early insights and identify patterns. We can start the process by simply plotting the number of news per each category: we import matplotliba new variable called category_dist is created, which counts the number of times the numbers 1 and 0 appear within the classification columnwith plt and a series of commands, we can choose the size, type of chart, color, labels, title, and whether or not we want the gridlines. The dataframe is composed of approximately 40.000 records with slightly more fake news. Given this difference is not very wide, we don't need to implement any imbalanced data handling techniques like upsampling. Feeding the model selection pipeline with over 40.000 records, each containing 750 words or more, could result in a RAM memory overload. A simple solution to the problem is to select a smaller portion of records. pandas offers an easy-to-use command that allows the developer to sample a specified number of records while setting a random_state for results reproducibility. I decided to select 3000 records, it's an arbitrary number that represents a little less than 10% of the dataframe and it should not cause RAM overload. Even with 3000 records, the distribution between fake and truthful news stays the same. This means we can move on to the next phase.


Text Cleaning Text cleaning in NLP is fundamental. Stopwords, special characters, and HTML tags all add unneeded complexity for our models to process. Eliminating all of the above could result in an improvement in the training and testing time. After completing the cleaning process on the sample of truthful news I showed at the beginning of the article, the final result looks like the following: The above is an example of a news corpus the model will be trained and tested on.


Model selection It's time for the implementation of the model selection mechanism. The code pipelines data through a count vectorizer and all the models previously listed, finally executing k-fold cross-validation. The average accuracy is then calculated and organized in a boxplot chart. Let's break it down step by step: scikit-learnand matplotlib are the only two libraries needed? Sklearn includes all the functions needed to perform machine learning on data. Each model is imported through a "sub-library": LogisticRegression, KNeighborsClassifier, DecisionTreeClassifier, SVC, GaussianNB, and LinearDiscriminantAnalysis.At this stage, we can define the input and the target variable from that data. The input variable xis "text", containing the corpus of the review; the output variable yis "classification", presenting the labels 0 and 1.We know that some models took into analysis, including decision tree classifiers, require a dense matrix. Dense matrices contain mostly non-zero values. The ToDenseTransformerclass makes sure all matrices are dense to avoid any error.The list modelsis then created, and the object assigned to each model is then appended to the list. On the other hand, the list resultswill contain all the different model scores associated with their names.The kfold the parameter indicates how many k-folds we want. This introduces the concept of cross-validation. Cross-validation aims at better estimating the accuracy of a model. It is defined as k-fold cross validation, with k being the number of sub-groups data is divided. The model is trained on a subgroup and tested on the remaining k-1. The accuracy is calculated on the average score.The pipeline applies the count vectorizer, the dense transformer, and the model of choice on the k-fold of choice and performs cross-validation. It then prints the results on the console.matplotlib finally generates a boxplot chart so we can better interpret the results. A boxplot gives you the information about five critical numbers of our models in terms of accuracy: the minimumthe first quartile (25th percent)the median (second quartile)the third quartile (75th percent)the maximum. On the x-axis, there are all the models taken into analysis for cross-validation. On the y-axis instead, we have the accuracy score. From the chart alone, it's hard to tell exactly how well each model performed; for example, the logistic regression and the decision tree classifier are very close to each other. The values printed on the console could help in this scenario: The number on the left expresses the average accuracy and the one in parenthesis is the variance calculated after cross-validation. It's clear how CART is the best performer in terms of accuracy but the logistic regression is slightly better (by a very thin margin) at delivering consistent results.


Logistic Regression Hyperparameter Tuning Logistic regression is one of the two models with the highest accuracy and lowest variance which makes it a great candidate for the hyperparameter tuning phase. The two parameters related to the logistic regression model are: A solver is an algorithm that helps the model better adapt to the data. In the case of logistic regression, there are 5 types: liblinear, lbfgs, newton-cg, sag, and saga.The C parameter expresses the regularization strength. A higher value assigned to C implies a stronger regularization. Regularization is a synonym for generalization when the model is presented with unseen data. The following code adjusts the above hyperparameters along with the count vectorizer ones to find the optimal combination: Libraries were already imported in the previous code cell but I decided to re-import them to make this code snippet independent. In this case, we only need sklearn with sub-packages related to cross-validation, model deployment, pipelining, and count vectorizer.The pipeline this time only includes the count vectorizer and the logistic regression model. No dense transformer is requiredMoving onto the parameters' list, these contain all the possible combinations of hyperparameters the grid_search tries for each component of the pipeline. For example, the parameter max_df of the Count, Vectorizer is responsible for the generalization of the model. max_df removes words that appear too frequently, and a max_df of 0.7 ignores terms that appear in more than 70% of the documents. In one scenario vect__max_df will combine a max_df of 0.7 with a ngram_range of (1,1), using a kernel poly and a C parameter of 10. The total "fits" (combinations) are 405 because each cross-validation is performed 5 times.after launching grid_search.fit, the last portion of code starts printing the results on the console as calculations are performed The most performing hyperparameter setting for the logistic regression model is the following: C = 100type of kernel = liblinearmax_df = 0.7ngram_range = (1, 3) With the settings, the model achieves a 98% accuracy when deploying the cross-validation method on only 3000 news. The result is already extremely promising. Given the optimal hyperparameters such as C and max_df, we can already tell the generalization of the algorithm should be enough to correctly classify unseen data. Amazingly, the model presents 100% accuracy when deployed on approximately 9000 unseen testing records. Especially in a case like this one, there could be a variety of reasons why a trained model is so precise. "Overfitting" is one of them. If the model classifies records really well within the same dataset, but when it's deployed on new data, it loses accuracy, then overfitting is likely happening. A second reason could be a "lucky pull". If a model trains on the right portion of a dataset and it's tested on easy records to classify, then it's possible a lucky pull has happened. On the other hand, though, cross-validation should eliminate the issue at its roots. A third reason could be the dataset has a common feature that external data could not have. It's possible all the fake news in this dataset contains the term or the hashtag "fake". It would be much easier for a model to understand truthful and fake news within the same dataset, but once presented with new data, accuracy might decrease. A common solution to all of the above would be to deploy the model on a different dataset and assess whether performance stays the same or decreases.


Decision Tree Classifier (CART) The Decision Tree Classifier (CART) shows the best accuracy out of the 6 algorithms taken into analysis. After hyperparameter tuning, it should perform even better. The three parameters of the CART model are: max_featuresis "the number of features to consider each time to make the split decision" [5]. If the data has 50 columns, then you can set this parameter to 10 to only include 10 out of this 50 during the training phase.max_depth is the maximum number of "ramifications" a tree can have. A higher value might cause overfitting.min_samples_leaf is "the minimum number of samples required to be at a leaf node" [5]. The value decides the minimum split points the tree needs to have. Aside from hyperparameters, there are no other changes in the code snippet for the CART model compared to logistic regression. The most performing hyperparameter setting for the CART model is the following: max_depth = 4max_features = 0.8min_sample_leaf = 0.04max_df = 0.7ngram_range = (1, 1) With the above hyperparameters for the count vectorizer and the algorithm, the average accuracy reaches a 99% score. Once we deploy the model on the 40.000 records and test it on 9000, the accuracy stays the same. A 99% precision remains a great achievement, on the other hand, though, 1% of approximately 4000 records has been misclassified, and the logistic regression seems overall a better solution for deployment on new records.


Testing the model on unseen data from a different dataset We've finally come to the final phase. As a reminder of what has been done so far: we've identified the logistic regression as the best modeltuned a logistic regression modeltrained it on 31.000 records and tested it on the remaining 9000 This section aims at understanding whether the previous hypothesis of a common feature within the dataset could be true. How do we do it? We do it by testing the model on a separate fake news dataset yields the same result or not [4]. The dataset presents only one limit, it only contains fake news. The following code snippet shows how to proceed: We import pandascreate the dataframe from the CSV file and re-naming it fake_testselect relevant columns out of the 21 available and filter out all the records that are not in Englishafter we set the input and output variables, all the records need to be transformed with the bag of words methodfinally the command LR.predict classifies the records with the trained model As anticipated, the dataset only contained fake news, and the classifier correctly predicted each one of them, maintaining what had been promised during the tuning and testing phase. Our model classifies with 100% precision whether the news is fake or not, even with an external dataset with minimum computational cost. The overall precision, though is mitigated by the recall score. Now, if precision is mathematically defined as: the recall score is defined as: So you can see from the confusion matrix below that 420 records that were supposed to be classified as fake were actually classified as true by the model. Still, considering the algorithm has been deployed on a different dataset, the performance is more than satisfactory. It allows us to eliminate the overfitting and common feature hypotheses.


Conclusion The project shows above any other teaching the importance of implementing mechanisms of model selection and hyperparameter tuning, they allow the practitioner to explore a wide variety of different options, finally resulting in great performing models under multiple perspectives. On top of that, it reminds us of the potential of machine learning in terms of rapidity compared to human counterparts. The algorithm only took a few minutes to train and subsequently classified tens of thousands of records. A human performing the same tasks would have taken weeks. Possible next steps from here include deployment on another dataset with both fake and truthful news or implementing the algorithm in a machine learning-based product such as BS detector. Finally, the article is meant to raise awareness on the fake-news issue that is maybe too popular on social media platforms. In the era of information, misinformation is a curse we need to solve, and AI represents a great tool to do it.


Which NLP Task Does NOT Benefit From Pre-trained Language Models?

Named Entity Recognition (NER) Look no further than the original BERT paper titled "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" to see a detailed analysis of how pre-trained BERT embeddings improve NER performance in section 5. The BERT diagram below shows a typical machine learning workflow for exploiting any language model for general NLP tasks. The papers also show significant improvement on Question Answering (QA) evaluated against SQUAD, and a hodgepodge of natural language understanding (NLU) tasks called GLUE.


Entity Disambiguation (ED) The global ED task also achieved new state-of-the-art results across multiple datasets using BERT. See the related work section of this "Global Entity Disambiguation with BERT" for a rundown of various workflows for applying BERT as a preprocessing step for ED.


Extractive Summarization (ES) A simple variant of BERT achieving once again state-of-the-art performance on several ES datasets can be found in "Fine-tune BERT for Extractive Summarization".


Sentiment Analysis (SA) Once again, sentiment analysis is equally graced by the existence of BERT language models in the recent paper "BERT for Sentiment Analysis: Pre-trained and Fine-Tuned Alternatives".


NMT usually doesn't benefit from Pre-trained language models It's difficult to find papers that discuss why it doesn't work, and it's easy to imagine why. Writing papers about what doesn't work is not very popular... and unlikely to gain recognition or be frequently quoted. Ah shoot - so why am I writing this article again? I found one paper that covered this topic: "When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?" and it was an interesting read. They break NMT down into two categories of tasks: NMT for low-resource languagesNMT for high-resource languages What they mean by low/high resource language is in reference to the size of the parallel corpus that can be obtained. For the world's most popular languages, it can be easy to find open-source large parallel corpora online. The largest such repository is OPUS, the Open Parallel Corpus, which is an amazing resource for any machine learning engineer looking to train NMT models. The image above shows that the open parallel corpus between English and Chinese has 103 million parallel sentences or 172K parallel documents. But what if you wanted to train an NMT model to translate Farsi to Chinese? In that case, you only have 6 million parallel sentences from 517 documents to work with. As you might expect, low-resource languages benefit from pre-trained language models and are able to achieve better performance when fine-tuning the embeddings while back-propagating errors through the NMT network. Surprisingly, however, for high-resource languages, the effect of using pre-trained language models as a pre-processing step before NMT model training does NOT result in performance gains. It's critical to point out that language models only make sense to use for machine translation if they are trained on both source and target language (for example, Chinese and English in the first example). These are commonly referred to as multilingual embedding models or language agnostic embeddings. They are able to achieve the interesting result that words in multiple languages achieve similar vector representations in the embedding space. But how are multilingual language models trained? Turns out they are trained over the exact same data as NMT: a massive parallel corpus between the source and target language. So, is there a fundamental shortcoming to language models that prevent them from being effective for this NLP task? No, language models use the same data as NMT models, and they are both built from the same powerhouse building block: the transformer. To review, language models and NMT are trained over the same data, using very similar fundamental architectures. When you consider the similarities, there isn't really anything new that the language models are bringing to the table so it shouldn't be surprising to you that BERT, ELMo, ERNIE, and our other sesame street friends aren't appearing in NMT papers touting huge breakthroughs in model performance. A skeptical reader will likely be able to poke holes in this explanation. There are certainly devisable use cases were training an LM on a large parallel corpus but then training BERT + NMT workflow on a much smaller corpus would intuitively result in performance gains. But I think it's unlikely that a serious deep learning engineer would attempt to build an NMT model without all available data at their disposal... outside of purely academic curiosities. I tip-toed over some hairy details, so I recommend reading the original paper if you're interested! I hope you enjoyed this short exploration into the intuition behind what makes NLP algorithms successful. Please like, share, and follow for more deep learning knowledge.


Differences and Connections between Natural Language Processing and Computer Vision from a Machine Learning Perspective

Differences 1. NLP is language-specific, but CV is not. NLP in a different language could be very different. Different languages have different vocabulary and grammar. It is not possible to train one ML model to fit all languages. However, computer vision is much easier. Take pedestrian detection, for example. An image taken in the US and in China usually has no significant difference. An ML model trained with images taken in China usually works also well on images taken in the US. 2. NLP labeling is harder, but CV is easier. NLP labeling is complicated due to two reasons. First, you need a specific-language speaker to label the text of a specific language. You don't expect a German speaker to label the text in Chinese. Furthermore, if the language population is small, like Uyghur or Danish, finding a good speaker to work for you might be already difficult. Not to mention how to label properly. Second, you need to understand the text in order to label. Take sentiment analysis, for example. You need to read through the text, understand it, and then decide whether it is positive, negative, or neutral. If the text is long or needs some background knowledge, it may take some time to do the sentiment analysis. Sometimes, you even need help from a linguist. On the contrary, computer vision labeling is much easier. First, the task is not language or country specific. For example, labeling for object detection or semantic segmentation can be done by any language speaker. Second, you don't need to be an expert to do the labeling. A 3rd-grade pupil can even do the labeling of object detection or semantic segmentation. 3. NLP relies a lot on rules, but CV does not. It was actually quite astonishing for me to see that rules are heavily used in NLP when I came to this domain at the beginning. But after a while, I understand why. Take named entity recognition, for example. You want to predict the label, like Loc (location) or Org (organization), of the words in a text. What will you do? Will you train an LSTM network with big training data or simply use some rules to memorize some words? For some words, like Google, Meta, Berlin, and Paris, I would just create some rules to predict Google and Meta as Org and Berlin and Paris as Loc. It is much easier to create rules for such words than to train an ML model. However, in the CV domain, rules are rarely used. People don't expect a rule-based system to predict the digits from MNIST or detect a pedestrian from a street image. ML models are absolutely dominant in the CV domain. 4. NLP data uses less disk space, but CV data takes more. In NLP, data is text. It is usually saved in notepad, excel, or a database. Since it is text, the space taken is usually a few megabytes or, at most, a few gigabytes. However, CV data is usually jpg or png images. They are usually saved directly in folders. If you have millions of images, it can easily take many gigabytes or even more.


Connections Despite many differences between the two domains, they do share many similarities. Here I will introduce two ML techniques, LSTM, and transformers, which are used in both domains. LSTM Time series problems exist in both domains. In NLP, there are problems, including part of speech and named entity recognition. In such problems, the input text is read into an ML (e.g., LSTM) model word by word, and the model predicts the label for each input word. In CV, there are problems including scene text recognition, where given an image containing a text, the model predicts the text. See the examples below. An LSTM model reads in the bounding box containing a text, transforms the bounding box into slices from left to right, and finally predicts the output letters.


References: [1] Kai Wang, Boris Babenko and Serge Belongie, End-to-End Scene Text Recognition, ICCV, 2011. [2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko, End-to-End Object Detection with Transformers, ECCV, 2020.


Trends in AI - August 2022

Research Every month we analyze the most recent research literature and select a varied set of 10 papers you should know of. This month we're covering topics such as Reinforcement Learning (RL), Scaling Laws, Information Retrieval, Language Models, and more.


Trends in AI - August 2022

1. Beyond neural scaling laws: beating power law scaling via data pruning By Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, Ari S. Morcos.  Why -> Scaling laws¹ is a pervasive empirical phenomenon in modern Neural Networks, where the error is observed to off as a power of the training set size, model size, or both. While some have embraced this fact to devise a research agenda focused on scaling up, many think there must be ways to build better models without the need for outrageous scale. This paper explores a technique - data pruning - that can improve the learning efficiency of NNs "beating" scaling laws.  Key insights -> In the context of this work, pruning refers to dropping training data samples from the training dataset, not pruning weights of the Neural Network. The intuition behind the method proposed is simple: imagine you could rank samples in a training dataset from "easy to learn" to "hard to learn" for a model. A typical dataset will contain too many of the easy-to-learn samples - that is to say, fewer would suffice to reach good performance on those samples - and too few from the hard-to-learn side - meaning you'd need more examples to train the model in properly. One way of solving this problem is to scale up the whole training dataset because given enough scaling up - assuming the data distribution is uniform - you'll get enough of the "hard to learn" samples eventually. But that's very wasteful. What if we could use a priori to curate a training dataset that contains a better balance of easy-to-learn and hard-to-learn samples? This is what this paper investigates. This problem can be formalized as trying to find a pruning metric to assign to each training sample (a proxy for its hardness), which is then used to prune the training dataset to the desired size. They propose a new metric in this paper that is comparable to existing work requiring labeled data. However, the most interesting contribution, in my opinion, is their section on data pruning without labels. They perform k-means clustering on the embeddings of a pre-trained ImageNet model and define the hardness of each sample as its distance to the nearest centroid: the intuition is that prototypical samples that are easy to learn will be closest to a centroid, whereas hard-to-learn samples will fall further away from their cluster centroids. The results show how around 20% of the training samples from ImageNet can be pruned without sacrificing performance. To be fair, the results in the paper are not jaw-dropping, but the key ideas behind it have the potential to be useful in other tasks such as image segmentation, language modeling, or any other multimodal dataset curation.


Trends in AI - August 2022

2. Denoised MDPs: Learning World Models Better Than the World Itself  Project Page  Code By Tongzhou Wang, Simon S. Du, Antonio Torralba, Phillip Isola, Amy Zhang, Yuandong Tian.  Why -> The world contains a lot of information that's irrelevant for you to be able to navigate it. At the core of many Machine Learning techniques is the ability to discern relevant and useful signals - or patterns - from noise.  Key insights -> This work formalizes the problem of separating "the good from the irrelevant information" in the context of reinforcement learning by identifying information that's both controllable by the agent and relevant for the reward, as described in the figure below. Based on this idea, the authors propose Denoised MDPs (Markov Decision Process), a method for learning a factorization of state representations that disentangles the controllable and reward-relevant bit of the state using information theoretic principles. The gist of it is that different factors of the state should be maximally or minimally predictive of other factors depending on their relationship, which enables the authors to set up a variational objective for the agent to optimize (you'll need to check out the paper for the real mathy goodies). The result is a world model that explicitly models what information should be discarded as noise and what information should be used for modeling the agent's decisions. The authors prove how this approach is competitive in the DeepMind Control Suite, and fascinatingly, they showcase qualitatively how the Denoised MDP representations work by training a decoder on reconstructing the input so you can understand what the signal representation of the state learns to capture. You can watch those on their project page.


Trends in AI - August 2022

3. Parameter-Efficient Prompt Tuning Makes Generalized and Calibrated Neural Text Retrievers  Code By Weng Lam Tam, Xiao Liu, Kaixuan Ji, Lilong Xue, Xingjian Zhang, Yuxiao Dong, Jiahua Liu, Maodi Hu, Jie Tang.  Why -> Prompting has been making strides in NLP for the past couple of years, and now it also seems promising for Information Retrieval.  Key insights -> Prompt tuning is a technique for adapting a pre-trained frozen model to a given task by adding trainable prompt tokens to the input of the sequence model. One of the main advantages of this approach with respect to the more common full model finetuning is that it only requires retraining a small subset of parameters which is much more efficient and also enables higher reusability of the original pre-trained model. Now they train both a Dense Passage Retriever (retrieval by nearest neighbor search of query and doc embeddings) and a ColBERT⁴ type model with late interaction (includes joint modeling of query and document). The difference here is that instead of finetuning the whole model, they only finetune a prompt while keeping the pre-trained LM frozen. They base their implementation on the P-Tuning v² method, which is a more advanced form of prompt tuning where the trainable prompt is not only appended to the input but also at each layer of the Transformer. The most interesting part of the results is that of generalization. While prompt tuning performs comparably to full fine-tuning on in-domain benchmarks, it outperforms substantially in various cross-domain datasets from the BEIR³ benchmark. Yet again, this work strengthens the hypothesis that prompting and all its advantages are a viable alternative to fine-tuning and will probably keep increasing in popularity.


Trends in AI - August 2022

4. DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale  Code By Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, Yuxiong He.  Why -> DeepSpeed - the framework for large-scale distributed training of large NNs developed and used by Microsoft - now also for inference besides training.  Key insights -> The landscape of large Transformer architectures has diversified in the past year since the success of a sparse Mixture of Expert models in scaling up models in the Trillion parameter regime. The key advantage of these architectures is that they have increased expressive power from their larger sizes, but at inference time, only an input-dependent subset of the weights is used, which makes them more efficient (if the implementation is also optimized!). The downside is that training and running these models efficiently is more involved because most existing Deep Learning libraries and hardware were not designed with this type of computation. While DeepSpeed was previously designed to enable training large Transformers, the latest update focuses on making inference faster both in latency and throughput on all kinds of Transmodels, including sparsely activated architectures. We're talking about a system that enables parallelism in heterogeneous hardware at a scale of hundreds of GPUs, CPU, and NVMe memory that enables high-speed inference even when large models can't fit in aggregate GPU memory. Let's be honest, though, most people reading this article won't ever have the need to use such a framework for training Trillion-scale models themselves, but it's still a fascinating read if you're interested in the engineering that goes behind training and running massive Neural Networks.


Trends in AI - August 2022

5. Language Models (Mostly) Know What They Know By Saurav Kadavath et al.  Why -> Performance is far from the only desirable property of ML models. Accurately knowing how confident they're in their outputs can be more important, especially in safety-focused applications.  Key insights -> Calibration is the concept used in Machine Learning to indicate how well-tuned a model's prediction confidence is (e.g. a perfectly calibrated model with an output with 90% certainty should be right 9/10 times, no less, no more). This work first investigates the calibration of LMs answering questions by formulating the prompt in a multiple choice fashion where the output of the model is a single token with the answer. Given that a single token is an answer, the probability can be calculated directly from the model's likelihood of that token, normalized over valid token answers (e.g. A, B or C). While LMs are highly sensitive to prompt design and formatting, given an appropriate question formulation, large LMs are very well calibrated. Interestingly, this capacity breaks down at smaller scales (see figure below), and also at the tail of very low or very high confidence. The paper dives into more modes of calibration comparisons and analysis than you'll find in the paper, but the takeaway remains: LMs know what they know (are well calibrated), but results are still very vulnerable to prompt variations, and models need to be large enough.


Trends in AI - August 2022

6. Towards Grand Unification of Object Tracking (Unicorn)  Code By Bin Yan, Yi Jiang, Peize Sun, Dong Wang, Zehuan Yuan, Ping Luo, Huchuan Lu.  Why -> Decluttering and unifying Machine Learning model architectures have proven to be fruitful in NLP in the past few years, now time for video Computer Vision tasks.  Key insights -> When it comes to video-related tasks, existing top-performing models still tend to rely on task-specific designs and overspecialize for their specific application as a result. The authors propose a single model architecture that performs competitively in 4 modes for Object Tracking: Single Object Tracking (SOT), Multiple Object Tracking (MOT), Video Object Segmentation (VOS), and Multi-Object Tracking and Segmentation (MOTS). This architecture is quite complex and is best understood through the figure below. In broad strokes, it starts with a unified backbone for embedding images, then a unified embedding is computed for the reference and current frame. A Transformer is used for the feature interaction between the unified embedding and the current frame to output class, box, and masks corresponding to all object tracking flavors. The system is benchmarked on several object tracking benchmarks such as LaSOT, TrackingNet, MOT17, BDD100K (and others) and sets new state-of-the-art performances in most of them.


Trends in AI - August 2022

7. Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? By Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q. Tran, Dani Yogatama, Donald Metzler.  Why -> Are we missing phenomena by benchmarking models at a particular compute/data scale? Yes.  Key insights -> The authors perform hundreds of experiments across scales for a wide range of architectures: vanilla, efficient and advanced Transformers, MLP mixer, and Convolution-based models. The experiments consist of pretraining with autoregressive language modeling, which results in an upstream performance (language perplexity), followed by supervised fine-tuning on GLUE, SuperGLUE, and SQuAD (downstream performance). The TL;DR is straightforward. While not the best option in all scale regimes, the vanilla Transformer shows the most robust and consistent performance scaling results across scales. Other useful tidbits of wisdom are: Convolution and MLP-based architectures do well on pretraining (upstream performance) but fail to transfer properly when finetuned. This points toward the importance of architectural inductive biases when it comes to transferring learning.Efficient Transformers only perform competitively with their full-attention counterparts in a certain scale regime and lag behind if scaled up sufficiently.


Trends in AI - August 2022

8. Discrete Key-Value Bottleneck By Frederik Träuble, Anirudh Goyal, Nasim Rahaman, Michael Mozer, Kenji Kawaguchi, Yoshua Bengio, Bernhard Schölkopf.  Why -> As the benchmarking culture in ML slowly but surely shifts focus to out-of-domain generalization, inductive biases tailored for it will become more relevant.  Key insights -> This method is part of a longer research agenda that we've highlighted before from Bengio's group (e.g. Coordination Among Neural Modules Through a Shared Global Workspace) focused on how introducing bottlenecks of information flow in architectures leads to representations that are robust and generalize better in an out of distribution sets. In this case, the proposed method consists of 3 steps: Encoding a high dimensional input (e.g. image) into an embedding with an encoder pre-trained on a large dataset.Splitting the embedding into C lower dimensional heads and finding the nearest neighbor from a set of predefined vectors which are frozen during training. Then use that nearest neighbor's representation across heads to reconstruct the embedding. The discretized value codes are frozen during the training of the decoder.The decoder gets the reconstructed embedding as input and produces the task-specific output. The experiments focus on the setting where a model is trained for one distribution and then adapted to a new one, as you can see in the figure below. The model is initialized by training on i.i.d. data, updating the decoder weights, and keeping the codebooks. When adapting the model to a distribution shift, the decoder is frozen, and only the codebook values are updated. Their experiments prove how this approach reduces catastrophic forgetting and results in more robust predictions. This work will not have a big short-term impact - results are not groundbreaking - but some of these ideas could be key catalyst for the next leap in true generalization.


Trends in AI - August 2022

9. Language Model Cascades By David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A. Saurous, Jascha Sohl-dickstein, Kevin Murphy, Charles Sutton.  Why -> Large Language Models have become so powerful that they're increasingly used as a black-box building block for other applications such as Reinforcement Learning or data augmentation.  Key insights -> This work formalizes the interaction of Language Models from the lens of probabilistic programming: directed graphical models of random variables, which map to natural language strings. This turns out to be a powerful language to analyze existing advanced prompting techniques such as Scratchpad⁶ or chain-of-thought⁷. Moreover, it can be used to design new forms of interaction between LMs or advanced promoting techniques. The practical value of this method is still a bit of a question, but it has the potential to be a powerful conceptual tool for building complex ML systems by abstracting away the classic Deep Learning toolbox (weights, embeddings, gradient descent, objective functions), creating the space for new higher-level mechanisms for reasoning.


Trends in AI - August 2022

ZeroC: A Neuro-Symbolic Model for Zero-shot Concept Recognition and Acquisition at Inference Time By Tailin Wu, Megan Tjandrasuwita, Zhengxuan Wu, Xuelin Yang, Kevin Liu, Rok Sosič, Jure Leskovec.  Why -> Neurosymbolics delivering on its promise..?  Key insights -> ZeroC is a method to represent concepts as graphs of constituent concept models (i.e. primary shapes like lines). The main objective of this paper is to build a system that can recognize unseen concepts at inference time. For instance, in the figure below, the letter F was not seen by the model, but it's able to disentangle its components (lines) and their relationships (angle and positions), representing them as an explicit graph with 3 nodes and 3 edges. The recipe for training such a system relies on Energy-Based Models (EBMs): feed positive and negative image/graph representation pairs and minimize the energy of positive pairs while accounting for the graph invariances of the representation. The experiments show success in modest gridworld environments where the primary shapes and relationships are quite simple, but this represents a first step toward learning structure-rich representations that could become useful in the context of few-shot learning and generalization.


The Curious Case of How MS-excel Was a Nightmare for Bioinformatics

if you have found it interesting: You can look for my other articles, you can also subscribe to get notified when I publish articles, and you can also connect or reach me on LinkedIn. Thanks for your support! Here is the link to my Github repository, where I am planning to collect code and many resources related to machine learning, artificial intelligence, and more.


How To Make STGNNsCapable of Forecasting Long-term Multivariate Time Series Data?

STEP: First of all, Do Not Be Confused: Spatial-temporal graph data = multivariate time series Here, the data (traffic flow) used is recorded time series data on the road by sensors. Did you see those two patterns in Figure 1 above?? Answer: there are two repeating patterns: 1. daily 2. weekly periodicities First, STGNNs is the abbreviation of "Spatial-Temporal Graph Neural Networks" for those who don't know/ know meager. (not difficult, it just needed to be googled; mentioned for those who don't want to lose time or be distracted) STGGNNs = Sequential Networks + Graph Neural Networks (GNNs) We use GNNs for dealing with relationships between time series and Sequential models for instructing time series patterns. By the combination of these two terms, we can grasp outstanding results. By the way, there is no free lunch - as researchers said. It means powerful models demand complicated architectures; consequently (in most cases), the computational cost rises (linearly or quadratically) with the input length. Also, don't forget the size of our time series, which is usually considerable. STGNNs, like other models, can predict small windows to make forecasting. This ability to rely on small windows makes the model unreliable. Problem: 1. STGNNs can't capture long-dependencies. 2. The dependency graph is missing. Solution: STEP (STGNN is Enhanced by a scalable time-series Pre-training) · a modified version of STGNNs Illustration: Two initiatives: 1. proposing TSFormer, a transformer-based block with an autoencoder (encoder-decoder) structure as an unsupervised model. This TSFormer able to capture long dependencies. 2. Proposing a graph architecture learner to learn dependency graphs. After proposing these two, we just need to weld them for a joint model, that is the final solution. That's it!! Sounds easy?! Let's make them as simple as possible.  Let's see the proposed architecture: As you can see from figure 2, the model includes two phases: phase 1) pre-training The scheme is a masked autoencoding model which is trained for time series data relying on Transformer blocks (TSFormer). This model is able to capture long-dependencies and turn out segment-level representation which includes some valuable information. phase 2) forecasting In this phase, the pre-trained model from the previous phase (which captured long-dependencies) is used to modify the downstream STGNN. Additionally, a discrete and sparse graph learner is designed just in case the pre-defined graph is missing. That's all I have done in general. Thus, let's dive more into the details of these two phases:


1. The Pre-Training Phase This attempt, I mean using a pre-trained model, is due to an increase of interest (and, of course, results) in applying them in NLP projects. Though pre-trained models are adopted widely in NLP (which is sequential data), there are some differences with time series. You can read its full description in my previous article: "How to Design a Pre-training Model (TSFormer) For Time Series?"


2. The Forecasting Phase The input here is divided into P non-overlapping patches of length L. Our TSFormer produces indications for each input (Si) of the forecasting phase. One of the STGNNs' features is that they take the newest. Therefore, based on those produced indications by TSFormer, we will modify STGNNs.


STEP From ZERO  The Process The encoder section in this phase is the same as it is in TSFormer; its description is not provided here to prevent making this article too long. If you want to know the details, you can read another article I have demonstrated completely(How to Design a Pre-training Model (TSFormer) For Time Series?).


The structure of learning in graphs Problem) most graphs depend on a pre-defined graph that is unavailable or not good enough in most cases-also, mixing the way of learning (seeking the relationship between nodes (for ex. i and j) of the time series and STGNNs leads to great complexity. Solution) pre-trained TSFormer Interpretation) Proposing a discrete sparse graph. How? 1. graph regularization to fit supervised information. 2. a KNN graph to rein the sparsity. Its formulations are summarized below:


Downstream spatial-temporal graph neural network problem) usual STGNNs' input: last patch+dependency graph solution) STEP (which adds the input patch's representation to the input) interpretation) As we discussed in my previous article, "How to Design a Pre-training Model (TSFormer) For Time Series?", TSFormer captures long-dependencies; consequently, it makes H rich in aspects of information. Also, WaveNet is selected as our backend, which assists in capturing multivariate time series properly. But how?? It blends graph convolution with dilated convolution. Consequently, our forecasts are supported by WaveNet's output latent, hidden representations. How?? By using MLP. Q) If you look at the Forecasting phase architecture, you'd see two streams into Spatial-Temporal Graph NN Block. So, how can we manage that? A) by using Eq7: In the end, the forecasts are made by MLP: The output of the downstream STGNN:


data: The model is trained on three traffic speed datasets in three regions in the USA: METR-LAPEMS-BAYPEMS04


Metrics: MAE (Mean Absolute Error)RMSE (Root Mean Absolute Error)MAPE (Mean Absolute Percentage Error)


What's It Like to Work in Applied AI?

Disclosure and Accuracy This article is written from the perspective of an early-career stage: 2+ years in the tech space, 1+ years of applied AI/ML experience, with an (expected) bachelor's level degree in engineering. The bulk of experience lies in start-up, small, and medium-sized environments, which are inherently less structured than established organizations. The author strives to maintain accurate information that is useful to a general audience. However, note that information on this topic evolves quickly, and practices can be wildly different from organization to organization, so there is an abundance of conflicting opinions from various sources.


What's It Like to Work in Applied AI?

Intended Audience This article is intended for other early-career engineers in the field of applied AI, software engineers interested in a career transition, and anyone else with interest in this topic.


What's It Like to Work in Applied AI?

What is Applied AI? There are plenty of existing resources to dig deeper into this, but in summary, applied AI is: In essence, applied AI is taking scientific research and applying it to business problems.


What's It Like to Work in Applied AI?

What is a Machine Learning Engineer? It would be helpful to start with a definition of machine learning. It's defined as: Essentially, "AI" is the idea of an "intelligent" machine. Machine learning is the process used to develop that "intelligence". So in this context, a Machine Learning (ML) Engineer designs and builds the infrastructure needed to enable the process of machine learning. We'll discuss more on the particular responsibilities of an ML Engineer later.


What's It Like to Work in Applied AI?

Expectation vs. Reality How does this differ from how we traditionally understand the role of an engineer who works in the field of "Machine Learning" and "Artificial Intelligence"? It's likely we think of someone who does the following: a lot of research and reading into published papers,approaching new and novel problems,planning and conducting experiments to invent new methods to solve these problems,publishing new papers. These responsibilities are actually fulfilled by someone we'd call a "Research Scientist" or even a "Data Scientist". We're falsely attributing theory to practice. In reality, this is actually what we can expect to see in practice: We can see that staying informed of current research and approaches to problems by reading papers is unchanged. However, each other point has an analogous responsibility more reflective of a machine learning engineer's day-to-day. We'll talk more about these distinctions a bit later. First, we have to discuss some industry jargon that is thrown around in the "Applied AI" space.


What's It Like to Work in Applied AI?

Job Titles This is the area where you'll experience the most conflicting opinions. The separation of responsibilities according to different job titles is not consistent between organizations or even sub-teams within the same organization. Trying to understand different descriptions and convoluted diagrams is largely a waste of time and will make you even more confused about the distinctions. Trust me, when I tried for this article, I started to question what my own job even is. You'll hear some subset of the following titles: Data EngineerData ScientistApplied Data ScientistData AnalystBusiness AnalystML EngineerMLOpsData Infrastructure EngineerDevOps Organizations describing job postings, as well as individuals describing past experiences, use many of these terms interchangeably, so it's very hard to find consistent data. At the risk of adding more confusion to this argument, I'll define my own generic titles that have no major significance outside of this article, according to how they fit into the data science process.


What's It Like to Work in Applied AI?

Data Science Process We can see five distinct "roles" in the above diagram: Business: This stage involves understanding and formulating the business problem and defining the goal of the project. This role would typically be called a "Business (and/or Data) Analyst".Data: This stage involves iterating until the desired amount and quality of data is achieved for feature engineering. These include understanding potential data sources that could be used to solve the business problem (assessing key factors such as availability, usability, quality, and cost), collecting said data, and cleaning (mitigating outliers, resampling, etc.) the collected data. This would be assigned to a combination of the "Data Engineer" and "Data Labeler". Building the data pipeline would typically be assigned to the "Data Infrastructure Engineer".Model: This stage involves two parts: feature engineering and modeling. The collected data is used to perform feature engineering. A model is selected, built, and trained. The model is then evaluated against the metrics set in the "Business" stage. If it does not achieve the required metrics, we have to assess the problem and return to a previous step. This role has the most inconsistency in terms of job title. It can be any of: "Data Scientist" (the most incorrect), "Applied Data Scientist," or "Data Analyst".Operations: This stage involves deploying, serving, and monitoring the completed model. Monitoring is a continuous action for the lifetime of the model. This role would traditionally be called "DevOps" (software development and IT operations) but is typically called "MLOps" (machine learning operations) for machine learning projects.Maintenance: Finally, if monitoring detects the model is no longer achieving the desired metrics, we must revert to a previous stage in the process. Covering the missing title from above, the responsibilities of an "ML Engineer" are Data, Model, Operations, and Maintenance, typically with an emphasis on Data and Model.


What's It Like to Work in Applied AI?

Early-Career Perspective Now that we understand what "applied AI" is from a high-level and the common roles you'll hear in the field, what does this mean for you? In this section, we'll discuss what you can expect as an early-career ML Engineer.


What's It Like to Work in Applied AI?

Common Misconceptions We'll start by walking through some misconceptions about the field. If you tell people you work in machine learning, you'll often hear, "wow, you must be really good with statistics and math," and my response is usually, "Maybe? But it's not incredibly relevant for my job". This is to say that this statement is true to an extent but heavily exaggerated. At its backbone, machine learning is just a new application of old mathematics: statistics, linear algebra, and calculus. It's important to understand the fundamentals to understand how to apply machine learning truly effectively. However, this only goes so far. As an ML Engineer, will you ever need to implement backpropagation from scratch? Probably not. Modern ML frameworks abstract all of this fundamental understanding away (for better or for worse). Will you need to understand the math used in research papers (at a high level)? Absolutely. This is required knowledge to adapt their approach to your own business problem effectively. This is untrue from an early-career perspective in the applied AI field. Many that work in this field do have advanced degrees, but that doesn't imply that you need an advanced degree. Many that I have worked with prove that this isn't necessarily true (at least anecdotally). Practical machine learning engineering is quite a bit more traditional software engineering than you'd think. Machine learning engineers are engineers at their core. ML engineers utilize scientific research to solve business problems. They care more about how to make the solution work in practice. Most problems are solved problems, and understanding and adapting existing solutions to a business use case is the primary goal.


What's It Like to Work in Applied AI?

Where do you fit in? As an early-career machine learning engineer, where in this process do you fit? This can heavily depend on the organization, but I'll describe this as generally as possible. Referring to the "Data Science Process" diagram, I'll break this down into individual sections. Business: Generally, you will not be involved in this stage. You may be approached to discuss technical feasibility and, in some rare cases (specifically start-ups), may be involved in prioritizing business problems.Data: You'll be quite involved in this stage-specifically, understanding data sources and cleaning data. Data collection is often outsourced in larger organizations (if it's needed at all). In rare cases (again, start-ups), you'll collect the data yourself. This is generally the most lengthy (and most important) stage.Model: You'll be very involved in this stage. Feature engineering and modeling are generally your full responsibility.Operations: You'll likely not be involved in this stage. Most organizations have a dedicated DevOps team that will handle this. The exception would be configuring model monitoring tools specific to your application.Maintenance: You'll be responsible for future maintenance of your deployed models. I'd suggest you make this process as easy as possible for yourself (if you're wondering what could go wrong, check out a previous post).


What's It Like to Work in Applied AI?

Conclusion We've seen what "Applied AI" is and the job titles of those who work in the field. We also saw the various stages of the data science process and where the "Machine Learning Engineer" fits in. Using this knowledge as a foundation, we explored the key responsibilities of an early-career machine learning engineer and some common misconceptions about the field. I hope this article paints a realistic picture of the early-career environment of the applied AI field and helps guide you on the next step of your career.


What's It Like to Work in Applied AI?

Sources [1] "Applied Artificial Intelligence," Cognizant Technology Solutions. [Online]. Available: https://www.cognizant.com/us/en/glossary/applied-ai. [Accessed: 24-Jul-2022]. [2] "Artificial Intelligence vs. Machine Learning," Microsoft Azure. [Online]. Available: https://azure.microsoft.com/en-us/solutions/ai/artificial-intelligence-vs-machine-learning/#introduction. [Accessed: 24-Jul-2022]. [3] C. Nantasenamat, "The Data Science Process," Towards Data Science, 27-Jul-2020. [4] A. Burkov, Machine Learning Engineering. Québec, Canada: True Positive Inc., 2020. [5] D. Wang, "The Engineering Profession," in ECE 190, 2018.


Meta AI's Make-A-Scene Pushes the Boundaries of AI Art Synthesis

Make-A-Scene Meta AI's Make-A-Scene addresses some of the aforementioned challenges by combining text-to-image and image-to-image generation in a single and highly scalable model. The use of image inputs in the form of sketches helps to improve the accuracy of the produced output, refining aspects such as positions, relationships between objects, expressions, etc. Make-A-Scene's architecture is comprised of an autoregressive transformer that uses text and image tokens as input. Additionally, the transformer uses conditionally controlled scene tokes using segmentation maps. During the inference phase, the segmentation tokens are generated from the input images or directly from the transformer. To effectively manage the loss, Make-A-Scene uses Vector-Quantized Variational Autoencoders (VQVAE) to encode and decode the image and scene tokens with explicit losses targeted at specific image regions correlated with human perception and attention. The result was a model that can output high-resolution images from text inputs while improving the controllability and human perception of the generated outputs.


Make-A-Scene in Practice To validate the capabilities of Make-A-Scene, Meta AI made a version available to well-known AI artists such as ofia Crespo, Scott Eaton, Alexander Reben, and Refik Anadol. Some of their tests resulted in astonishing creations: Make-A-Scene represents an interesting evolution in text-to-image synthesis. Meta AI didn't open source implementation, but there have already been some attempts to recreate the model outlined in the paper.


Trends in AI - July 2022

Research Every month we analyze the most recent research literature and select a varied set of 10 papers you should know of. This month we're covering topics such as Machine Translation (ML), Reinforcement Learning (RL), Diffusion Models, Information Retrieval, Radiance Fields, and more.


Trends in AI - July 2022

1. No Language Left Behind: Scaling Human-Centered Machine Translation  Blog post  Code By NLLB Team, Marta R. Costa-jussà, James Cross, Angela Fan, et al. Why -> A single Machine Translation Model with state-of-the-art quality for over 200 languages which improves the average BLEU score by 44%. That's by itself a big deal. Add the inclusion of many low-resource languages for which MT was not available before, and the fact that all of this is now available as open-source, and you get the impact. Key insights -> NLLB manages to train a single sparse Mixture of Experts Transformer model with 54 Billion parameters on automatically mined bitext data from over 200 languages. A lot of the effort, described in great detail in the excellent 190-page paper, is in the methods to mine the data, clean it up and make sure the model does not produce toxic hallucinations. Training the model involved many neat tricks like Layer Dropout, Curriculum Learning, and Model Distillation. A new multilingual sentence embedding model called LASER3 plays a central role in all of this. For some of the finer technical details, also check out the earlier paper "Tricks for Training Sparse Translation Models" or the exclusive Zeta Alpha interview with Angela Fan, one of the key scientists on the project. The NLLB paper, or book, really, is a great read, covering everything from technical detail, linguistic analysis, ethical considerations, and impact analysis up to benchmarking. The average BLEU score improvement by 44% on the FLORES-101 data set is impressive... But of course, everyone is curious about how this model compares to Google Translate. And that was evaluated as well. On average, the quality of the NLLB-200 model is similar with a clear edge for lower resource languages. It adds 70 or so languages for which no (good) MT models were available before. The most important difference? NLLB-200 is fully open source under the MIT License. Among other goodies, this makes available: The Flores-200 Evaluation dataset in 204 languages. NLLB-Seed: Seed training data in 39 languages. LASER3: sentence encoders for identifying aligned bitext for 148 languages. Training data recreation: Scripts that recreate the NLLB training data. And finally, the MT Models cover 202 languages, including: NLLB-200: A 54.5B Sparsely Gated Mixture-of-Experts model3.3B and 1.3B Dense Transformer models1.3B and 600M Dense transformer models distilled from NLLB-200Training and generation scripts to reproduce our models Applause to Meta for sharing this with the world. It will for sure greatly advance further MT research, global access to knowledge, and cross-language communication. We hope it also helps with better content moderation in conflict zones, a known issue for Facebook, and that the open-source nature of the model will not contribute to an increase in mass surveillance and censorship in those same language areas. After all, the road to hell is paved with good intentions.


Trends in AI - July 2022

2. Evolution through Large Models By Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, Kenneth O. Stanley.  Why -> While large Language Models (LMs) themselves are fertile ground for research, a trend is emerging where LMs are used as a building block of algorithms to solve other tasks. For instance, augmenting a dataset [1]. Now they're used in the context of evolutionary computing: as a mutation operator.  Key insights -> Evolutionary algorithms are algorithms that follow evolutionary principles to solve a task: having a population of candidate solutions that can be recombined, mutate and interact with their environment to gradually adapt to changing conditions or optimize some fitness objective. Just like how other lifeforms on Earth came to exist. Genetic programming is the task of generating computer programs (program synthesis) with evolutionary algorithms. One of the biggest challenges of using evolutionary computing for program synthesis is that if you mutate a candidate program to obtain a new candidate purely randomly, you'll search through the space of possible programs very inefficiently. Is there a public source of code diffs along with annotations on the changes a mutator operator could learn from? Bingo. GitHub's code diffs and commits messages! This work constructs a dataset with these and trains a large LM to learn human-esque ways to improve code, discovering mutations that are more sophisticated than those following handcrafted rules. Moreover, this large LM can be continuously finetuned in a Reinforcement Learning setting where positive reward mutations are fed again for training the LM. Making this all work is quite complex, involving a lot of careful curation of training data and balancing all moving parts. The authors showcase how this algorithm works with a game called Sodarace, whose objective is to invent 2D creatures made of point masses and springs that move in space. The model has to generate raw Python programs that invent these creatures, starting from a basic seed which is a program generating a basic sodaracer. The authors believe that the evolutionary approach directed by an expressive LM leads to superior open-endedness, the main overarching theme across the paper. This could revive the interest in evolutionary algorithms, which had arguably stalled in the past years under the shadow of Deep Learning.


Trends in AI - July 2022

3. Diffusion-LM Improves Controllable Text Generation By Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, Tatsunori B. Hashimoto.  Why -> Diffusion models keep conquering domains.  Key insights -> It's tricky to control the generation of text from a Language Model to satisfy certain constraints such as sentiment and structure (e.g. generate a sentence that's positive and is subject-verb adjective object). Previous works on diffusion applied to text treated language as discrete, applying the diffusion process by iteratively corrupting tokens discretely, which the model had to learn to reconstruct (a little bit like iterative masked language modeling). In this work, however, they apply the diffusion process directly on the continuous embeddings, which is closer to the diffusion process we're used to in images. Embeddings iteratively drift in the high dimensional space and are only discretized into tokens at the very end of the process. One of the main benefits of using a diffusion model as a language model is that it lends itself very easily to steering the generation throughout the diffusion process by evaluating how well a constraint is being satisfied at each diffusion step and using the gradient of that objective to update the diffusion intermediate states (see figure above). The authors show how this setup works in several controlled language modeling tasks, controlling for aspects such as semantic content, parts-of-speech, syntax trees, and syntax spans. The empirical results show how this method strongly outperforms previous works doing controlled text generation.


Trends in AI - July 2022

4. Memory-Based Model Editing at Scale By Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, Chelsea Finn.  Why -> One of the main limitations of current GPT-like LMs is that their knowledge about the world is frozen once training is finished. How can we overcome this limitation?  Key insights -> Previous works tackled the problem of updating knowledge in LMs by letting them interact with some sort of explicit knowledge base that can change. For instance, web-GPT [5] can search documents from the web and append the top-k results to the prompt such that it can attend to that context to answer questions like "Is Messi at Barça"? which improves the factual accuracy of the model when compared to regular LMs. The method proposed by this paper is conceptually very simple: allowing users to add entries to an edit memory consisting of question-answer pairs. During inference, a scope classifier determines whether the prompt is within the scope of any of the memory edits and redirects the input accordingly to one of two models: the original frozen LM or a counterfactual model which takes the relevant edit memory as a context for the prompt (see figure below). One of the main advantages of this approach is that it doesn't require constant curation of the whole knowledge corpus unlike something like web-GPT, but simply iteratively updating user-added facts. Moreover, it can be added to existing large frozen LMs out-of-the-box without the need to retrain from scratch with some non-parametric memory system.


Trends in AI - July 2022

5. A Neural Corpus Indexer for Document Retrieval By Yujing Wang et al.  Why -> We highlighted the Differentiable Search Index (DSI) back in march: fully parametrizing a whole corpus in a neural network and performing document retrieval by the autoregressive generation of document identifiers. This work takes that to the next level!  Key insights -> The Differentiable Search Index [6] is a wild approach to Information Retrieval (IR) that we highlighted back in March. Instead of retrieving a document by doing some lexical matching, embedding nearest neighbor search, or reranking via cross-encoding; the model simply memorizes the document corpus and generates autoregressively a list of document ids that are relevant to the given query. This is a priori mind-blowing cause document ids contain no semantically relevant information: if a new document appears and is given a new id, you can't infer anything about its content. This retrieval paradigm has several obvious downsides such as the fact that the document collection cannot be easily extended after training is finished, but the initial results were extremely encouraging, especially in the zero-shot setting where the model learns without human-annotated query-doc pairs, and it learns from a self-supervised objective that relies solely on the document collection. This work basically squeezes more performance from the DSI idea by applying several design improvements: a query generation model (Doc2Query [6]) that extends the coverage of what queries, using the best performing semantic ids (proposed in DSI), and a special weight decoder. They test their model on the NQ320k retrieval task and their results are insanely good, although there's a catch. Some of the models they compare themselves to were only trained on MS-Marco and then tested on NQ320k so it's not really a fair comparison. Still, compared to the original DSI, they achieve an incredible performance bump (e.g. a recall@1 from 40% to 89%). Judging by their ablations, only the query generation module has a massive impact on performance, whereas the other proposed methods such as their special decoder barely make a difference, which is a very relevant finding.


Trends in AI - July 2022

6. Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos  Blog post  Code By OpenAI et al.  Why -> Labelling is expensive and Reinforcement Learning tends to be very inefficient, especially in open games such as Minecraft. This paper proposes a very technique to work around this by reliably labeling gameplay videos automatically.  Key insights -> Learning what will be the next move of an agent in a videogame given previous images and moves (in other words autoregressive) is hard. However, if you let the model access the whole past and future frames of a videogame, guessing the action being taken at a given frame becomes easier. They leverage this simple trick to be much more efficient with how much human labeling is required. First off, let me say the title of this paper is arguably misleading: Video PreTraining (VPT) actually does label actionsfrom videos. But instead of having people do it, they train a model to annotate raw videos with actions. Instead of doing it autoregresssively (i.e. guess what action will be taken given past action/frames) they let their model access past and future frames, which makes the task of annotating actions much easier to learn (i.e. guess what action is being taken given past and future video frames). They call the model that learns to label raw videos an Inverse Dynamics Model (IDM), which is trained on 2k hours of data collected by having people play Minecraft while logging all their actions. The video dataset they collect consists of 270k hours of Minecraft gameplay from the internet, which are filtered and cleaned down to 70k, hours which are then automatically labeled with the IDM model. An agent is trained on this dataset via Behavioral Cloning: predicting the action that should be taken by only seeing the previous frames and actions. One of the key results is how much it helps to look into the future for labeling videos to construct the IDM model. Besides Behavior Cloning, they also investigate fine-tuning with Reinforcement Learning, where the model has the opportunity to explore arbitrary trajectories and get rewards for it. While a randomly initialized RL agent fails to discover important complex action sequences in the game, RL finetuning works much better when initialized from the VPT model trained on BC, which discovers complex action sequences much more consistently and efficiently.


Trends in AI - July 2022

7. Solving Quantitative Reasoning Problems with Language Models  Demo page  Blog post By Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh et al.  Why -> Solid reasoning ability - especially quantitative - is often pointed out as one of the weaknesses of large Language Models. Is this a fundamental limitation of large LMs or can it be circumvented simply by scaling up and being more clever about the training data?  Key insights -> It turns out that by training on the right data and with the right tricks, LMs are skilled solvers of quantitative reasoning problems such as math or physics. This work builds upon PaLM [8] (with up to 540B parameters!) adding 118GB of data with mathy text from arXiv and math web pages. To encode the math, they simply leave the rax LaTeX expressions instead of flattening them out like previous works did. All models they train are a finetuning of the base PaLM. They also find that using advanced prompting is essential for performance: chain of thought and scratchpad prompting; which consist of simple prompt reformatting strategies that have shown to empirically improve general reasoning abilities of pretrained LMs. Finally, they also find that using majority voting also substantially improves the performance of the model, where a pool of candidate solutions for a problem is generated and a solution is chosen among the solutions that share the most common result. This is undoubtedly still far from robust symbolic reasoning, it shows how far plain language modeling can get in reasoning with the right tricks and optimizations. If you're interested in the subfield of reasoning with LMs, check out the recent Unveiling Transformers with LEGO: a synthetic reasoning task.


Trends in AI - July 2022

8. A Path Towards Autonomous Machine Intelligence (JEPA) By Yann LeCun.  Why -> After Bengio's consciousness prior [3] and Hinton's GLOOM [4], Lecun joins his co-Turing Award Winners by presenting what he considers the path towards human-level AI.  Key insights -> This long position paper from Yann LeCun explains his vision on how human-level intelligence can be achieved, although he humbly makes it clear that getting there is a whole other story from just stating the hand wavy principles. Drawing from what we know of child development and brain function, he focuses on the need for "hierarchies of world models", through which a model can continuously match its perception with how well it maps to its existing model of how the world works across a wide range of time and space scales, which he calls energy minimization. In current Reinforcement Learning terms, he's basically advocating for a Model-based paradigm with advanced representation learning-powered largely by self-supervised learning. Honesty, it's hard to tell how useful such a paper is. On one hand, it's laudable that the chief scientist at a large private research lab would openly share his research vision. On the other hand, this paper proposes a cognitive architecture like we've seen hundreds of in the past with some explanatory success in the neuroscience field but not so much on the engineering side of building intelligent machines. Will this path get us closer to human-level AI, or should we be looking somewhere else cause like Plank said "science progresses one funeral at a time"?