Simple adjustment to Improve Baseline Methods The model predicted probabilities p are subject to estimation error. Trained models can have a biased propensity to predict specific classes over others, particularly when the classes in the original dataset are imbalanced. To account for these issues, we adjust the predicted probabilities using class confident thresholds¹⁴, forming new OOD scores based on the MSP/Entropy of the resulting adjusted predictions.


A Simple Adjustment Improves Out-of-Distribution Detection for Any Classifier

Calculating Class Confident Thresholds Letting yᵢ denote the class label for the i-th example in our training data and pᵢₖ denote the probability that this example xᵢ belongs to class k according to our model, we compute a confident threshold vector whose k-th element is defined as: The confidence thresholds are the average probability of a class predicted by our model amongst the examples labeled as that class. This vector thus represents our model's propensity to predict a particular class for examples labeled as that class and has been proven to be a natural threshold for determining the reliability of probabilistic predictions¹⁴.


Adjusting Model Predicted Probabilities for Noise For any new example x, its predicted probability vector p = h(x) is subsequently adjusted by the class confident threshold as follows: Here cࠡ is the largest value in the confident threshold vector (to ensure nonnegative probabilities) and Z is the normalizing constant (to ensure the probabilities sum to one over the classes): It is important to note that while confident thresholds vector c is always calculated using the train-predicted probabilities and labels. Any model output predicted probabilities (i.e., for additional test data) could be adjusted using these thresholds.


Computing adjusted OOD scores Improved OOD scores for x are achieved simply by plugging in the adjusted predicted probabilities p̃ in the place of p into either of the respective MSP/Entropy formulas. This adjusted OOD detection procedure thus remains extremely simple and is easy to implement in practical deployments.


Benchmarking Performance Following standard OOD benchmarking procedures, existing image classification datasets are grouped in pairs where: one dataset is used to train a Swin Transformer⁸ classifier and considered in-distribution training data, while examples from the second dataset are mixed in with the testing data of the first dataset (at a 5050 ratio) as out-of-distribution images. Each OOD scoring method is applied to all images in the test set (without knowledge of their source or their labels) to produce a ranking of these images, which we evaluate using the AUROC for how well these scores detect OOD examples. We consider 2 different OOD detection problems based on popular image classification datasets: CIFAR-10⁵ vs. CIFAR-100⁵ and MNIST⁶ vs. FASHION-MNIST⁷. Our first benchmark relies on the original versions of these datasets, where classes naturally occur in equal proportions. We also run a second benchmark in which we introduce class imbalance in each training set. Here we create new imbalanced training sets for CIFAR-10, MNIST, and FASHION-MNIST, where in each training set: 6 classes each contain 2% of the total examples and 4 classes each contain 22% of the examples. We also create an imbalanced training set for CIFAR-100 in which 90 classes each have 0.63% of the examples, and 10 classes each have 4.25% of the examples. This allows us to evaluate how well our OOD scores perform in settings where the classes occur in unequal proportions in the labeled training data, as is often the case in real-world applications.


Improved Baseline Method Results Tables 1 and 2 list the AUROC performance achieved by both adjusted and original (non-adjusted) OOD scoring methods for each benchmark setting. For many in-distribution / OOD dataset pairs, there is a clear improvement that results from our proposed adjustment. With only a minor adjustment to the predicted probabilities output by a trained classifier, the performance of both Entropy and MSP-based out-of-distribution detection scores is increased. Code to run this OOD method on any dataset has been made available here.


References [1] Yang, J., Zhou, K., Li, Y., and Liu, Z. Generalized out-of-distribution detection: A survey. arXiv:2110.11334. 2021. [2] Ran, X., Xu, M., Mei, L., Xu Q., and Liu Q. Detecting out-of-distribution samples via variational auto-encoder with reliable uncertainty estimation. Neural Networks. 2022. [3] Cao, S., and Zhang, Z. Deep Hybrid Models for Out-of-Distribution Detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. [4] Kirsch, A., Mukhoti, J., van Amersfoort, J., Torr, P. H. S., and Gal, Y. On pitfalls in OOD detection: Entropy considered harmful. ICML Workshop on Uncertainty and Robustness in Deep Learning. 2021. [5] Kuan, J., and Mueller, J. Back to the Basics: Revisiting Out-of-Distribution Detection Baselines. ICML Workshop on Principles of Distribution Shift. 2022 [6] Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations, 2017. [7] Angiulli, F. and Pizzuti, C. Fast outlier detection in high dimensional spaces. In European conference on principles of data mining and knowledge discovery, 2002. [8] Lee, K., Lee, K., Lee, H., and Shin, J. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018. [9] Fort, S., Ren, J., and Lakshminarayanan, B. Exploring the limits of out-of-distribution detection. Advances in Neural Information Processing Systems, 34, 2021. [10] Krizhevsky, A. Learning multiple layers of features from tiny images. 2009. [11] Deng, L. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141142, 2012 [12] Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017. [13] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. [14] Northcutt C, Jiang L, Chuang I. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research. 2021.


This AI newsletter is all you need #16

What happened this week in AI by Louis Diffusions are being used for everything: images, videos, 3D models and soon even audio thanks to StabilityAI, the company behind Stable Diffusion. StabilityAI announced an "AI Music Generator'' called Harmonai based on the "Dance Diffusion" Model, which is basically a stable diffusion model for generating sound instead of images. Diffusion-based models are definitely replacing convolutions and vision transformers even though they are using some of the same principles. We believe they will be here for a bit and are definitely worth learning more about, as they are just getting started and the field is still in the first experimentation phase with this architecture. Our friends at Delta Academy with whom we are partnering, might be building a complete and really innovative course on diffusion models. It would allow you to learn about the models and implement them by competing in game competitions with your peers. If that sounds interesting, stay tuned because we will make the announcements on the Discord channel! If you'd like to learn more about diffusion models, have a look at this article on Towards AI.


This AI newsletter is all you need #16

Hottest News Deepmind tackles algorithm discovery with AlphaTensorAs I mention in the second paper below, Deepmind published "Discovering faster matrix multiplication algorithms with reinforcement learning" in which they tackle the task of discovering new algorithms using deep reinforcement learning to allow for faster and unique discoveries of new algorithms. In their blog post, they mention that it is the first extension of AlphaZero applied to mathematics already unlocking new possibilities for research! Read more in their blog post."Blueprint for an AI Bill of Rights"The White House Office of Science & Technology Policy released a list of suggestions that developers, businesses, users, and lawmakers can follow to reduce AI's potential harms! It has a lot of details covering all the what, why, and how behind the suggestions. Have a look!Activeloop releases Deep Lake, the data lake for deep learningDeep Lake stores complex data, such as images, videos, annotations, embeddings, and tabular data in the form of tensors and rapidly streams the data over the network to Tensor Query Language, in-browser visualization engine, and deep learning frameworks without sacrificing GPU utilization. Deep Lake outperforms all open-source data loaders (Webdataset, PyTorch, Torchdata, & others) on the landscape in a variety of scenarios. Read the announcement post, download the whitepaper, or watch the demo.


This AI newsletter is all you need #16

Most interesting papers of the week Protein structure generation via folding diffusionA new diffusion-based generative model that designs protein backbone structures via a procedure that mirrors the native folding process ("We describe protein backbone structure as a series of consecutive angles capturing the relative orientation of the constituent amino acid residues, and generate new structures by denoising from a random, unfolded state towards a stable folded structure.").Discovering faster matrix multiplication algorithms with reinforcement learning"We report a deep reinforcement learning approach based on AlphaZero for discovering efficient and provably correct algorithms for the multiplication of arbitrary matrices."NOVEL VIEW SYNTHESIS WITH DIFFUSION MODELS3DiM: a diffusion model for 3D novel view synthesis from as few as a single image.


This AI newsletter is all you need #16

Featured Community post from the Discord An E-commerce text classification notebook! This notebook by Sugata Ghosh, one of our members, goes over e-commerce text classification using TF-IDF and Word2Vec. It is thorough, well explained and very professional. A great resource for product classification and playing with text data. Check it out and let Sugata know your thoughts and feedback!


This AI newsletter is all you need #16

AI poll of the week! Join the discussion on Discord.


This AI newsletter is all you need #16

Article of the week Building Feedforward Neural Networks from Scratch by Nicolò Tognoni Deep learning techniques are based on neural networks, commonly referred to as Artificial Neural Networks (ANNs), which are a subset of machine learning. You will learn more about Feed Forward Neural Networks in this article, starting with the fundamentals such as what a perceptron is and working your way up to backpropagation. With the use of interactive pictures, the author skillfully addressed all the crucial points while keeping the article's length in mind.


This AI newsletter is all you need #16

Our must-read articles Adaptive Learning for Time Series Forecasting by Reza Yazdanfar Building an End-to-End Linked Data Engineering Project by Edoardo Bianchi If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.


This AI newsletter is all you need #16

Job offers Research Scientist in ML for Climate Modeling @ The Allen Institute for AI (Hybrid Remote) Senior AI Software Engineer @ Spot AI (Remote) Senior Machine Learning Researcher for Copilot @ Github (Remote) Senior Software Engineer @ Captur (Remote, +/- 2 hours UK time) Machine Learning Apprentice @ HingeHealth (Remote) Senior ML Ops Engineer @ BenchSi (Remote) ML Research Intern @ Genesis Therapeutics (Burlingame, CA) Interested in sharing a job opportunity here? Contact sponsors@towardsai.net or post the opportunity in our #hiring channel on discord! If you are preparing your next machine learning interview, don't hesitate to check out our leading interview preparation website, confetti!


This AI newsletter is all you need #15

What happened this week in AI by Louis Tons of new research is being released because of ICLR 2023's paper submission deadline. Many others are coming in the next couple of days, so stay tuned. One I am highlighting below is DreamBooth, a fantastic approach to adapting text-to-image models to specific subjects. Another is DreamFusion, a text-to-3D approach, one dimension further than text-to-images adding a coordinate and lots of complexities to the problem. This is very similar to Meta's most recent publication Make-A-Video, also released last week, which is a text-to-video approach also in 3 dimensions. Both of these approaches, even though very different, heavily rely on 2D diffusions and adapt them to another dimension, using another spatial dimension for a 3D effect, or a temporal dimension for videos. I already covered Make-A-Video in an article on Towards AI and expect me to cover both DreamBooth and DreamFusion shortly for sure! What you've been waiting for is coming soon. I just sent a private message to the 10 lucky winners of the NVIDIA Deep Learning Institute Gift Cards for our giveaway! The GPU winner will be announced in the next iteration of my personal newsletter in partnership with NVIDIA!


This AI newsletter is all you need #15

Hottest News Tesla AI Day 2022 was last Friday!If you missed it or just couldn't manage to stay there for 3 hours, no worries; an amazing member of our community, tomi.in.ai, wrote a summary for us in the community section below!OpenAI removed the waitlist for DALL·E! Sign up and start creating!OpenAI may have sensed some pressure thanks to other alternatives getting better and better like Stable Diffusion, or maybe they are finally confident enough that the results won't hurt anyone. Either way, they decided to go forward and remove the waitlist for their DALL·E image generation model. You can now use it right away!An AI that generates videos from text! Make-A-Video by Meta AIMeta AI's new model Make-A-Video is out and in a single sentence: it generates videos from text. It's not only able to generate videos, but it's also the new state-of-the-art method, producing higher quality and more coherent videos than ever before! Learn more.


This AI newsletter is all you need #15

Most interesting papers of the week DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven GenerationA new approach for "personalization" of text-to-image diffusion models (specializing them to users' needs) by fine-tuning a pre-trained text-to-image model such that it learns to bind a unique identifier with that specific subject.Improving alignment of dialogue agents via targeted human judgementsAn information-seeking dialogue agent trained with reinforcement learning (from human feedback) to be more helpful, correct, and harmless compared to prompted language model baselines.EMB-GAM: AN INTERPRETABLE AND EFFICIENT PREDICTOR USING PRE-TRAINED LANGUAGE MODELSThey use a pre-trained language model to extract embeddings for each input before learning a linear model in the embedding space (a generalized additive model (GAM), a transparent and interpretable linear function of its input features and feature interactions). Enjoy these papers and news summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #15

Meme of the week! Happened to all of us . Meme shared by friedliver#0614.


This AI newsletter is all you need #15

Featured Community post from the Discord As most of you already know, last Friday was Tesla AI Day 2022, the second iteration of this new annual event by Tesla featuring the advancements and projects Tesla worked on and is working on as well as Q&A, all hosted by Elon Musk and his team. If you couldn't attend, you can watch the rediffusion, but it's 3.5 hours long. Fortunately, a fantastic member of our community, tomi.in.ai, wrote a summary for us: Tesla Humanoid Robot - Optimus Optimus would cost less than $20,000 and the prototype was developed "technically" within a year. It has 11 dof (degrees of freedom), and hands that can carry a 20lb (9 kg) bag, and can easily be overpowered by a human. Its vision is based on similar techniques used in Tesla autopilot and has a damage control system (should it fall face down) similar to Tesla vehicles collision structure. Full Self Driving (FSD)/ Autopilot FSD's customer base increased from 2,000 to 160,000 between 2021 and 2022. The model training powerhouse consists of 3 supercomputers with 14,000 GPUs each (4,000 GPUs for auto labeling and 10,000 GPUs for training). Tesla has trained 75,777 models and shipped 281. It uses a PyTorch extension for accelerated video library, resulting in a 30% increase in training speed. Automated 3D labeling powered by multi-trip reconstruction replaces 5 million hours of manual labeling with 12 hours on clusters for 10,000 trips. A data engine identifies mispredictions (e.g. a parked car it thinks is about to cross an intersection), corrects the label and categorizes clips into evaluation sets. 13,900 clips have been re-labeled so far, keeping the data dynamic. The dojo system processor is pretty advanced, unfortunately I can't summarize as I understand little about this. You may have to watch this section of the presentation for context. Some of Musk's remarks during Q&A: "Optimus is going to be incredible in 5 years, 10 years will be mind-blowing... and I'm really interested to see that happen, I hope you are too." "We're gonna get a lot of sh*t done, it's gonna be really cool, and it's not gonna be easy. But if you are a super talented engineer, your talents will be used to a greater degree than anywhere else."


This AI newsletter is all you need #15

AI poll of the week! What is your preferred way of learning? Join the discussion on Discord.


This AI newsletter is all you need #15

Article of the week LogBERT Explained In Depth: Part I by David Schiff In this article, the author explains and simplifies the LogBERT method for detecting anomalies in log sequences. The major part of BERT, the transformer block, is dissected in this article. The embedding layer comes first, followed by position encoding, self-attention, and, finally, multi-head attention. The LogBERT is relatively simple to understand thanks to the graphical representations, mathematical equations, and suitable examples.


This AI newsletter is all you need #15

Our must-read articles A Gentle Intro to AWS ML Related Services - Sentiment Analysis With AWS by Kaan Boke Ph.D. A Chatbot With the Least Number of Lines Of Code by Chinmay Bhalerao If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.


This AI newsletter is all you need #15

Ethical Take by Lauren on Meta's contribution to Molly Russell's death I would be remiss to not address the biggest news in tech ethics of late: the recent British ruling of Meta's responsibility in the death of 14-year-old Molly Russell in 2017. The ruling is solely related to the cause of her death, pursued by Molly's family to raise awareness over the devastating shortcomings of child Internet safety. The coroner presiding over the case concluded that Meta and other social media companies contributed to her death, which is possibly the first time legal blame for a death has been put on social media companies. This case sets an important legal precedent for other families whose children died similarly and for future child protection legislation. It's a very difficult and sensitive area that we are forced to explore as a society - what can be shown and what shouldn't be, how to let kids be themselves and keep them safe. Answering these questions requires a great deal of care and honesty for the world we're leaving for the next generation. It demands not shying away from the horrors that youths are exposed to constantly on platforms without sufficient protection, horrors that reduced a courtroom of adults to tears and nightmares. Moving fast and breaking things has consequences, and this tragedy is one of many. As a former teenage girl on Instagram, I am grateful for everyone advocating for increased youth rights and protections online and look forward to the advancements to come.


This AI newsletter is all you need #15


EU Accelerates AI Regulation

If you have found it interesting: You can look for my other articles, you can also subscribe to get notified when I publish articles, and you can also connect or reach me on LinkedIn. Thanks for your support! Here is the link to my GitHub repository, where I am planning to collect code and many resources related to machine learning, artificial intelligence, and more. Or feel free to check out some of my other articles on Medium:


Adaptive Learning for Time Series Forecasting

PROBLEM) Time series features should be changed over time, so we call this type of data (as a function of time) non-stationary. We cannot work with stationary data; I mean, what's the point if data is not changing over time (really simple!) ??!! So, this fact of being non-stationary is equivalent to having various distributions over time; this is one of the points that make time series capture. For a better understanding, you can look at the figure below: The dataset in Figure 1 is a multivariate time series dataset divided into three periods A, B, C, and Test. We can easily see those distributions are not equal.


SOLUTION) AdaRNN


INTERPRETATION) We can consider RNNs as one of the most popular algorithm architectures, so we can start by using the most popular of its type. In order to prevent this article longer, I'm not going to describe RNNs in detail; however, you can give Stanford deep learning sheet about RNNs. Well, well, well; I told you that there are two steps for AdaRNN: Temporal Distribution Characterization (TDC) algorithmTemporal Distribution Matching (TDM) algorithm In summary: TDC separates the training dataset into K most diverse intervals. (with the biggest distribution gap) And then: TDM decreases the divergence distribution utilizing RNN.


Distribution Usually, there are two main approaches for distribution: Domain Generalization (DG) algorithmsDomain Adaptation (DA) algorithms In summary: DG expresses how well the model is performing on unseen data and is dealing with several various domains which are connected, while DA tries to improve the model on output data which includes a lot of anomalies or unexpected data behaviors. These two are different in the point that one accesses the test dataset and the other doesn't. However, this is not the case in this article because we solve this with our method. :)


AdaRNN in general You can easily see two color boxes expressing two main steps for this model, TDC and TDM, first specifying distribution by cracking it into courses and later adapting distributions to feed the main model (M) for forecasting. Finally, the adaptive RNN model is forecasting. One point!! It is reasonable to learn/practice/exercise and prepare yourself for worst cases; so, it is true for AdaRNN, the researchers trained model with difficult tasks (I mean distribution).


TDC (Temporal Distribution Characterization) in point The fundamental of TDC is the principle of maximum entropy; it tries to make the usage of shared knowledge in time series maximum, how?? by training the model on the wide range of distribution; in other words, with the maximum number of non-similar distribution. The logic is simple if the model can handle this, it will be able to manage future/unseen data. Have a look at Figure 3: From a mathematics perspective, we have Equation 1: We use hyper-prams to prevent over-splitting and predefined prams to prevent trivial solutions. The aim of Eq 1 is maximizing distance d (which can be any type of distance function from Euclidean to etc.) If we choose a good distance metric, we can accomplish TDC's destination by using dynamic programming (DP). In this research, the researchers divided the dataset into N=10 proportions, which we can not divide into smaller proportions. A significant large or small value of K can result in no desirable model performance.


TDM (Temporal Distribution Matching) in point We use TDM to study the shared common knowledge by using various distribution periods. Its loss function can be formulated as below: In this research, the regularization for the distribution is implemented on the final outputs of the RNN model. Equation 3 is the period-wise distribution matching on the final hidden states for a pare (Di, Dj): However, this regularization term is lost on time-dependent for hidden states. Why?? Because each hidden state individually includes a proportion of the whole information about distribution. Well, we have to contemplate each hidden state during building a distribution matching regularizer. So what does it mean?? First, have a look at the TDM architecture: The point of TDM is forecasting time series data and matching distributions among RNN cells simultaneously. There is a parameter called the importance vector, which equips the model to decrease distribution divergence. Equation 4 tells the loss of time-series distribution matching: By using Equations 4 and 2, we can derive Eq 5 (The terminal objective function for our model):


Dataset Four-time series datasets (human activity recognition, air quality prediction, household power consumption, and stock price prediction) from various industries with high and low dimensions are used for the model evaluation (for both classification and regression). You can see a summary of the size of the datasets in Table 1.


Results The AdaRNN is compared with other deep/machine learning models, including: Traditional ones (ARIMA, Prophet, GRU, etc.)latest ones (STRIPE, etc.)Transformers You can see the results of ADaRNN with other models in Table 2: Five metrics have been used for the evaluation, including ACC (accuracy), P (precision), recall (R), F1, and AUC (area under the curve). As you can see, AdaRNN shows better results in comparison with its peers. In this research, this method is applied to transformers (AdaTransformer) in each self-attention block. The results of AdaTransformer showed better results in comparison with its original version (Vanilla Transformer) as you can see in Table 3:


This AI newsletter is all you need #14

What happened this week in AI by Louis This week we saw some interesting AI news once again involving stable diffusion-like models: Getty Images banning AI-generated content. If you are selling AI-generated content, be careful! Indeed, Getty Images has banned the upload and sale of illustrations generated using AI art tools like DALL-E, Midjourney, and Stable Diffusion fearing copyright issues. "There are real concerns concerning the copyright of outputs from these models." And I do agree with that. There are real concerns in figuring out who really is the creator of such a piece and who can claim authorship and sales, not even talking about copyright issues. This is similar to all new fields and applications like the start of the Internet: we need to figure out the legal answers to these questions, especially (but rarely possible) before such problems appear, even though it seems much more like a gray area than clearly "outline-able" rules. What do you think about copyright issues with these deep nets? Can we sell the art it makes? We've already seen controversy from AI arts winning competitions, and now people that make money from it. The million-dollar question is, who should profit from it? Should it be the algorithm's creator, the creator(s) of the millions of images used in training, the company hosting and developing the model (e.g., OpenAI), or the model's user and prompt engineer that produced the results?


This AI newsletter is all you need #14

Hottest News A Demo for Stable Diffusion fine-tuned on Pokemon!A very cool demo for Stable Diffusion fine-tuned on Pokemon to generate new Pokemon (very funny ones, see the cover image!). Try it here or with the Google colab here. Made by Justin Pinkney. Justin even published a guide covering how to fine-tune stable diffusion for your own use.Stable Diffusion implemented using Tensorflow and Keras!Comment by the creator of Keras, François Chollet: "It's faster, it's more concise, and the code is elegant. You can add TPU and multi-GPU inference in 1 line. You can export the model to TF.js or TFLite for on-device inference." A quick summary of the Tensorflow implementation: Converted pre-trained models, easy to understand code, and minimal code footprint. Code, ColabECCV is in one month... and I will be there!If you are hesitating about going to ECCV, just go for it! Take this opportunity to learn and expand your professional/research network (and meet with me!). Let me know if you are going. I would love to chat with you! This year, ECCV is in Tel Aviv. I'll be there for the event and staying a few more days, so please reach out if you are in or near Tel Aviv too!


This AI newsletter is all you need #14

Most interesting papers of the week VToonify: Controllable High-Resolution Portrait Video Style TransferA controllable high-resolution portrait video style transfer framework.I2DFormer: Learning Image to Document Attention for Zero-Shot Image ClassificationI2DFormer is a novel transformer-based zero-shot learning (ZSL) framework that jointly learns to encode images and documents (from Wikipedia) by aligning both modalities in a shared embedding space.[NVIDIA] NeuralMarker: A Framework for Learning General Marker Correspondence"We propose a novel framework NeuralMarker, training a neural network estimating dense marker correspondences under various challenging conditions, such as marker deformation, harsh lighting, etc." Enjoy these papers and news summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #14

Meme of the week! Sure, it will be fixed... Meme shared by Brooke#5801.


This AI newsletter is all you need #14

Featured Community post from the Discord A really cool initiative by one of our members arsenaultk9 #2059: "If you're interested in learning how to use machine learning to generate music, I've published this article, including code to generate midi: https://medium.com/@arsenaultk9/artificial-intelligence-music-generation-melody-harmonization-pt1-cb3e92c9a0ca" arsenaultk9, 09/20/2022


This AI newsletter is all you need #14

Article of the week Stop Using Grid Search! The Complete Practical Tutorial on Keras Tuner by Poulinakis Kon Even experts are often trapped in rabbit holes of trial and error procedures until they find a good hyperparameter combination for their Neural Networks. Keras-Tuner is a tool that will help you optimize your neural network and find a close to optimal hyperparameter set. In this article, you will not only learn how to use KerasTuner but also some unique tricks, such as tuning parameters in each layer separately or tuning the learning rate in conjunction with the optimizer.


This AI newsletter is all you need #14

Our must-read articles A Gentle Introduction to Automatic Differentiation by Pablo Monteagudo This Is How You Can Build a Churn Prediction Model Using Spark by Paul Iusztin If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.


This AI newsletter is all you need #14

Ethical Take on AI Art's Economic Responsibility Gap by Lauren I will take a stab at Louis' million dollar question! How do ownership and profit rights apply to AI generated art? We can take a page from another area of AI thought: the responsibility gap in Robert Sparrow's 2007 paper titled Killer Robots. The issue is not with rights to profit, but with jus in bellum, a condition for just war in which someone can be held responsible for a death; a condition that autonomous weapons systems can't satisfy with their nonpoint development process. Our systems only allow for one source and AI is a product of multiple sources, so which source is the true source? Even many years later, we still struggle with this gray area in its various forms. Coming back to AI-generated art, we're in need of legislation that tells us the best way to move forth into this new unknown, offering an accountability framework for proceeding ethically. But how do we form this structure? For this problem, it's essential to look at the visual art community and economy as a whole. What issues exist that could be exacerbated by the presence and profits of AI art? Who has unfair advantages or disadvantages within the space? These questions will reveal where the weak points are and inform what areas need the most support by future legislation. For the time being, proceeding with caution and developing a sustainable long-term multilayered approach is the best way forward.


This AI newsletter is all you need #14

Job offers Senior Software Engineer @ Captur (Remote, +/- 2 hours UK time) Machine Learning Apprentice @ HingeHealth (Remote) Senior ML Ops Engineers @ BenchSi (Remote) ML/Algorithms Engineer @ Aurora Insight (Hybrid Remote) Data Scientist @ Electra (Remote) ML Research Intern @ Genesis Therapeutics (Burlingame, CA) Senior Staff Data Scientist @ One Concern (Remote) Interested in sharing a job opportunity here? Contact sponsors@towardsai.net or post the opportunity in our #hiring channel on discord! If you are preparing your next machine learning interview, don't hesitate to check out our leading interview preparation website, confetti!


From UNet to BERT: Extraction of Important Information from Scientific Papers

Objective Despite an increase in the number of papers published each year, little work is done on the application of Machine Learning to scientific papers. Machine learning models are becoming smarter as well. Even though scientific papers are hard to understand even by us humans, they contain unique structures, formatting, and languages which set them apart from other documents. In this project, I will demonstrate how we can extract and summarize important information from such documents following a multidisciplinary approach of Natural Language Processing and Computer Vision. Please note that this is a continuation of my previous blog on the Extraction of important information from scientific papers.


Recap I have covered the computer vision part in my previous blog here. The following is a summary of what was covered. The computer vision part contains a UNet-OCR pipeline that does the following: Extraction of important sections learned by UNet andConversion of learned sections into text using the OCR The UNet generates a black and white mask that highlights the Title, Author/Authors, and Abstracts sections of a given paper. A post-processing step follows where the masked image is reconstructed into an RGB image. This image is then passed to an Optical character recognition(OCR) engine to be converted into text. I have used Tesseract with default parameters for the OCR.


Text Summarization with BERT Text summarization is a machine learning technique that aims at generating a concise and precise summary of a text without overall loss of meaning. Text summarization is a popular and much-researched domain of Natural language Processing. There are two approaches to text summarization which are: Extractive text summarizationAbstractive text summarization In Extractive text summarization, the summary is generated using excerpts from the texts. No new text is generated; only existing text is used in the summarization process. This can be done by scoring each sentence and generating the k most important sentences from the text. Abstractive summarization is how a human would do a summary of content i.e explain in one's own words. The summary would include words, phrases, and sentences not in the text. I followed an Extractive text summarization approach for this project and used the open source Bert-extractive summarizer[1] repository. BERT[2] stands for Bidirectional Encoder Representations from Transformers[3] and is a model that has learned sentence representation from training on a very large corpus. Only the encoder part is taken from the Transformers' encoder-decoder architecture for the BERT model. In short, the encoder part accepts the word embedding of an input text. Positional encoding transforms the word embedding to add meaning to each word with respect to its positional context. The attention block (Multi-head Attention) computes the attention vector for each word. These vectors are then fed to the feed-forward Neural Network one vector at a time. This result consists of a set of encoded vectors for each word which is the final output of the BERT model. The Bert-extractive-summarizer repository is based on the paper Leveraging BERT for Extractive Text Summarization on Lectures [4]. The paper explains using BERT to generate sentence representation and then using the K-mean algorithm to cluster these representations around k concepts. The k sentences closest to their respective centroids are then returned as a representative summary for their cluster.


Getting started Start by cloning my repository available at Dagshub[5]. Install DVC. Install dagshub. Install Tesseract. Install Bert. Configure your DVC origin. If you feel confused about configuring your DVC, refer to this documentation. Next, pull my tracked dataset into your system using this command. Run tesseract on your image inside a python shell. Pass this result to the Summarizer model. The above code block calls the model with your text (body) and a number of sentences = 3. This means the model will summarize your text in 3 sentences. You can increase or decrease the number of sentences depending on your use case. If you are on Google Colab Notebook refer to this notebook.


What did I use? Colab notebook free versionDagshub as my online repositoryDVC to track my datasetBert-extractive-summarizer


Project Pipeline The Figure below shows the output on 1409.1556_0.jpg found here.


Conclusion You can train the UNet to extract any set of sections from scientific papers but for this experiment, I chose the Abstract, Author/s, and Title sections. BERT gives us the final result which is a summary of sections found in Abstract, Author/s, and Titles. While most papers have more or less similar formatting, structure, and sections, the challenge for this project was to extract a given section where that section might/might not be there for a given paper. With that, we have reached the final milestone. Congratulations on making it this far! Feel free to reach out for any questions or feedback. Would be happy to hear from you!


References [1] https://github.com/dmmiller612/bert-extractive-summarizer [2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 41714186, Minneapolis, Minnesota. Association for Computational Linguistics. [3] Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. "Attention is all you need." Advances in neural information processing systems 30 (2017). [4] Miller, Derek. "Leveraging BERT for extractive text summarization on lectures." arXiv preprint arXiv:1906.04165 (2019). [5] https://dagshub.com/docs/ https://www.youtube.com/watch?v=TQQlZhbC5ps


This AI newsletter is all you need #13

What happened this week in AI This week, my attention was on Emad and the amazing work he and his team (Stability.ai) have been involved with. This week, Emad announced OpenCLIP, an open-sourced version of CLIP that beats state-of-the-art CLIP results, improving text-image encodings pair, which has a dramatic impact on current research as such a pre-trained open-sourced model will be used by thousands of researchers to create amazing new applications and models involving images and text. What is OpenCLIP exactly? OpenCLIP is an open-source implementation of OpenAI's CLIP (Contrastive Language-Image Pre-training). And what do they bring exactly? Why is this so cool? It's cool because, as stable diffusion, their goal is to bring CLIP to you, the regular researcher (in terms of computing resources, not in terms of skills, you are definitely better than a regular researcher if you are reading this! ). This means that they want to enable training/facilitate the study of models with contrastive image-text supervision. As they mention, "our starting point is an implementation of CLIP that matches the accuracy of the original CLIP models when trained on the same dataset." This means they bring many new SOTA pre-trained models you can implement thanks to more efficient implementations and open-source code (see below). Check out their repository to learn more and implement their models!


This AI newsletter is all you need #13

Hottest News Diffusion Bee, a Stable Diffusion GUI App for M1 Macs!"Diffusion Bee is the easiest way to run Stable Diffusion locally on your M1 Mac. Comes with a one-click installer. No dependencies or technical knowledge needed."MIT researchers have made DALL-E more creative!As the article highlights, the researchers developed a new method that uses multiple models to create more complex images with better understanding. The method, called Composable-Diffusion, achieves better results by generating an image using a set of diffusion models, with each responsible for modeling a specific component of it.I was nominated in the 2022 Noonies by Hackernoon!I would love it if you could support me there and vote for my work if you like it! (both for tech YouTube and data science newsletter). Thank you in advance for this amazing community! YT: https://www.noonies.tech/2022/emerging-tech/2022-top-tech-youtuberNewsletter: https://www.noonies.tech/2022/emerging-tech/2022-best-data-science-newsletter This issue is brought to you thanks to Delta Academy:


This AI newsletter is all you need #13

Most interesting papers of the week im2nerf: Image to Neural Radiance Field in the WildA learning framework that predicts a continuous neural object representation given a single input image in the wild, supervised by only segmentation output from off-the-shelf recognition methods.Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language ModelsTest-Time Prompt Tuning (TPT): A method that can learn adaptive prompts on the fly with a single test sample.StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation"We first propose the task of story continuation, where the generated visual story is conditioned on a source image, allowing for better generalization to narratives with new characters. [...] Our work demonstrates that pretrained text-to-image synthesis models can be adapted for complex and low-resource tasks like story continuation." Enjoy these papers and news summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #13

Meme of the week! Oh no... EDA stands for Exploratory Data Analysis and is the process of investigating the data you will use in an ML model before thinking about which model to apply. Meme shared by friedliver#0614.


This AI newsletter is all you need #13

Featured Community post from the Discord Such a cool project by deep2universe#6939 from the Learn AI community. The algorithm will run on any YouTube videos and perform pose and motion detection on the people in the video. Quite cool! As he mentions, the author, unfortunately, has to stop the development of the extension, but open-sourced the code for further use. We strongly encourage anyone looking to tackle an interesting project to reach out to deep2universe on discord to take on his work. It's a fantastic project for people interested in computer vision, especially motion or pose detection. Watch more in the demo video: Download the Chrome extension. Check out the code:


This AI newsletter is all you need #13

AI poll of the week! What do you think about working on weekends, weekdays, or both? Join the discussion on Discord.


This AI newsletter is all you need #13

Article of the week Difference Between Normalization and Standardization by Chetan Ambi.One of the crucial processes in the machine learning pipeline is feature scaling. Normalization and standardization are the two commonly utilized methods for feature scaling. But what is the difference between them and when to use them? This is a very common question among people who have just started their data science journey. The author beautifully explains the difference using formulas, visualizations, and code.


This AI newsletter is all you need #13

Our must-read articles Improve Your Classification Models With Threshold Tuning by Edoardo Bianchi Imbalanced Data - Real-Time Bidding by Snehal Nair If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.


This AI newsletter is all you need #13

Lauren's Ethical Take on increasing DALLE's creativity I think it's so incredible that this model is able to adhere to natural language prompts, thanks to MIT researchers! However, I don't quite agree with the general claim that this ability increases the model's creativity. This enhancement appears more to be focused on accuracy, which creativity as a concept does not always correlate with. Art often requires breaking the rules or working with accidents, and this "error" is part of the creative process and often makes art as amazing as it is. Increasing the model's ability to color in the lines doesn't follow this principle. Similarly, increased complexity does not necessarily correlate with increased creativity either - incredibly simple things can be incredibly creative. A counterargument to my first stance might be that exercising abilities within the natural language constraints IS actually more of a creative exercise than being allowed to go off script. However, I will reassert that breaking parameters can be more creative than working within them. Part of what made the initial wave of DALLE creations so impactful was this lack of regard for preconceived notions based on language (other than what it was trained on, of course!). We may lose some of that creative impression by optimizing for accuracy in this way. Whichever direction we choose will continue to shape how we attempt to cultivate creativity and the future of AI digital art tools!


This AI newsletter is all you need #13

Job offers Senior Software Engineer @ Captur (Remote, +/- 2 hours UK time)Senior ML Engineer @ Safe Security (Remote)ML/Algorithms Engineer @ Aurora Insight (Hybrid Remote)Data Scientist @ Electra (Remote)ML Research Intern @ Genesis Therapeutics (Burlingame, CA)Senior Staff Data Scientist @ One Concern (Remote)Research Scientist - Speech recognition @ Abridge (Remote)Computer Vision Scientist @ Percipient AI (Santa Clara, CA)Senior Data SciJob Application for Senior Data Scientist at EvolutionIQentist @ EvolutionIQ (Remote) Interested in sharing a job opportunity here? Contact sponsors@towardsai.net or post the opportunity in our #hiring channel on discord!


Machine Learning & Artificial Intelligence to Reduce Financial Crimes

Our Current Problem statement Looking at the above facts and changing economies, Insurance companies have realized subrogation can be a carrier's leading revenue stream during periods of difficult financial markets. Post-Covid, the economic slowdown in the auto insurance sector and particularly on uninsured claimants, can be observed. In the USA itself, 32+ million uninsured vehicles have a 35x spike in subrogation workload leading to 15% missed opportunities of subrogation (deteriorating) & 32% of recoverable paid claims are lost. If anyone of us gets into a situation when we just know the accident wasn't our fault, but we couldn't really do anything, even though it was the opposite party's fault? This is exactly where we could benefit from something we call Subrogation in Insurance. The opportunity to pursue subrogation occurs in less than 10% of claims. Subrogation remains a manual process for most insurers, resulting in missed clues and costing some claims departments up to two percentage points of loss ratio.


Why do we need Subrogation? Automated subrogation settlements around the world are drastically increasing, with countries such as the UK and Australia having automated settlement rates as high as 40%. However, the US automated settlement rate is between 10 and 15%, low compared to its global counterparts. An Intelligent system driven by AI & ML will raise alerts to adjusters for those claims that are most likely to have a subrogation opportunity and provide an estimate of the recoverable amount with integration in the claims processing workflow. This new information would prevent adjusters from closing claims flagged by predictive models and refer them to subrogation professionals. Finding and recovering lost subrogation opportunities may appear to be negligible when compared to all other losses and expenses - yet this can result in a significant increase in underwriting profit. Some insurance companies add the deductible amount, too, in the case of a subrogation. So, in case of such a situation where the damage is done by the third party, people get their claim amount plus the deductible once the third party pays the compensation to the insurance company. The solution is based on Machine learning and will lead to: - · Elimination of leakage associated with subrogation identification. · Reduce false positives up to 36%. · Enable productivity gains up to 20%. · Improve cycle time by over 30%.


The overall Benefits to Customers & Insurer will be: - · Reduced fraud · Improved customer retention · Improves customer trust and experience · Increased customer engagement · Improves employee productivity · Reduced costs and response times


A real-life example, a core aspect of this use case To understand things easily, connect them with a story or an example. Suppose one fine day, your car happens to hurt after a reckless driver bumps into your car. Now, the back of your car is damaged, and this guy doesn't even seem to accept his fault. You have no time to fight this out or argue further, so you move on with your damaged car and instead get your comprehensive car insurance to pay for the damages. Your insurance company here will hold the third-party responsible for the damage and reimburse both the amount of money you spent from your pocket plus the amount of money your insurance company paid for due to the damages and losses caused.


On the above aspect, how can analytics contribute to the value chain ..? Our initial approach involves a standard model of insurance for accident risk and is supported by the ability of insured individuals to bring suit for harm that they suffer. We assume that identical risk-averse individuals face a chance of suffering harm from an accident; that they purchase insurance policies that maximize their expected utility, and that if an accident occurs, they will be able to sue a potentially liable party for compensation, where the suit would involve a litigation cost and result in success only with a probability.


Heuristic Model y = initial wealth of an individual U(∙) = utility of an individual from wealth, where U′ > 0 and U″ < 0 p = probability of an accident; p 0 (0, 1) h = harm if an accident occurs; h > 0 k = cost of bringing a suit; k ≥ 0 q = probability of winning a suit, resulting in an award of h; q 0 [0, 1]. π = insurance premium c = insurance coverage c ≥ 0 N = instruction whether the insurer sues under a subrogation provision; if N = 0, suit is not brought; if N = 1, suit is brought s = insured's share of the award h under a subrogation provision if a suit is successful; s 0 [0, h] Under a pure insurance policy, the premium constraint is π = pc If the insured would bring a suit in that case the expected utility would be EU = (1 - p)U(y - π) + p[q U(y - π + c - k) + (1 - q)U(y - π - h + c - k)]


Analytical Approach Our next approach to Machine learning will make use of text data and Algorithms. This will help to lower fraud, detect semantic relation patterns, and automatic categorization. It enables us to prioritize claims by scoring claims files as High, Medium, or Low based on recoverability likelihood. The scored cases will be sent to appropriate investigation and recovery experts. All this ensures that subrogation actions will be better informed and help assist insurers to eliminate instances of false or excessive pay-outs in the event of major claims. The goal is to learn pattern characteristics for claims with high subrogation potential through the analysis of textual notes of historical claims with known outcomes. To perform a joint analysis of patterns extracted from textual notes and structured attributes associated with the claim, a model is developed for predicting the subrogation potential of each claim. The created model enables the automated ranking of claims with respect to their subrogation potential. The ability to focus on claims with the highest subrogation potential enhances the efficiency of the work of human analysts and increases the recovery rate. An automated subrogation prediction system ensures the consistency of the analysis across all claims and across time. It helps minimize the number of missed subrogation opportunities. A good subrogation prediction model can discover a significant number of good subrogation candidate claims, which may be missed in the manual analysis.


Superior Data Integration Pulling out data from different sources and transforming it to identify claims with the highest potential for subrogation recovery. Variables we can create or use for the model: Displaying the columns and data from the sample dataset. The data is at the claim id level from 2019 to 2022 with 48 features. (This is just a mock-up dataset created from open sources) Let's have a look at a sample of our unstructured data We will augment our claims text data to uncover important aspects of the uninsured or underinsured claimant and thus use variables to build a machine learning model to understand the likelihood of paying the subrogation. Looking at the data, we can easily identify some of the keywords directed toward a potential accident. We will move ahead with only unstructured data at the moment


AI/ML Model Build and Explainability (XAI) We are now ready to begin modeling. The approach we like to take is to build a simple model as a baseline right away, then use that Model to compare more complicated methods. If we can't achieve significant improvement over the baseline, it isn't fruitful to use more complex models. Here, we will start with the Logistic Regression model & some hyperparameters to see the performance. From that result, we'll move to BERT and check if we have a superior model to play around with. Our models work by collecting a plethora of different data inputs ranging. These inputs are then run through the models to detect anomalies within the claim, and also, we can apply the ran business rule to detect if high subrogation instances occurred automatically.


a. Logistic Regression The Algorithm commonly deals with the issue of how likely an observation is to belong to each group. This model is commonly used to predict the likelihood of an event occurring with the logit function. This is a useful model to take advantage of for this problem because we are interested in predicting whether a claim is for subrogation (1) or not (0). Logistic regression predicts the outcome of the response variable (turnover) through a set of other explanatory variables, also called predictors. b. BERT (Bidirectional Encoder Representations) This deep learning model has given state-of-the-art results on a wide variety of natural language processing tasks. It stands for Bidirectional Encoder Representations for Transformers. It has been pre-trained on Wikipedia and BooksCorpus and requires (only) task-specific fine-tuning. It is basically a bunch of Transformer encoders stacked together (not the whole Transformer architecture but just the encoder). The concept of bidirectionality is the key differentiator between BERT and its predecessor. Each line of the dataset is composed of the Claims report text and its label. Data preprocessing consists of transforming text to BERT input features: input_word_ids, input_mask, segment_ids/input_type_ids · Input Word Ids: Output of our tokenizer, converting each sentence into a set of token ids. · Input Masks: Since we are padding all the sequences, it is important that we create some sort of mask to make sure those paddings do not interfere with the actual text tokens. Therefore we need a generate input mask blocking the paddings. The mask has 1 for real tokens and 0 for padding tokens. Only real tokens are attended to. · Segment Ids: For our task of text classification, since there is only one sequence, the segment_ids is essentially just a vector of 0s.


Explainability Of BERT If we assume that our model can understand the context of the text very well with respect to what we are trying to predict, it will be wonderful. So, when we use models like BERT to understand the context of our text columns and investigate what aspect of language BERT is learning. It helps in verifying the robustness of our model. We will know what words are deemed important by our model in our text with models like BERT. Let's look at a simple method of attribution approach, It scores the input data based on the predictions our model makes, i.e., it attributes the predictions to its input signals or features, using scores for each feature. Integrated Gradients use the approach of feature * gradient. The gradient is the signal that tells the neural network how much to increase or decrease a certain weight/coefficient in the network during backpropagation. It relies heavily on the input features to do so. Therefore, the gradient associated with each input feature with respect to the output can help us get a clue about how important a feature is. Integrated Gradients allow us to attribute the selected output feature from the BERT model to its inputs. We can generate word importance for each of the output features as below:-


How do we move forward from here .? We would like to integrate various data sources and research possible ways to improve the model performance and get results closer to real-life scenarios. By using Google Street View (based on Collison date time & location) & Digital Identity Mapping for the Claimant. The model performance will be compared w.r.t holdout data and gauged the performance against one another to determine the most trustworthy predictors of claim subrogation. At an enterprise level, we need to work with leaders & Experts from the claims department to introduce our model output into the claims process. It is usually for insurers to assign alerts to adjusters and to the subrogation specialists when the probability of subrogation exceeds a predetermined limit. Claims with slightly lower probabilities (520%) will usually be assigned a "watch" status, while claims with even lower probabilities (most claims) will get no message because a message on these claims is often seen as a distraction. An API will be created to score claims, claims center will use the HTTPClient to call the REST Service. Also, configuration needs to be done for triggering events that initiate these calls. Please feel free to share your opinions and thoughts. Happy learning!


Stable Diffusion Based Image Compression

Intro Stable Diffusion is currently inspiring the open source machine learning community as well as upsetting artists world wide. I was curious to see what else this impactful technology release could be used for other than making professional artists and designers ponder their job security. While experimenting with the model, I found that it makes for an extremely powerful lossy image compression codec. Before describing my method and sharing some code, here are a few results compared to JPG and WebP at high compression factors, all at a resolution of 512x512 pixels:


Stable Diffusion's latent space Stable Diffusion uses three trained artificial neural networks in tandem: a Variational Auto Encodera U-Neta Text Encoder The Variational Auto Encoder (VAE) encodes and decodes images from image space into some latent space representation. The latent space representation is a lower resolution (64 x 64), higher precision (4x32 bit) representation of any source image (512 x 512 at 3x8 or 4x8 bit). How the VAE encodes the image into this latent space, it learns by itself during the training process and thus the latent space representations of different versions of the model will likely look different as the model gets trained further, but the representation of Stable Diffusion v1.4 looks like this (when remapped and interpreted as 4-channel color image): The main features of the image are still visible when re-scaling and interpreting the latents as color values (with alpha channel), but the VAE also encodes the higher resolution features into these pixel values. One encode/decode roundtrip through this VAE looks like this:


Compression Method To use Stable Diffusion as an image compression codec, I investigated how the latent representation generated by the VAE could be efficiently compressed. Downsampling the latents or applying existing lossy image compression methods to the latents proved to massively degrade the reconstructed images in my experiments. However, I found that the VAE's decoding seems to be very robust to quantization of the latents. Quantizing the latents from floating point to 8-bit unsigned integers by scaling, clamping and then remapping them results in only very little visible reconstruction error:


Evaluation To evaluate this experimental compression codec, I didn't use any of the standard test images or images found online in order to ensure that I'm not testing it on any data that might have been used in the training set of the Stable Diffusion model (because such images might get an unfair compression advantage, since part of their data might already be encoded in the trained model). Also, to make the comparison as fair as possible, I used the highest encoder quality settings for the JPG and WebP compressors of Python's Image library and I additionally applied lossless compression of the compressed JPG data using the mozjpeg library. I then used the compression strength 1 less than what would result in a data size smaller than the Stable Diffusion result, or maximum strength otherwise (many images don't get smaller than the SD result even at maximum lossy JPG or WebP compression). It's important to note that while the Stable Diffusion results look subjectively a lot better than the JPG and WebP compressed images, they are not significantly better (but neither worse) in terms of standard measurement metrics like PSNR or SSIM. It's just that the kind of artifacts introduced are a lot less notable, since they affect image content more than image quality - which however is also a bit of a danger of this method: One must not be fooled by the quality of the reconstructed features - the content may be affected by compression artifacts, even if it looks very clear. for example, looking at a detail in the San Francisco test image:


Limitations Some features are currently not well preserved by the Stable Diffusion VAE, in particular small text and faces seem to be problematic (which is consistent with the Limitations and Biases of the V1.4 SD model listed on its model card). Here are three examples that illustrate this:


How does this approach differ from ML based image restoration? Deep learning is very successfully used to restore degraded images and videos or to up-scale them to higher resolution. How does this approach differ from restoring images that have been degraded by image compression? It's important to understand that Stable Diffusion is not used here to restore an image that has been degraded by compression in image space, but instead it applies a lossy compression to Stable Diffusion's internal understanding of that image and then tries to 'repair' the damage the lossy compression caused to the internal representation (which is not the same as repairing a degraded image). This for example preserves camera grain (qualitatively) as well as any other qualitative degradation, whereas an AI restoration of a heavily compressed jpg would not be able to restore that camera grain because any information about it has been lost from the data. To give an analogy for this difference: Say we have a highly skilled artist who possesses a photographic memory. We show them an image and then have them recreate it and they can create an almost perfect copy just from their memory. The photographic memory of this artist is Stable Diffusion's VAE. In the case of restoration we would show this artist an image that has been heavily degraded by image compression and then ask them to recreate this image from memory, but in a way they think the image could have looked before it was degraded. In the case implemented here however we show them the original, uncompressed image and ask them to memorize it as good as they can. Then we perform brain surgery on them and shrink the data in their memory by applying some lossy compression to it which removes nuances that seem unimportant and replaces very very similar variations of concepts and aspects of the memorized image by the same variation. After the surgery we ask them to create a perfect reconstruction of the image from their memory. They'll still remember all the important aspects of the image, from content to qualitative properties of the camera grain for example and the location and general look of every building they saw, although the exact location of every single dot of the camera grain isn't the same anymore and some buildings they'll remember now with weird defects that don't really make sense. Finally we ask them to draw the image again, but this time, if they remember some aspects in a really weird way that doesn't make much sense to them, they should use their experience to draw these things in a way that does make sense (note that "doesn't make sense" isn't the same as "bad" - a blurry or scratched photo is bad, but that defect makes perfect sense). So in both cases the artist is asked to draw things to look like what they think they should look based on their experience, but since the compression here has been applied to their memory representation of the image, which stores concepts more so than pixels, only the informational content of the concepts has been reduced, whereas in the case of restoring an image degraded by image compression both the visual quality AND the conceptual content of the image have been reduced and must be invented by the artist. This analogy also makes it clear that this compression scheme is limited by how good the artist's photographic memory works. In the case of Stable Diffusion v1.4 they are not very good at remembering faces and also suffer from dyslexia.


Code If you would like to try this experimental codec yourself, you can do so with the following Google Colab: https://colab.research.google.com/drive/1Ci1VYHuFJK5eOX9TB0Mq4NsqkeDrMaaH?usp=sharing


Conclusion Stable Diffusion seems to be very promising as basis for a lossy image compression scheme. My experiments here were quite shallow, and I expect that there is a lot more potential to compressing the latent space representation beyond the technique described here, but even this relatively simple quantization and dithering scheme already yields very impressive results. While a VAE specifically designed and trained for image compression could likely perform better on this task, the huge advantage of using the Stable Diffusion VAE is that the 67 figures that were already invested into Stable Diffusion's training can be directly leveraged without having to invest vast amounts of money into training a new model for the task.


Quick Take On Text to Image Conversion With AI - Using Stable Diffusion

What is text to image conversion model? Simplistically, it's a model which produces matching images to the provided text description as closely as possible. It falls under the domain of generative AI and is one of the use cases for deep learning.


Generative AI Artificial intelligence, although in its nascent stage, has come a long way to penetrate the way we interact, engage and express. Generative AI is one facet of this evolution that allows algorithms to imagine words and voices into pictures and expressions. It creates unbiased results, which generally result from human thoughts and experiences. Generative AI models are mostly based on techniques such as generative adversarial networks (GANS), transformers, and variational autoencoders.


AI in art Although I do not understand much when it comes to art, I am definitely fascinated by the idea of an AI doing it for me! Recently there was a lot of buzz around an AI-generated art winning an international competition. Although such arts will never surpass the legacy and era of various artists around the globe over history, I believe it will definitely make art more accessible to the masses and carve its own niche. Read the complete article at NYTimes.


What is Stable Diffusion, and how it works? Text-to-image converters have been out there for quite some time now, but tools released this year (2022) - like DALL-E 2, Imagen, Midjourney, and Stable Diffusion - make it possible for almost anyone to create photorealistic works just by typing in some text. While there are multiple programs out there that support text-to-image conversion, in this article, we explore Stable Diffusion as one of the models. No specific reason of choice as such - but just because I found it simple for a first try out! :D How does it work? From a user's perspective, that's pretty straightforward. You type in your imagination in words, and the model will churn out interesting art. It uses a complex process of "diffusion" to turn text into images. In the case of text-to-image conversion, the model tries to learn the underlying pattern of the input and then uses that info to generate close-fit images. It may not always produce a new image, rather, it tries to reach the closest outcome by mixing and matching the images it already has.


Infinite possibilities I tried giving a few prompts to the Stable Diffusion model, and this is how it stunned me -


Some user-generated images from Stable Diffusion blogs - Courtesy: https://stability.ai/blog/stable-diffusion-public-release


Challenges - Mainstream Blockers Most models are trained by web scraping images at large and therefore undergo no scrutiny. As of today, while writing of this article, this can lead to potential misuse, unpredictable outcomes, and other ethical problems with the widespread use of this technology. Although we are not far from a stage where AI becomes capable of doing the majority of human chores, the challenge of modeling ethics into its' core remains an unsolved puzzle.


Conclusion Generative AI is one domain that is fast rising to the mainstream as we speak. With its ever-increasing use cases like text-to-image conversion, image-to-image conversion, image resolution enhancements, face aging, photos to emojis, audio synthesis, sentiment analysis, and trend evaluation, it's a boon to us. The advances are likely to increase, and generative design techniques are likely to empower the machines to do more than just manual labor and take up creative tasks.


Wrapping it up Do share in the comments  your thoughts about this super cool generative art model, its future, and how you would like to use this further. Also do share with me the interesting art you generate with Stable Diffusion.   - send a few claps if this quick round-up helped you in ways - do share this article with curious folks looking to explore - press follow to tune up on more such simplified stuff around cloud, technology, and science Connect with me on LinkedIn.


Latent Diffusion Explained Simply (with Pokemon)

Text to Image Text to Image is probably the most famous application of latent diffusion models (at least according to my Twitter feed). Much like a conditional-VAE, we use a text prompt (y) to generate conditionally from the distribution p(zy), thus serving as a guide to an area of the latent space. I have experimented using "Pikachu" which I figured the model would have seen pictures of. Additionally, I have inserted the location prompts to be about The Starry Night by Vincent van Gogh and the Canary Islands. The former should push the model to draw a 2D Pikachu to fit the art style, while the latter, being a real-world location, could result in a 3D Pikachu.


Results This is quite pretty - even considering the occasional deformities of the Pikachu. The first picture is probably my favorite as it even picked up the church-like building and added a field. Definitely more successful in getting the surfboard and the water. An interesting detail is the dunes of sand in some of the images which you can find in the Canary Islands:


How it works The input y is pre-processed by a domain-specific encoder (tau_theta) to produce a representation tau_theta(y). The underlying UNet architecture is implemented with cross-attention layers, meaning that at each unit of the UNet, attention is calculated and then passed to later layers. Attention is usually expressed with the Query, Key, Value terminology. Here, these values will be modified by taking into account the output of the domain-specific encoder tau_theta(y) as such: Meaning we are encoding our prompt to condition the de-noising step and therefore output an image that would fit the description.


Image to Image This task is quite similar to the Text To Image. The input, however, is an image that serves as a prompt. The image is then encoded using the domain-specific encoder tau_theta(y) and fed into the model just like text. In addition to the image, you can also input a text prompt to guide the generation process further.


Results My inputs to the models were: "A fire-type Pokemon spitting blue fire."This doodle (I know, it looks terrible): The results surprised me: Most of these look Pokemon-like, and several of them integrate the blue as fire. Unfortunately, none of them used the tophat as a tophat (we can blame my drawing skills for this), but most of them used the black color on the head.


Inpainting Inpainting is the practice of removing or replacing objects in an image based on a mask. I remember back in 2010 (or potentially earlier) when Photoshop released it as a feature as part of their "Healing Brush," and I was just astonished. Latent diffusion models, you guessed it, can be used for this too! HuggingFace has a great intuitive User Interface where you can paint the mask on the image itself:


Results The hat looks somewhat like a squashed version of the Harry Potter sorting hat - in the AI's defense, we did give it quite a small mask. I also like that it adapted it to the style of the video game. There are a few other elements to notice: There was an attempt at fixing the flooring around the hat. It's not amazing, but it is a start.Dawn's eyes appear darker and almost empty in the generated image. Her mouth also changed shape slightly. I believe this is due to the loss of information in the encoding and decoding step Finally, I also tried to create a new character: It has a backpack and a weirdly-shaped hat and legs. The "blue" prompt was taken up for the trousers and the hat partially.


How it works Very similar to Image To Image, in the Inpainting process, we modify the masked area ("unknown region") while keeping the rest of the image ("known region") fixed. As with other models, we can input a text prompt which will be encoded and passed on together with the known region of the image. The model will then fill in the unknown region with an image that fits the shape of the region AND the text prompt. Earlier, I mentioned how the latent representation offers advantages in computation and therefore speed. For Inpainting, latent diffusion models are at least 2.7x faster with a 1.6x increase in Frechet Inception Distance (FID). This metric quantified the difference in distribution (mean and standard deviation) between real images and generated images rather than pixel-by-pixel metrics. The idea is that it tries to imitate the "perception" of similarity between images.


Other Uses The paper presents other ways latent diffusion models could be used: image super-resolution and layout-to-image synthesis. Image super-resolution is the process of generating a high-resolution image from a low-resolution one. It is particularly useful in biomedical applications like MRI scans. I doubt the model could be used effectively in such tasks without any re-training. However, with normal images, the results are visually pleasing: Layout-to-image synthesis aims at generating images that fit a bounding box which guides the generation of an element in a specific position of the image. This is an example:


Limitations & Ethical Considerations One very intuitive limitation is that the model cannot generate images without any prior knowledge about a topic. For instance, without knowing what The Starry Night looks like, it would be impossible to generate images related to it. It would be interesting whether similar-looking images could be generated purely based on a text description and without any painting from van Gogh.Potential misuse of the AI model can have a significant societal impact, for example, creating fake images related to spam/news or offensive images. The original repository has several NSFW filters which hide the offensive images and also place an invisible watermark on the image, although it is not impossible to bypass it given the open-source nature of the model.These models are particularly susceptible to language bias and may display it in the generation of the images. Therefore, it is possible for the AI to misunderstand the prompt and generate potentially unrelated images.This model has been used to generate art and has even won art competitions (without the judges knowing it was AI-generated). Should this be allowed? Also, would you be interested in an art competition using (only) latent diffusion models?


Conclusion In this article, I covered three uses of latent diffusion models: Text To Image, Image to Image, and Inpainting. I explored (in the context of Pokemon) how each of these could be used to generate images and peculiar details for each generated image. There are other ways these models can be used, for example for image super-resolution and layout-to-image synthesis. If you would like a more mathematical explanation of some of the parts of this blog post, I highly recommend the original paper: or this blog post: Disclaimer: all the generated images in this post have been created by stable diffusion (and me) using the official GitHub code: If you do find any mistakes or inaccuracies, please feel free to comment/contact me, and I will update the article (and include you in the special thanks section).


This AI newsletter is all you need #12

Hottest News An artist won an art competition using AI!This brings a lot of controversy both to the AI and art fields. Many people find it fantastic, and many others think it is cheating (even up to the point of sending hate mail...). Honestly, it doesn't really surprise me when seeing Stable Diffusion and DALLE results and what some artists do with those generated images using Photoshop to further improve realism and the quality of the results. Plus, many artists have to learn how to code and most of them don't only use DALLE. They have to plug in multiple networks, like an upsampler model and some outpainting SOTA models. That's an art in itself!What do you think?NVIDIA RTX 3080Ti Giveaway!Once again, many YouTubers, including Yannic Kilcher and myself (What's AI) are giving away graphics cards thanks to NVIDIA in collaboration with their GTC events. Yes, this is another chance for you to win an NVIDIA RTX 3080Ti! Also, for the GTC event in collaboration with NVIDIA! Learn more about the giveaway and how to participate in my video :)Professional AI Whisperers have launched a marketplace for DALL-E promptsYes, this is true. You can now buy prompts that "guarantee" good results..."PromptBase is at the center of the new trade-in prompts for generating specific imagery from image generators, a kind of meta-art market. [...] the platform lets "prompt engineers" sell text descriptions that reliably produce a certain art style or subject on a specific AI platform."How does this sound to you? Honestly, I think it may be worth it in some very narrow use cases where you really want a specific style and have tried everything without success. I have no idea. What do you think? - I'd love to know.


This AI newsletter is all you need #12

Most interesting papers of the week MotionDiffuse: Text-Driven Human Motion Generation with Diffusion ModelThe first diffusion model-based text-driven motion generation framework, responding to fine-grained instructions on body parts, and arbitrary-length motion synthesis with time-varied text prompts.FETA: Towards Specializing Foundation Models for Expert Task ApplicationsFETA, a novel benchmark for evaluating and improving Foundation (V&L) Models performance on expert data tasks (understand technical documentation, via learning to match their graphical illustrations to corresponding language descriptions).In conversation with Artificial Intelligence: aligning language models with human values [DeepMind]This paper proposes interesting answers to common questions involving an agent discussing with humans and how to align such agents with us (ethically, socially...): A few of the questions discussed are: What does it mean to align conversational agents with human norms or values? Which norms or values should they be aligned with? And how can this be accomplished? Enjoy these papers and news summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #12

The Learn AI Together Community section! It's finally time... Our RTX 3080Ti Giveaway in collaboration with NVIDIA! Once again, we are giving away an NVIDIA RTX 3080Ti to one lucky member of the Learn AI Together community for the GTC event in collaboration with NVIDIA! We are also giving the community 10 Deep Learning Institute (DLI) codes! Learn more about the giveaway and how to participate in the most recent post shared in the weekly-ai-news-video channel  NVIDIA GTC's second 2022 conference will be happening September 1922. Attend one or more talks and send me a screenshot to enter the giveaway. Check out the ai-events channel, where we shared the talks that seemed most interesting for this community.


This AI newsletter is all you need #12

Meme of the week! Meme shared by our fantastic Roberto, from the Towards AI team.


This AI newsletter is all you need #12

Featured Community post from the Discord We love it when you share research, whether it be your own or simply interesting papers, blog posts, or code you've seen. This week, I personally thank Inazuma#9447 for sharing a very interesting paper with the community! Special thanks to sin yee#3513 as well for sharing another very interesting paper related to NLP!


This AI newsletter is all you need #12

AI poll of the week! Join the discussion on Discord


This AI newsletter is all you need #12

Article of the week Fake News Detection using BERT Model Python: The rise of social media has increased the impact of Fake News on our culture. People frequently believe what they read or hear to be true, and this has a huge impact on the globe, both politically and financially. The author develops a BERT Classification model to detect Fake News using Python in this post, employing a step-by-step methodology, well-stated model architecture, and instructive graphics. If you are interested in writing for Towards AI, please sign up here and we will publish your blog to our network if it meets our editorial policies and standards. https://contribute.towardsai.net/


This AI newsletter is all you need #12

Lauren's Ethical Take on "In conversation with Artificial Intelligence: aligning language models with human values" We love to see an ethics paper! I really like the approach that authors Atoosa Kasirzadeh and Iason Gabriel at DeepMind took for this analysis. By focusing on the linguistic basis of speech production goals and connecting ethics to these situations and outcomes, it really highlights the nuance that's needed for applying ethics to different settings. A model's ability to generalize is laudable in many contexts, but not every context - some require extreme specificity for success. In addition to this, determining the extent and impact that the value of the speech affects the recipient. Types of utterances only add to this complexity. Why is this linguistic focus so important? By drawing their conclusions from ideals based on agent-to-human interactions, the goal of the speech is automatically more realistic and replicable. This is an essential foundation to develop because language models are often built with the success condition of replicating human-to-human conversation, which can be unattainable or limiting based on the goal of the chatbot. Evolving our ethical applications is just as important as the ethics themselves, as nothing takes place in a vacuum - especially not applied AI ethics! Join me next week as I dive further into the ethical implications of this paper!


This AI newsletter is all you need #12

Job offers Senior Software Engineer @ Captur (Remote, +/- 2 hours UK time) Senior ML Engineer @ Safe Security (Remote) ML/Algorithms Engineer @ Aurora Insight (Hybrid Remote) Senior Staff Data Scientist @ One Concern (Remote) Research Scientist - Speech recognition @ Abridge (Remote) Computer Vision Scientist @ Percipient AI (Santa Clara, CA) Research Scientist - Machine Learning @ DeepMind (London, United Kingdom) Senior Data Scientist @ EvolutionIQ (Remote) Interested in sharing a job opportunity here? Contact sponsors@towardsai.net or post the opportunity in our #hiring channel on discord! If you are preparing your next machine learning interview, don't hesitate to check out our leading interview preparation platform, confetti!


Fake News Detection using BERT Model Python

Fake News Detection So Let's concentrate on our task and start the classification - As said, we need to pass tokenized Values as Input to the BERT Model. So let's use the tokenizer from the transformers library. Tokenization Let's start this by installing transformers that provide BERT Model. Let's start the tokenization - In this method, we will use AutoTokenizer from BERT (base) Model - Similar to all neural networks, transformers cannot process the raw input text directly, So we need to tokenize the input and convert them into numbers. Tokenization performs the following functions - It splits input text into tokens (words, letters, etc.), Mapping each token with a unique integer and arranging them according to the model. Since we are using the BERT BASE Model for our application, we have also used the corresponding AutoTokenizer for tokenization. Let's tokenize the values - Designing Model Importing Necessary Libraries - Designing BERT Function - Loading the BERT Model - Let's plot the BERT Model Created - We have created the Model according to the above explanation. Training Our Model As we can see, we got 99.9% Training Accuracy. Let's Evaluate our model - We can see our model is performing very well even on the evaluation data. We Can use this model, even to evaluate any daily news that we see. That's all, Folks...... To follow my content and know more about me, refer my below article - Happy Coding....


This AI newsletter is all you need #11

What happened this week in AI Stable diffusion drew our attention again, but more precisely to how much the "stable diffusion initiative" is influencing new research and advancing the field. It is incredibly cool to have such a powerful model open-sourced. Most of our friends in the image generation domain are currently playing and implementing various versions of it day and night. One of which we find very interesting and promising is a new paper titled "An Image is Worth One Word." "An Image is Worth One Word" allows you to personalize the results of a pre-trained text-to-image model like Stable Diffusion using your own images of an object with very little training time (~2 hours). It learns the concept from 35 images and formulates it into what they refer to as a "pseudo word" which you can then use in your prompt generations. It is very cool and has incredible potential for amazing game-changing products, and it's just one new research brought thanks to stable diffusion out of many more, and even more to come. We are living in exciting days for the image generation industry, and we will closely follow it with the Towards AI team for you!


This AI newsletter is all you need #11

Hottest News DALL·E: Introducing OutpaintingOpenAI just introduced outpainting to DALLE! Outpainting can extend the original image, creating large-scale images in any aspect ratio (see the cover image of this newsletter iteration). It takes into account the image's existing visual elements to maintain the context of the original image and can be conditioned with text to add specific elements.Top-22 AI influencers to follow on Twitter by 2023We are not sure how but our Co-founder and Head of Community, Louis Bouchard, was featured in this "Top 22 AI influencers to follow by 2023" article on Bytescout! We know most of the people on this list and we are incredibly grateful and excited that Louis is part of it. Check it out and give a follow to the other amazing people there!You've all heard and tried Stable Diffusion, but what is it?What do all recent super-powerful image models like DALLE, Imagen, or Midjourney have in common? Other than their high computing costs, huge training time, and shared hype, they are all based on the same mechanism: diffusion. Diffusion models recently achieved state-of-the-art results for most image tasks, including text-to-image with DALLE but many other image generation-related tasks too, like image inpainting, style transfer or image super-resolution. But what is diffusion and how does it work? Learn more in the article.


This AI newsletter is all you need #11

Most interesting papers of the week Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Model"A unified framework supporting a three-dimensional control over the image synthesis from sketches and strokes based on diffusion models [with which users can] decide the level of faithfulness to not only the input strokes and sketches but also the degree of realism."Turn-Taking Prediction for Natural Conversational SpeechWhile a streaming voice assistant system has been used in many applications, it is merely powerful for one-way discussions and basic question/answer unnatural interactions. As you know, it works pretty badly if you pause to think or accidentally repeat words. They present a turn-taking predictor built on top of the end-to-end (E2E) speech recognizer to help with fluent, "real", discussions.MULAN: A JOINT EMBEDDING OF MUSIC AUDIO AND NATURAL LANGUAGEMULAN: "a first attempt at a new generation of acoustic models that link music audio directly to unconstrained natural language music descriptions." "Human listeners prefer source estimates of bass and drums that have been post-processed by MSG." Enjoy these papers and news summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #11

Featured Community post from the Discord One of the Learn AI Together members, Ravioli#7085, has published their first independent research and preprint! Congrats Arav and we are excited to see the next publications (I have insights and a couple more are coming soon!)  Read Arav's publications. If you do have some publications already or coming up, please share them with us on the server!


This AI newsletter is all you need #11

Article of the week The Mathematical Relationship between Model Complexity and Bias-Variance Dilemma: Most data science enthusiasts will agree that the "Bias-Variance Dilemma" suffers from analysis paralysis since there is an enormous literature on the idea of Bias-Variance, its decomposition, derivation, and connection with model complexity. The author demonstrates why, despite our best efforts, simplistic models display significant bias while complex models exhibit minimal bias. If you are interested in writing for us at Towards AI, please sign up here and we will publish your blog to our network if it meets our editorial policies and standards. https://contribute.towardsai.net/


This AI newsletter is all you need #11

Lauren's Ethical Take on the future of LLMs I wanted to write about an awesome article from the MIT Technology Review that highlights a lot of ethical perspectives of large language models and our future with them. By posing the question, What Does GPT-3 "Know" About Me?, author Melissa Heikkilä brings a personal lens to a huge phenomenon. Beginning with her own information and expanding to cover others, she examines the striking dissonance between both the accurate and inaccurate responses (dubbed "hallucinations") given by LLMs. There is a necessity to explore a future where this information is ubiquitous, as all the models that we have are not going away anytime soon and there's more to come. Increased size and abilities naturally comes with increased vulnerabilities. While we have very different concepts and ways of enforcing privacy standards (like this big story regarding Meta), mitigating risks will require that we continue to innovate towards an ethical protection of privacy. Many support the idea that all public information is fair game is no longer going to cut it for approaching large scale privacy concerns. I'm excited to see where this future of protection takes us, and how we choose a direction to make progress in. Personal, regional or cultural differences affect how we understand what privacy looks like and how it should be protected. I encourage you to examine what that looks like for yourself!


This AI newsletter is all you need #11

Job offers Senior ML Engineer @ Safe Security (Remote) Research Scientist - Speech recognition @ Abridge (Remote) Computer Vision Scientist @ Percipient AI (Santa Clara, CA) Research Scientist - Machine Learning @ DeepMind (London, United Kingdom) Senior Data Scientist @ EvolutionIQ (Remote) Senior ML Engineer - Semantic Search @ Algolia (Hybrid remote) Interested in sharing a job opportunity here? Contact sponsors@towardsai.net or post the opportunity in our #hiring channel on discord! If you are preparing your next machine learning interview, don't hesitate to check out our leading interview preparation platform, confetti!


TransUNet - Revolutionize Traditional Image Segmentation

Table of Contents ·  Intuition·  TransUNet ∘ Down-Sampling (Encoding)  ∘ Up-Sampling (Decoding)·  Results·  Implementations· Reference


Intuition Nowadays, U-Net has dominated image segmentation tasks, especially in the field of medical imaging. Among most of the hitherto proposed U-Nets, Convolution Neural Networks (CNN) are widely used as their underlying structures. However, CNN's can only exploit short-range (or local) information effectively due to their small convolutional kernel sizes, failing to explore long-range information sufficiently in tasks that have features with long-range relations. Transformers, commonly used to handle Natural Language Processing tasks, can explore long-range information effectively, but they are not robust in exploiting short-range information as CNNs do. To use the power of CNNs can compensate for the shortcomings of Transformers and vice versa in image segmentation tasks, Chen et al. proposed TransUNet, which is also the first image segmentation model built from Transformer. It's also worth mentioning that the authors verified the promising outcome of combining CNNs and Transformers by firstly attempting to use pure Transformer architecture for image segmentation. However, it did not work as well as introducing CNNs in their architecture because transformers are not as good as CNNs in exploiting local features.


TransUNet We know that in U-Net (Fig. 1) there is an encoder (the down-sampling path) and a decoder (the up-sampling path). The down-sampling path will encode the image's features to a high-level map, and its details will be used by the up-sampling path to produce the final mask of the same dimension as the input. Similarly, TransUNet also comprises an encoder and a decoder for encoding and decoding image information to produce a segmentation. Different from traditional U-Nets, TransUNet instead uses a hybrid CNN-Transformer architecture as an encoder to learn both the high-resolution spatial information from CNNs and global context information from Transformers. To break down the steps:


Down-Sampling (Encoding) Firstly, CNN is used as a feature extractor to generate a feature map for the input, as shown in the pink box in Fig. 2.For each level of the feature extractor, the output feature map (that encodes for intermediate high-level feature maps) is then concatenated to the decoder path of the same level, as shown by the dotted arrows in Fig. 2.Then, the feature map is tokenized (vectorized) into a 2D embedding of shape (n_patch, D) by linear projection, and D is the total length of the embedding. The embeddings were pre-trained and will retain the positional information of the feature map (if you don't understand how don't worry about it for now, as it won't hinder your comprehension of TransUNet too much).After obtaining the embeddings, they are fed into 12 Transformer layers to encode less short-range and more long-range information from the image. Each layer is illustrated in Fig. 2 (a), which uses multi-head self-attention (MSA) and multi-layer perceptron (MLP) modules. MSA is the basic building block for transformers and is explained here, and MLP is just comprised of several fully connected layers.Lastly, to prepare for the up-sampling path, the output is reshaped to (D, H/16, W/16). H/16 and W/16 mean that the heights and widths by this time have been shrunk by 16 times because of the previous operations.