Trends in AI - June 2022

4. Large Language Models are Zero-Shot Reasoners By Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, Yusuke Iwasawa. Why -> An unbelievably effective and simple hack for prompting language models. Key insights -> This paper basically discovers that adding the simple phrase "Let's think step by step" to the prompt of GPT-3 and other similar models dramatically improves reasoning performance. This is a finding that goes in line with Chain of Thought prompting (CoT)⁹, which showed how reasoning and factual accuracy can be improved in one-shot learning by explicitly unpacking the reasoning as part of the natural language prompt. The figure below succinctly illustrates how all these techniques relate to each other. Upon such discovery, the authors propose a zero-shot-CoT which is a template to elicit a type of chain of thought reasoning in a zero-shot setting, without requiring a handcrafter chain-of-thought prompt as it was done previously. For instance, the template can be adding first the "Let's think step by step" prompt suffix, use the output of the model as a chain of thought and then perform answer extraction by adding another prompt suffix such as "therefore the answer in Arabic numerals is", running the model for one final inference step. The results show how big of a difference simple choices of templates and prompts make. It's still fascinating how after 2 years of the release of GPT-3, so much low-hanging fruit is still waiting to be collected in the space of prompting.


Trends in AI - June 2022

5. Vision GNN: An Image is Worth Graph of Nodes By Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang & Enhua Wu. Why -> We often assume images should be canonically represented as a grid of pixels. What if we represented them as graphs? Key insights -> This paper proposes Vision GNN (ViG), an architecture to extract graph-level features for visual tasks. The graph representation of an image is simply obtained by dividing an image into patches, embedding those patches, and constructing a graph where edges represent nearest neighbors in the embedding space. This graph is then fed into a GNN consisting of a clever combination of Graph Convolutions (GCNN), linear transformations, and activation functions that avoid the common pitfalls of GCNNs such as over-smoothing. Similar to classical ResNets, the authors find that the GNN benefits from a "pyramid structure", where spatial features are iteratively aggregated into deeper features that combine information from an increasingly large neighborhood of the image/graph. The results on Imagenet are either SOTA or comparable to existing ResNets⁵, ViTs⁶, or MLP-based⁶ architectures. Once again, this reinforces the growing suspicion that specific architectures in Deep Learning matter less than we used to believe, and scale and data are instead of the main drivers of performance.


Trends in AI - June 2022

6. Unifying Language Learning Paradigms By Yi Tai, Mostafa Dehghani et al. Why -> Self Supervised Language Modeling based pretraining is now the bread and butter of NLP. Still, different flavors of this technique exist, empirically showing complementary strengths and weaknesses. Wouldn't it be nice to get the best of all of them? Key insights -> Consider the following taxonomy for existing LM pretraining methods: Vanilla or causal Language Modeling. "Predict the next token" where each token can only attend to preceding tokens.Prefix Language Modeling: "Predict the next token" but now all input tokens can attend to each other.Span corruption, Masked Language Modeling (MLM), or bidirectional language modeling. "Fill in the gaps", where all tokens can attend to all tokens. While autoregressive LMs excel at text generation and prompting, MLM is better at extracting powerful general-purpose representations of language. The unification of pre-training tasks can be understood through the perspective of denoising an "input-to-target" task: a model learns to reconstruct a corrupted input and predict a target. By defining different qualitative types of denoising combining long and short spans in the input and target domain, the resulting objective function combines the qualities of the different objectives we described initially. In grossly oversimplified terms, R-denoiser mimics the T⁵⁸ learning objective, S-denoiser is GPT-like and X-denoiser is a combination of both. The result is a model that can successfully operate both in the supervised finetuning and one-shot prompting paradigms. When compared to previous learning objectives in isolation, UL2 often outperforms them, although not universally across the board. This is a promising direction for future foundation models as a service that can simultaneously be used to (1) generate powerful general-purpose embeddings of text and (2) solve zero-shot tasks via prompting and text generation a la GPT.


Trends in AI - June 2022

7. UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes By Alexander Kolesnikov, Andre Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby. Why -> While NLP has seen a unification of techniques across tasks with pretraining based on language modeling and Transformers, Computer Vision (CV) still remains more fragmented in terms of architectures and methods for different tasks such as object segmentation, coloring, inpainting or depth mapping. Key insights -> A base model learns a pixel-level output given an input image. Traditionally, pixel-level dependencies are hard to model so task-specific inductive biases are often introduced in the learning process. This work proposes to instead introduce an auxiliary restricted oracle model whose objective is to guide the training of the base model across a wide range of pixel-level CV tasks. This restricted oracle takes the ground truth output (e.g. a per-pixel segmentation map) and generates a short sequence of discrete tokens which contain useful information about the task at hand. The intuition is that by jointly optimizing the base model and the restricted oracle, the oracle will find ways to aid the predictions of the base model, enabling it to learn more efficiently complex tasks without requiring task-specific design choices. Note that during training, the model doesn't purely perform prediction because the information from the ground truth label is purposefully leaked into the model via the guiding code. While the results are not challenging the state of the art, it proves that the restricted oracle guidance improves the performance of the base model and is a viable research direction for more generic vision methods.


Trends in AI - June 2022

8. Retrieval-Enhanced Machine Learning By Hamed Zamani, Fernando Diaz, Mostafa Dehghani, Donald Metzler, and Michael Bendersky. Why -> Relying on growing the parameter count to improve the performance of the ML model seems successful but is also brittle, unsustainable, and opaque. Recent works combining IR and Language Models display a much more promising parameter efficiency while being more generalizable, scalable, interpretable, and robust. Key insights -> Previous works this year such as Web-GPT (OpenAI), RETRO (Deepmind) or LaMDA (Google), already showcased how language models that interacted with a retrieval model displayed better parameter utilization and robustness; with the added benefit of being able to add information to the system without the need to retrain the whole model. This is a very useful position paper that lays out the conceptual foundations of ML models that interact with retrieval systems. A retrieval-enhanced ML model is simply a machine learning predictive model which interacts explicitly with a memory module that can be queried for information through a retrieval model (see figure below). The authors also provide examples of case studies as potential avenues where retrieval enhanced ML is being investigated and shows a promising future. These include knowledge grounding, memory augmented ML, retrieval enhanced input representation, and generalization through memorization.


Trends in AI - June 2022

9. A Modern Self-Referential Weight Matrix That Learns to Modify Itself By Kazuki Irie, Imanol Schlag, Róbert Csordás, Jürgen Schmidhuber. Why -> Meta-learning (aka learning to learn, and learning to learn, and so on) has long been one of those ideas in AI whose importance appears self-evident in the abstract, yet has proven hard to popularize. While this work first appeared earlier this year in February, it has recently been accepted at the ICML conference and generated more discussion. Key insights -> This is a work that applies modern techniques to dust off a meta-learning idea dating back to 1991 from Schmidhuber's group. The motivation for this line of work can be best understood through Schmidhuber's blog post covering the historical context on meta-learning. The idea is to reconceptualize many of the multi-head attention used in Transformers as a combination of fast and slow NNs (or weight matrices), where the purpose of the slow NN is to modify a fast NN, which becomes a more general formalism to build NNs that can meta-learn (i.e. that can "improve themselves" in a virtuous cycle). They test the approach in supervised learning (especially few-shot) and multitask reinforcement, showing that this is a viable technique, though not achieving jaw-dropping performance. To be honest, I'm somewhat skeptical about how useful it is to revisit existing methods under alternative terminologies and abstractions that relate to older methods: at the end of the day this is arguably not all that different from multi-head attention. It can feel like an unnecessary obfuscation of the contributions of the paper. Still, seeing the same thing in a new light can reveal valuable novel insights, so this is after all a challenging but worthwhile read!


Trends in AI - June 2022

10. Towards Understanding Grokking: An Effective Theory of Representation Learning By Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J. Michaud, Max Tegmark, and Mike Williams. Why -> Using the physics toolbox with effective theories and phase diagrams to understand a puzzling phenomenon that Neural Networks exhibit: grokking. Key insights -> Grokking is a puzzling phenomenon that NNs can exhibit where they drastically generalize to the test set way after overfitting the training set³. This has been shown on algorithmic datasets - e.g. learning binary operations between integers - which hold out certain combinations from the training set and use them on the test set to determine generalization. Often used in physics, an effective theory is a descriptive model of a certain phenomenon that doesn't claim to reflect any true underlying causal structure of said phenomena. To construct an effective theory of representation learning, this paper lays out a definition of structure based on the geometry of the embedding space. Based on this, it analyzes how big of a training set is needed to recover the underlying structure of an algorithmic dataset such that the model generalizes to the test set. The representation learning phases are defined as 4 regimes of learned representations in NNs: Comprehension: both training and test performance improve simultaneously.Grokking: training and test performance are high, but generalization happened after training performance plateaued.Memorization: training performance is high but test performance is low.Confusion: both training and test performance are poor. Then, phase diagrams can be drawn for various hyperparameters of choice such as representation encoder/decoder learning rates, or train/test split ratio among others. Interestingly, their effective theory correctly predicts the boundary conditions for phase transitions for parameters such as train/test data split. Most of the results in the paper come from algorithmic datasets in toy settings, but the authors conjecture that the analysis in the training dynamics will hold for more general cases and provide a simple example with a Transformer. Even if that's a hopeful assumption, it's useful to learn new grounded abstractions that help to intuitively conceptualize complex phenomena, which is something physicists excel at. Moreover, understanding complex emergent phenomena from NNs is very helpful in efficiently developing increasingly large models, where sweeps across design choices and hyperparameters are unfeasible computationally.


Saving Data in Tableau: A Road to Tableau Desktop Specialist Certification

Table of Content .twb files.twbx files.hyper/.tde files.tds files.tdsx filesSumming upSample Certification Questions from this topic


.twb files .twb files contain no data source but it constitutes how Tableau interacts with the data source i.e. all the connections and mapping. In addition, it contains one or more worksheets, and zero or more dashboards. .twb files are useful for local purposes only i.e. sharing .twb files makes no sense as it will crash once the data directory is modified. .twb files are XML files.


Creating a .twb file Step1: File>Save As Step2: Choose ".twb" extension.


.twbx files .twbx files are packaged workbook that contains the workbook along with the data source. .twbx files can be shared as it contains a local copy of your data source along with worksheets and dashboards. Others can easily open the twbx files with Tableau Desktop or Tableau Reader. The default directory to save .twbx files is "Tableau Repository>Workbooks" but we can change the default locations. Along with a copy of your data source, a .twbx also includes: Background imagesCustom geocodingCustom shapesLocal cube filesMicrosoft Access filesMicrosoft Excel filesTableau extract files (.hyper or .tde)Text files (.csv, .txt, etc.)


Creating a .twbx file There are two ways to create a .twbx file. Way1 -> Step1: File>Save As Step2: Choose ".twbx" extension. Way2 -> File>Import packaged Workbook


.hyper/.tde files In Chapter x, we learned about live and extract connections. To save our files in extract forms we use .hyper/.tde extensions. .hyper/.tde formats create a local copy of our data. This file format is useful when we want to share our data source with our colleagues. This format only contains a local copy of our data and no worksheet or dashboard. But make sure, you don't share any confidential data. .tde files are discontinued and the extension name has been changed to .hyper.


Creating a .hyper file There are two ways to create a .hyper file. Way1 -> Step1: Data>" Your data source name">Extract Data Step2: Filter the data(if required) and choose if you wish to refresh the data automatically. Click on "Extract". Way2 -> Step1: Right-click on the data source name and choose "Extract Data". Step2: Filter the data(if required) and choose if you wish to refresh the data automatically. Click on "Extract".


.tds files .tds files contain no data source but all the connections and mapping i.e. data model along with additional modifications done by you, such as parameters, calculated fields, default properties, etc. .tds contains no worksheets or dashboards.


Creating a .tds file Step1: Data>" Your Data Source">Add to saved Data Source. Step2: Choose ".tds" file. Locally saved Tableau data source (.tds) files appear under Saved Data Sources, on the Connect page. .tds files are by default saved at 'My Tableau Repository>DataSources'.


.tdsx files .tdsx files are packaged data source files. .tdsx contains a copy of your data, and modifications made by users in data(parameters, filters, calculations, etc).


Creating a .tdsx file Step1: Data>" Your Data Source">Add to saved Data Source. Step2: Choose ".tdsx" file.


Sample Certification Questions from this topic _____________ contains the visualizations, info needed to build the visualizations, and a copy of the data source. a. .twbb. .twbxc. .tdsd. .tdsx Solution: .twbx


References: [1] Tableau Help  Tableau Software [2] Personal Notes [3]Tableau Desktop Specialist Exam (New Pattern - 2021) - Apisero


Thanks for Reading! Feel free to give claps so I know how helpful this post was for you, and share it on your social networks, this would be very helpful for me. If you like this article and want to learn more about Machine Learning, Data Science, Python, BI. Please consider subscribing to my newsletter: Find me on the Web: www.dakshtrehan.com Connect with me at LinkedIn: www.linkedin.com/in/dakshtrehan Read my Tech blogs: www.dakshtrehan.medium.com Connect with me at Instagram: www.instagram.com/_daksh_trehan_


Want to learn more? How is YouTube using AI to recommend videos?Detecting COVID-19 Using Deep LearningThe Inescapable AI Algorithm: TikTokGPT-3 Explained to a 5-year old.Tinder+AI: A perfect Matchmaking?An insider's guide to Cartoonization using Machine LearningHow Google made "Hum to Search?"One-line Magical code to perform EDA!Give me 5-minutes, I'll give you a DeepFake!


Dashboards & Stories in Tableau: A Road to Tableau Desktop Specialist Certification

Table of Content Dashboards Layout of Dashboard Objects in Dashboard Dashboard Actions Best practices for creating a Dashboard Additional NotesStories Layout of Story Objects in Story Best practices for creating a Story Additional NotesSample Certification Questions from this Topic


Dashboards Dashboards can be referred to as the combination of various visualizations.


Layouts of Dashboard The layout of a Dashboard could be defined on: Phone, Tablet, Desktop, and Default. While defining the size of the Dashboard, we can choose from three options: Fixed Size -> We can define a fixed size for our dashboard, we can choose from multiple pre-defined options. Automatic -> The dashboard will automatically configure its size based on the objects it constitutes. Range -> We can give a range for the size of our dashboard.


Objects in Dashboard In addition to our visualizations, we can also add various objects to our dashboard to increase attractiveness and make it more appealing. Horizontal & Vertical -> These are the layout containers that let you add sheets either in horizontal or vertical format. Text -> This lets you add static text in the dashboard. Image -> We can add a background image or a static image to our dashboard using this. Web Page -> Using this object, we can add a webpage to our dashboard. Blank -> It lets you add some blank space to the dashboard. Navigation -> It would let you navigate to other dashboards or worksheets. Downloads -> It will let the audience download the dashboard in various formats. Extension -> It will let users add third-party extensions to the dashboards. Ask Data -> It lets users enter conversations that could be later answered by authors of the dashboard. We can add our objects either in Tiled or Floating format. A floating format is preferred as it helps to create more appealing and optimized dashboards.


Dashboard Actions To employ Dashboard Actions, go to Dashboard>Actions. There dashboard actions we can implement are: Filter, Highlight, Go to URL, Go to Sheet, Change Parameters, Change Set Values. For "Go To URL", we can specify where the link will be opened. The actions could be triggered by either Hovering, Selection, Menu. Hover -> This will change the set, once the user hovers over any mark in the view.Select -> This will trigger the action when the user will click on any mark in the view.Menu -> This will trigger the action when the user will right-click on the mark and select an option on the context menu. Choose how to run the action and source and target sheets.


Best Practices for Creating a Dashboard Put the most important visualization in the top-left corner of the dashboard.By default size of the Dashboard is set to fix, make it automatic.Allow flexibility to users by introducing filters and parameters in the view.Tooltips are a great option to add extra information to your dashboard.Use the limited number of visualizations in a single dashboard.Make it crisp and concise.


Additional Notes While creating Filter actions, use Hovering for highlighting, selecting for filtering, and Menu action for tooltips.The safe URL prefixes allowed in Tableau Dashboard are HTTP, HTTPS, and FTP.The visualization in the tooltip is a static image, not an interactive view.If you use Show Me in the source sheet to change the view structure, you will reset all tooltip edits, including Viz in Tooltip references.We can export dashboards in jpg, png, emf, BMP format. Exporting can be done by choosing Dashboard > Export Image.We can't delete a worksheet if it is present in a dashboard.When we modify a sheet or a dashboard the view is changed in both worksheets as well as dashboard.


Stories Stories can be referred to as the combination of various dashboards.


Layouts of Dashboard While defining the size of the Dashboard, we can choose from three options: Fixed Size -> We can define a fixed size for our story, we can choose from multiple pre-defined options. Automatic -> The story will automatically configure its size based on the objects it constitutes. Range -> We can give a range for the size of our story.


Objects in Stories We can add different sheets, dashboards, or titles to our Stories.


Best Practices for Creating a Story Keep it crisp & concise.Use a chronological approach.Use Filters to make it interactive.Decrease the load time.


Additional Notes The delete option for a worksheet is disabled if it is being used in any story.If we modify any sheet used in a story, the view will also be changed in the story and vice-versa.


Sample Certification Questions from this Topic Which of the following are the options to export the data used to build the view/visualizations? [Choose 2] a. JSON Formatb. MS Access Databasec. CSV filed. PDF file Solution: CSV file, MS Access Database


Thanks for Reading! Feel free to give claps so I know how helpful this post was for you, and share it on your social networks, this would be very helpful for me. If you like this article and want to learn more about Machine Learning, Data Science, Python, BI. Please consider subscribing to my newsletter: Find me on Web: www.dakshtrehan.com Connect with me at LinkedIn: www.linkedin.com/in/dakshtrehan Read my Tech blogs: www.dakshtrehan.medium.com Connect with me at Instagram: www.instagram.com/_daksh_trehan_


Hand Cricket Simulation Using CNN and OpenCV

Step-1: Data Collection and Formatting The data required for this project must be images that represent the numbers 0-6 (inclusive of 0 and 6). I have collected the data and uploaded it to Kaggle: Hand Symbols. I have collected this data using the OpenCV library of python. Here's a quick read on how I did it: Data Collection. I have collected a total of 28800 jpg images of dimensions 240x240 pixels. Of these 28000, 20000 are stored in the 'train' folder, and the remaining 8,800 images are in the 'test' folder. Why exactly 28000 images? Why a train to test ratio of 20000:8800? Why a picture of size 240x240? No particular reason! I just kept collecting more and more data while simultaneously testing how the model works in real-time. These numbers resulted in a good enough model and so I stuck with them. Anyways, moving on! Both the train and test folders contain 8 more folders named - '0', '1', '2', '3', '4', '5', '6', 'none'. Each of the folders in the train folder contains 2500 images and the ones in the test folder containing 1100 images. Confused? I would be too. Here's a better representation to make it easier. I have established this structure in hopes of training the model on the data in the train folder and testing it on the data in the test folder. However, I ended up using all the data for training the model, because, my final goal is to implement it in real-time and not report the accuracy achieved by the model. If any of you are interested in finding out the accuracy, a structure similar to the one above could prove useful. Some of you might notice the folder named 'none'. This folder has images with no hand symbols. The images in this folder are used to establish a state when no symbol is displayed by the user. It essentially contains images with a random background. In terms of Machine Learning, this is a Multiclass Classification problem where each symbol is a specific class. It was necessary to include a class that indicated the state of no hand symbols displayed by the user to recognize images in real-time when the user did not choose a hand symbol.


Step-2: Loading the data and processing it The previous step would have us store the data in the working directory. We now have to bring it into the environment (store it in variables in the required format). Assume that the data is stored in a folder called 'dataset'. We iterate through this folder and the files inside it using functions from the OS library of python. Every image in every folder is converted into a NumPy array and stored in a list named 'images_as_numpyarr'. Another list called 'labels' is maintained which stores the number represented by the image, at the same index as the image stored in the images_as_numpyarr list. For those of you familiar with supervised machine learning, the labels list is used to provide supervision by labeling the data. So, images_as_numpyarr[0] would contain the image pixel data as a NumPy array, and labels[0] would contain the number being shown in the image. So if this was the very first image, images_as_numpyarr[0] would store this image as a NumPy array, and labels[0] would store the number 3. The labels 0 through 6 indicate the number being shown whereas the label 7 indicates that no number is being shown (the 'none' state mentioned in the previous section). The labels list would label the above image as number 7 (indicating the class of the image) and store it in the list. If the following images were found, the labels list would be:


labels = [5, 4, 6, 7] These labels have then been encoded using the One Hot Encoding technique in line 21.


Step-3: Designing the CNN model architecture Now that we have the data readily available, we need to design a CNN model and feed the data to it. Instead of building an architecture from scratch, I used Transfer Learning to make it easier to build an effective model. Here's an article that explains it in detail: Transfer Learning. In simple words, it is the process of using a pre-trained model (with weights), and modifying the last few layers to suit our needs. This technique helps cut down the time taken for building and compiling complex architectures. Here, I've used a lightweight architecture called MobileNet. Here's the architecture. Complicated? I feel you. For now, just understand that MobileNet is pre-built and pre-trained (which means some appropriate weights are generated by training). We will import this architecture, modify the last few layers (usually by adding new layers) and train the model on our data. Here's the code. I have imported the MobileNet architecture in Line 8 and added a few layers at the end (Line 913). I then compiled it using Adam optimizer with a learning rate of 0.0001. I tried training with different parameters and the ones in the code below proved to be good enough.


Step-4: Training the CNN model to count fingers Now that we have the architecture ready, we pass in the images along with the labels as NumPy arrays (Line 4). The shape of images_as_numpyarr[0] is 240x240x3 (240x240 is the image size and 3 indicates the number of channels (RGB)) and that of labels[0] is 28800x8 (28800 is the number of images and 8 is the number of different classes possible in classification (0,1,2,3,4,5,6,7) (7 represents the none class)). The model is trained for 2 epochs. This process took about 5060 mins on my machine. This time may vary depending on the processing power available to you. The model is then saved to checkpoint the progress made and we can just load the model the next time we work with this, instead of training the model again.


Step-5: Testing the Model in real-time to count fingers Our model is finally ready to count the number of fingers in real-time. Using OpenCV we can use our webcam and have our model detect the number of fingers displayed. We draw a square on the right side of the screen. In every frame, we will extract this square and pass it to our model. Our model then returns an array of probabilities. Since there are 8 possible classes (0,1,2,3,4,5,6,7), each value in the array represents the probability that the image has a hand representing the number x at index x (except 7 where the probability value at index 7 represents the probability that there is no hand in the image). The value in the array that has the maximum probability is the value that the model predicts is being shown in the image. I displayed this value in the top left of the screen.


Step-6: My adaptation of Hand Cricket We now have a model that is capable of counting fingers. Now we move on to the technicalities of the game. This project was created to allow a single human to play the game with a bot. If 2 human players wanted to play hand cricket, they could do so in person, or over a video call. So the following program can be used to play only against a bot. To further ease it up, the human will always act as the batsman and the bot will always be the bowler. The bowler will randomly choose a number from 06 every second. The moment the batsman chooses a number (signals a number with his hand in the designated area), the number is added to his score. Since a new number is being chosen by the computer every frame, it is difficult to keep up with the bot. We just have to randomly choose a number, irregularly even, and hope that it does not match the number chosen by the bot (the bowler).


Step-7: Implementing the trained CNN model in real-time to simulate Hand Cricket If you made it this far, pat yourselves on the back because you have some serious dedication to learn. ( I am exhausted ) This is the final piece of code that combines the model that counts fingers (Lines 4449) with the rules of the game as described in the introductory section (Lines 5664). We once again use OpenCV to access the webcam because we need to implement this in real-time. There is a dedicated space on the right half of the screen for the user (batsman). The current score as well as the high score of the user is displayed on the screen (Lines 6780). Here's the game in action! (Ignore the Current and High Scores. Since every second the model detects a number chosen by me (the batsman), say 5, 5 points are added to my score (batsman's score). So every second 5 points will be added to my score until the bot randomly picks 5 as well which means Game Over for the Batsman (Me). When this happens, my current score is brought back to 0 and my game starts again) Thank you for sticking around till the end. I put a lot of effort into this project and I hope this was a learning experience for you as well. Until next time! Ciao!


Tesla's Self Driving Algorithm Explained

Version 1 At the start of autonomous driving development, the main objective was to let the car move forward within a single lane and keep a fixed distance from the car in front. At that time, all of the processing was done on an individual image level.


The Feature Extraction Flow or Neural Network Backbone The raw images of the car camera are processed by a residual neural network (RegNet) that extracts multiple blocks or feature layers in width (W) x height (H) x channel (C). The first feature output has a high resolution (160 x 120) and focuses on all the details in the image. Moving all the way to the top layer, with a low resolution (20 x 15) but with a greater channel count (512). Intuitively, you could visualize each channel as a different filter that activates certain parts of the feature map or image, for example, one channel puts more emphasis on edges and another on smoothing parts out. Therefore, the top layer puts more focus on the context by using a variety of channels. Whereas, the lower layer focuses more on the details and specifics by using a higher resolution. The multiple feature layers interact with each other via a BiFPN, a weighted bi-directional feature pyramid. For example, the first detailed layer thinks it sees a car but is not sure. At the top of the feature pyramid, the context layer provides feedback that the object is at the end of the road. So yes it is probably a car, but it is blurred out due to the distance.


Multi-Task Learning  The output features are not only used to predict cars but are used for multiple tasks such as traffic light detection, lane prediction, etc. Each specific task has its own decoder trunk or detection head, which is trained separately, but they all share the same neural network backbone or feature extraction mechanism. This architectural layout is called a HydraNet. So, why this HydraNet? Feature Sharing - It is more computationally efficient to run a single neural network backbone.Decouple Tasks - You can fine-tune tasks individually without affecting other predictions. This creates more robustness. Now that we understand how the network works for individual images, we can ask ourselves, how do we combine multiple images into a 3D space? After all, we want to have a thorough understanding of the surroundings, especially if the car is self-driving.


Stitching Images Tesla's engineer developed an occupancy tracker in order to stitch all the camera images together. Despite the effort, they quickly realized that tuning the occupancy tracker and all of its hyperparameters was extremely complicated. Lane detection looked great on the image but when you cast it out into a 3D space it looked terrible.


Multi-Scan Vector Space Prediction Instead of stitching the images explicitly by hand, you want them to be part of your neural network, which will be trained end-to-end. The image space predictions should be mapped on a raster with a top-down view. The first challenge is that the road surface could be sloping up or down and there could be occlusions due to cars passing by. If there is a car occluding a viewpoint you want to learn to pay attention to different parts of the image. To accomplish this, a transformer with multi-headed self-attention is used. Essentially how it works is that a raster is initialized with the size of the real-world output space (vector space). The yellow pixel, at a known position in that output space, will ask the neural network with the 8 cameras for information about that pixel - the query. The keys and the queries interact multiplicatively and then the values get pulled accordingly. After some processing, the output will be a set of multicam features about that pixel that can be used for predictions. To make this work, precise calibration and good engineering are key.


Rectify to a Common Virtual Camera If you build a car, the cameras will always be positioned in a slightly different way. Before you feed images into the neural network, you need to calibrate the camera first. The most elegant way to do this is to transform all of the images into a synthetic virtual camera using a rectification transform. This is a method of camera calibration that translates all of the images into a common virtual camera. The final result is a big performance improvement and better predictions coming directly out of the neural network. After we have the image prediction right, we still need to consider another dimension - time.


Adding Time to the Equation There are a large number of predictions that need time or video context. For example, if you want to identify whether a car is parked or traveling. If it is traveling, how fast is it going? If you want to predict the road geometry ahead, the road markings 50 meters ago can be helpful, etc.


Feature Queue Module Multiscale features are generated by the transformer module and are concatenated with kinematics, such as velocity and acceleration, as well as the tracking of the position of the car. These encodings are stored in a featured queue which is then consumed by a video module.


Time-based Queue The latest features are pushed to the queue within a specified timeframe of milliseconds. If a car gets temporarily obstructed, the neural network can look at the memory and reference it in time. Therefore, it can learn that something is occluded at a certain point in time.


Space-based Queue In order to make predictions about the road geometry ahead, it is necessary to read the line marks and road signs. Sometimes they occur a long time ago and if you only have a time-based queue you may forget the features while you're waiting behind a red light. In addition to a time-based queue, there is a space-based queue, and information is pushed every time the car travels a certain fixed distance.


Video Neural Network  The feature queue is consumed by a spatial recurrent neural network (RNN). The RNN keeps track of what's happening at any point in time and has the power to selectively read and write to the memory or cached queue. As the car is driving around only the parts that are near the car and where the car has visibility are updated. The result is a two-dimensional spatial map of the road with channels that track different aspects of the road.


Putting Everything Together First, the raw images are passed through the rectification layer, to correct for camera calibration. Then the images are passed to the residual network to process them into features at different scales and channels. The features are fused into multi-scale information through a transformer module to represent it in vector space. The output space is pushed into a featured queue in time or space that gets consumed by a video module like the spatial RNN. This video module then continues into the branching structure of the hydranet with trunks and heads for all the different prediction tasks. Thanks for reading this post on Tesla's autopilot. I hope it helped you to understand the general concepts of Tesla's autopilot. More details can be found in Jason Zhang's Medium blog post Deep Understanding Tesla FSD Series.


HydraSum: Disentangling Stylistic Features in Text Summarization... (Paper Review/Described)

What Problem Does This Paper Want to Solve? This paper focuses on the Text Summarization task and tries to give a sense of control over the predictions over the model's final user. They specifically came up with an approach to control both abstractiveness and specificity. It uses a multi-decoder architecture where each one will capture a stylistic feature of the dataset. The model uses a weighted average (called gate) of the decoders' output to generate summaries with a certain style.


The Contribution This paper explored the mentioned idea with a k-decoder architecture where the first m layers' parameters are shared between the k decoders where k=2 and m=8. (Fig. 1) There is also a gating mechanism (g) that basically is a weighted sum of the k decoders' output. The output of the mentioned shared layers will be passed to a feedforward layer + softmax to assign the weights. Now, let's look at how it works during Training and Inference.


Training They presented two approaches that studied different ways of setting the gate mechanism weights. (Fig.2 ) It is worth noting that the learning objective is to minimize the Cross-Entropy loss as usual. Unguided: Let the model decide the optimal weights during the training to reach the optimal contribution of each decoder to the final summary. We will have fixed weights g and 1-g optimized by the model.Guided: This method needs a bit more attention since the authors needed to handpick a number of samples (I couldn't find how many samples they used!) and assign the weight manually. For instance, we can choose samples where the summaries have a high number of common words with the articles if we want a decoder to learn copy/pasting from the input. Then, we will assign a higher weight to the first decoder's output to push the 1st decoder toward learning the mentioned feature.


Inference The authors experimented with sampling the output with different combinations. (Fig. 2) Individual Decoder: It means focusing on only one decoder only. Following the same copy/pasting example, the user can assign 1 to the first decoder (consequently, zero to the rest) to get summaries to enforce Extractiveness.Mixture: Let the model decide! Obviously, it is only accessible if we train the model using the unguided method.Manual Mixture: Let the user decide! The user can mix different styles to get a more desirable output. For example, if the first decoder learned the copy/pasting style and the second one learned paraphrasing. It's possible to put more emphasis on paraphrasing bypassing weights like [0.3, 0.7].


Results The authors did an excellent analysis to show different decoders will learn different styles. They compared the methods on metrics such as Abstractiveness, Specificity, Length, Readability, and Quality (ROUGE). I recommend you to read the "Results" section of the paper if you are interested since I will not cover all the points. The paper goes into detail about each column and what they mean. For example, It is visible that the first (D0) and second (D1) decoders are not different in capturing Abstractiveness in the XSum dataset. It is argued that the nature of the dataset can heavily impact this approach. Also, it is apparent that the mixture of decoders performs better than individuals because each decoder learns a style that is the best fit for some samples, not the whole dataset. You can see a sample of a generated summary using both individual decoders and a combination of them illustrated in Fig. 3. The first decoder (blue-colored) focuses on predicting short extravice (copy phrases from the article) summaries, while the second one can write longer abstractive (model uses its own words) output. The combination of these two decoders will generate a summary with both characteristics! The last interesting finding in the paper for me was the idea that they showed it is possible to train decoders independently and mix them during inference to generate text with different styles. There are additional experiments in the appendix with a 3-decoder architecture and different numbers of shared layers, namely 6 and 10. As well as the Human Evaluation process that favored the HydraSum summaries compared to baseline (BART).


Final Words, The authors used a simple idea to add control over the stylistic features of the generated summaries. I don't see this approach becoming the new standard in Text Summarization just because of the number of parameters we are adding to the network that will increase the memory usage! But, I believe it is a great starting point.


References [1] Goyal, T., Rajani, N. F., Liu, W., & Kryściński, W. (2021). HydraSum: Disentangling Stylistic Features in Text Summarization using Multi-Decoder Models. arXiv preprint arXiv:2110.04400.


Google Brain's New Model Imagen Is Even More Impressive Than Dall-E 2!

Watch the video If you thought Dall-e 2 had great results, wait until you see what this new model from Google Brain can do. Dalle-e is amazing but often lacks realism, and this is what the team attacked with this new model called Imagen. They share a lot of results on their project page as well as a benchmark, which they introduced for comparing text-to-image models, where they clearly outperform Dall-E 2, and previous image generation approaches. See the results in the video above! This benchmark is super cool as we see more and more text-to-image models, and it's pretty difficult to compare the results - unless we assume the results are really bad, which we often do. But this model, and dall-e 2, definitely defy these odds. tl;dr: it is a new text-to-image model that you can compare to Dalle-E 2 with more realism as per human testers. So just like Dall-E that I covered a month ago, this model takes text like "A golden Retriever dog wearing a blue checkered beret and red dotted turtle neck" and tries to generate a photorealistic image out of this weird sentence. The main point here is that Imagen can not only understand text, but it can also understand the images it generates since they are more realistic than all previous approaches. Of course, when I say understand, I mean its own kind of understanding that is different from ours. The model doesn't really understand the text or the image it generates. It definitely has some kind of knowledge about it, but it mainly understands how this particular kind of sentence with these objects should be represented this way using pixels on an image. But I'll concede that it sure looks like it understands what we send it when we see those results! Obviously, you can trick it with some really weird sentences that couldn't look realistic, like this one, but it sometimes beats your own imagination and just creates something amazing. Still, what's even more amazing is how it works using something I never discussed on the channel; a diffusion model. But before using this diffusion model, we first need to understand the text input. And this is also the main difference with Dall-e. They used a huge text model, similar to GPT-3, to understand the text as best as an AI system can. So instead of training a text model along with the image generation model, they simply use a big pre-trained model and freeze it so that it doesn't change during the training of the image generation model. From their study, this led to much better results, and it seemed like the model understood text better. So this text module is how the model understands text, and this understanding is represented in what we call encodings, which is what the model has been trained to do on huge datasets to transfer text inputs into a space of information that it can use and understand. Now, we need to use this transformed text data to generate the image, and, as I said, they used a diffusion model to achieve that. But what is a diffusion model? Diffusion models are generative models that convert random Gaussian noise like this into images by learning how to reverse gaussian noise iteratively. They are powerful models for super-resolution or other image-to-image translations and, in this case, use a modified U-Net architecture which I covered numerous times in previous videos, so I won't enter into the architectural details here. Basically, the model is trained to denoise an image from pure noise, which they orient using the text encodings and a technique called Classifier-free guidance which they say is essential for the quality of the results and clearly explained in their paper. I'll let you read it for more information on this technique with the link in the references below. So now we have a model able to take random Gaussian noise and our text encoding and denoise it with guidance from the text encodings to generate our image. But as you see in the figure showing the model above, it isn't as simple as it sounds. The image we just generated is a very small image as a bigger image would require much more computation and a much bigger model, which are not viable. Instead, we first generate a photorealistic image using the diffusion model we just discussed and then use other diffusion models to improve the quality of the image iteratively. I already covered super-resolution models in past videos, so I won't enter into the details here, but let's do a quick overview. Once again, we want to have noise and not an image, so we corrupt this initially generated low-resolution image with again some Gaussian noise, and we train our second diffusion model to take this modified image and improve it. Then, we repeat these two steps with another model but this time using just patches of the image to do the same upscaling ratio and stay computationally viable. And voilà! We end up with our photorealistic high-resolution image! Of course, this was just an overview of this exciting new model with really cool results. I definitely invite you to read their great paper for a deeper understanding of their approach and a detailed results analysis. And you, do you think the results are comparable to dall-e 2? Are they better or worse? I sure think it is dall-e's main competitor as of now. Let me know what you think of this new Google Brain publication and the explanation. I hope you enjoyed this article, and if you did, please take a second to leave a like under the video to support my work and follow the blog to stay up-to-date with exciting AI news! I will see you next week with another amazing paper!


References ►Watch the video: https://youtu.be/qhtYPhPWCsI►Paper: Saharia et al., 2022, Imagen - Google Brain, https://gweb-research-imagen.appspot.com/paper.pdf ►Project link: https://gweb-research-imagen.appspot.com/ ►My Newsletter (A new AI application explained weekly to your emails!): https://www.louisbouchard.ai/newsletter/


How Uber uses AI to serve you better

Watch the video How can Uber deliver food and always arrive on time or a few minutes before? How do they match riders to drivers so that you can *always* find a Uber? All that while also managing all the drivers. We will answer these questions in this article with their arrival time prediction algorithm: DeepETA. DeepETA is Uber's most advanced algorithm for estimating arrival times using deep learning. Used both for Uber and Uber Eats, DeepETA can magically organize everything in the background so that riders, drivers, and food are fluently going from point a to point b in time as efficiently as possible. Many different algorithms exist to estimate travel on such road networks, but I don't think any are as optimized as Uber's. Previous arrival time prediction tools, including Uber, were built with what we call shortest-path algorithms, which are not well suited for real-world predictions since they do not consider real-time signals. For several years Uber used XGBoost, a well-known gradient-boosted decision tree machine learning library. XGBoost is extremely powerful and used in many applications but was limited in Uber's case as the more it grew, the more latency it had. They wanted something faster, more accurate, and more general to be used for drivers, riders, and food delivery. All orthogonal challenges are complex to solve, even for machine learning or AI. Here comes DeepETA. A deep learning model that improved upon XGBoost for all of those. DeepETA is really powerful and efficient because it doesn't simply take data and generate a prediction. There's a whole pre-processing system to make this data more digestible for the model. This makes it much easier for the model as it can directly focus on optimized data with much less noise and far smaller inputs, a first step in optimizing for latency issues. This pre-processing module starts by taking map data and real-time traffic measurements to produce an initial estimated time of arrival for any new customer request. Then, the model takes in these transformed features with the spatial origin and destination, and the time of the request as a temporal feature. But it doesn't stop here. It also takes more information about real-time activities like traffic, weather, or even the nature of the request, like delivery or a rideshare pickup.All this extra information is necessary to improve the shortest-path algorithms we mentioned that are highly efficient but far from intelligent or real-world proof. And what does this model use as an architecture? You guessed it: a transformer! Are you surprised? I'm definitely not. And this directly answers the first challenge, which was to make the model more accurate than XGBoost. I've already covered transformers numerous times on my channel, so I won't go into how it works in this article, but I still wanted to highlight a few specific features for this one. First, you must be thinking "but transformers are huge and slow models; how can it be of lower latency than XGBoost?!" Well, you would be right. They've tried it, and it was too slow, so they made some changes. The change with the biggest impact was to instead use a linear transformer, which scales with the dimension of the input instead of the input's length. This means that if the input is long, transformers will be extremely slow, and this is often the case for them with as much information and routing data. Instead, it scales with dimensions, something they can control that is much smaller. Another great improvement in speed is the discretization of inputs. Meaning that they take continuous values and make them much easier to compute by clustering similar values together. Discretization is regularly used in production to speed up computation as the speed it gives clearly outweighs the error the duplicate values may bring. Now, there is only one challenge left to cover, and by far, the most interesting is how they made it more general. Here is the complete DeepETA model to answer this question. There is the earlier quantization of the data that are then embedded and sent to the linear transformer we just discussed. Then, we have the fully connected layer to make our prediction, and we add a final step to make our model general: the bias adjustment decoder. It will take the predictions and the "type features" we mentioned at the beginning of the article containing the reason the customer made a request to Uber to orient the prediction to a more appropriate value for the task. They periodically re-train and deploy their model using their own platform called Michelangelo, which I'd love to cover next if you are interested. If so, please let me know in the comments of the video or by DMs/email! And voilà! This is what Uber currently use in their system to deliver and give rides to everyone as efficiently as possible! Of course, this was only an overview, and they used more techniques to improve the architecture, which you can find out in their great blog post linked below if you are curious. I also just wanted to note that this was just an overview of their arrival time prediction algorithm, and I am in no way affiliated with Uber. I hope you enjoyed this week's article covering a model applied to the real world instead of a new research paper, and if so, please feel free to suggest any interesting applications or tools cover next. I'd love to read your ideas! Thank you for reading, and I will see you next week with another amazing paper!


How Uber uses AI to serve you better

References: ►Read the full article: https://www.louisbouchard.ai/uber-deepeta/ ►Uber blog post: https://eng.uber.com/deepeta-how-uber-predicts-arrival-times/ ►What are transformers: https://youtu.be/sMCHC7XFynM ►Linear Transformers: https://arxiv.org/pdf/2006.16236.pdf ►My Newsletter (A new AI application explained weekly to your emails!): https://www.louisbouchard.ai/newsletter/


A comprehensive cheat sheet on Tableau Charts: A Road to Tableau Desktop Specialist Certification

Table of Content Bar Chart Horizontal Bar Chart Stacked Bar Chart Side-by-Side Bar ChartLine Chart Continuous Line Chart Discrete Line Chart Dual Axis Chart Combined Axis Chart Dual Axis Chart vs Combined Axis ChartPie ChartArea Chart Continuous Area Chart Discrete Area ChartMaps Symbol Map Filled MapScatter PlotGantt ChartBubble ChartHistogramText Tables (Cross Tabs/Pivot Tables)Heat MapHighlight TableTreemapBox-and-Whisker PlotSample Certification Questions from this topic


Bar Chart A Bar chart is the most effective and easiest way to visualize data in Tableau. There are three types of Bar Charts:


Horizontal Bar Chart This is one of the most used charts in Tableau as it makes data ingestion and visualization as easy as possible. The chart clearly depicts the difference between various categories and hence is popular amongst data folks.


Stacked Bar Chart Stacked Bar Charts are an extended version of Horizontal Bar Charts. The motive is the same i.e. to show the difference between categories. But, Stacked Bar Charts are used when we want to further show the difference between sub-categories in categorical data. This increases the level of details in our viz.


Side-by-Side Bar Chart These are similar to stacked bar charts, the only difference is rather than categories being stacked over each other, the categories are spread like a nested bar chart i.e. bar chart in a bar chart. This makes the view even cleaner and easy to visualize.


Line Chart The line chart could be Continuous, Discrete, or Dual. When we create a Line chart, we get the "Path" option in our marks card.


Continuous Line Chart The continuous charts are useful when we try to depict a story of how things changed over time. We can add multiple categories in the view to show the difference between categories over time.


Discrete Line Chart It is as same as Continuous Line Chart, the only difference is it requires a discrete date rather than continuous dates and thus provides an even better level of details as we can split our dates into further categories such as quarters, months, etc.


Dual-Axis Chart This type of chart is useful when we want to compare the performance of two measures throughout continuous time. It is called dual-axis because we have two separate axes for two different measures, although we can synchronize both axes to create a better view, by right-clicking on any one axis and choosing "Synchronize Axis".


Combined Axis Chart If we synchronize and hide any axis of Dual-axis charts, we get a combined axis chart. The biggest advantage is we can add another dual-axis chart i.e. three measures in one view.


Dual Axis Chart vs Combined Axis Chart In the Combined Axis chart, both measures share the same axis. In the Dual Axis chart, both measures have a different axis.In the Combined Axis chart, only one mark is there. In the Dual Axis chart, multiple marks are created.In the Combined Axis chart, we can compare more than 2 measures. In the Dual Axis chart, we can only compare 2 measures.The combined Axis chart is also known as the Blended Axis Chart or Shared Axis Chart. The Dual Axis chart is also known as the Combination Chart.


Pie Chart It would be better if you don't use pie charts because they aren't really accurate. Look at the chart, all three slices apart from Basketball really look similar. Could you've differentiated if you weren't given exact data points? I am pretty sure not. In addition, if we had 20 different slices, it would've been a mess. So use Pie-charts only when you have a maximum of 6 slices and preferably in a percentage relationship. If not, you can always use bar charts. When we add a pie chart to the view, we get the "Angle" option in our marks card.


Area Chart There are two types of Area Charts:


Continuous Area Chart It is one of the most beautiful charts, it combines both line chart and bar chart together. The line shows the progress over years and the area shows the volume of the measure.


Discrete Area Chart It is similar to Continuous Area Chart but requires a discrete date rather than a continuous date.


Symbol Map It can be used to tell a story containing geographical data. To further enhance the view, we can play with size and colors. To increase the Level-of-Detail we can create a hierarchy in our dataset. We can also add custom shapes rather than circular dots. We can modify the map type to satellite, streets, outdoor, etc.


Filled Map The motive is as same as of Symbol map i.e. to employ geographical data. But rather than symbols we use colors. It looks more intuitive and appealing. We can modify the map type to satellite, streets, outdoor, etc.


Scatter Plot A Scatter plot is one of the best charts when we want to compare two different measures. Both axes contain two different measures, to add more functionality to our graph we can add a trend line to identify patterns amongst the data. We can change the shape of our data points from the marks card.


Gantt Chart The Gantt chart is great to compare the performance of any measure w.r.t. various categories for some time. It also makes a perfect project management tool.


Bubble Chart Bubble charts are one of the most appealing charts. It creates packed bubbles to use space efficiently. The size of bubbles depends on the measure, the higher the measure bigger the bubble.


Histogram A histogram helps to depict the distribution of data via frequency/count. Tableau automatically segregates the data into bins, we can also do it manually. This chart can be useful when we want to analyze our measures.


Text Tables(Cross Tabs/Pivot Tables) This is the most simple way to represent the data. This chart simply creates a spreadsheet containing dimensions and measures. This is one of the most boring graphs as it doesn't have any visual cue, but it does the work.


Heat Map It is an extension to Text-Table, it uses color and shape to enhance the view.


Highlight Table It is an extension to Text-Tables, it uses colored cells (similar to conditional formatting in Excel). The color gets darker as the value of the measure increases. This graph is more appealing than Text-Tables as it uses colors as cues. We can set the color scheme to either diverging or converging.


Treemaps Treemaps are ideal graphs for hierarchical data. The size of boxes depends on the measure, the graph is visually appealing as it used both size and color as its cues. The higher the measure, the bigger the boxes will be. We can change the color scheme to either converging or diverging. Treemaps only require marks pane and hence don't have any axes. But to increase the granularity we can add fields to the row/column shelf. A treemap requires size, color, and details.


Box-and-Whiskers Plot This is one of the most complex Tableau Charts. It compares the various categories as well as shows the distribution of each category. Upper Whisker and Lower Whisker denote Maximum and Minimum Value respectively. Upper Hinge and Lower Hinge denote Upper Quartile(75% of data points lie here) and Lower Quartile(25% of data points lie here) respectively. Median denotes the middle value of data when sorted in ascending/descending order.


Sample Certification Questions from this Topic Which chart is useful in identifying outliers?a. Paretto Chartb. Line Chartc. Area Chartd. Box Plot Solution: Box Plot


References: [1] Tableau Help  Tableau Software [2] Personal Notes [3]Tableau Desktop Specialist Exam (New Pattern - 2021) - Apisero


Thanks for Reading! Feel free to give claps so I know how helpful this post was for you, and share it on your social networks, this would be very helpful for me. If you like this article and want to learn more about Machine Learning, Data Science, Python, BI. Please consider subscribing to my newsletter: Find me on the Web: www.dakshtrehan.com Connect with me at LinkedIn: www.linkedin.com/in/dakshtrehan Read my Tech blogs: www.dakshtrehan.medium.com Connect with me at Instagram: www.instagram.com/_daksh_trehan_


Want to learn more? How is YouTube using AI to recommend videos?Detecting COVID-19 Using Deep LearningThe Inescapable AI Algorithm: TikTokGPT-3 Explained to a 5-year old.Tinder+AI: A perfect Matchmaking?An insider's guide to Cartoonization using Machine LearningHow Google made "Hum to Search?"One-line Magical code to perform EDA!Give me 5-minutes, I'll give you a DeepFake!


Calculations in Tableau: A Road to Tableau Desktop Specialist Certification

Tableau, Data Analytics, Business Intelligence, Data Science Welcome to the eleventh chapter, In this piece, we are going to learn about Calculations in


Table of Content Why use Calculation?Types of CalculationsEditing & Removing a CalculationSample Certification Questions from this Topic


Why use Calculations? Calculations are used when we want to add custom fields that are derived from generic fields in our dataset. In simple words, Calculations are used when we want our data to answer some indirect questions. Consider our database contains two rows: Name & Age. We can get our data to answer some indirect questions like if the person is above 18, mention "Eligible to vote". This is how calculations work, they're made to answer some indirect questions. When we create a new calculated field, we are creating a new field in our dataset. The type of calculated field depends upon the type of fields used in our calculations i.e. if we are creating a calculated field on dimensions, the new field would represent a dimension and the same goes for a measure. The dimensions that define how to group calculations are called partitioning fields. The dimensions upon which the table calculation is performed are called addressing fields. Some of the scenarios where Calculations might be helpful are: Converting data types of fields.Categorizing data.Filter Results.Aggregate Data.Calculating Ratios.Creating new data for visualization. A calculation, also known as a formula, includes some or all of these components: FieldsFunctionsOperatorsParametersComments


Creating a Calculated Field To create a calculated field, go to the data pane and click on the drop-down and choose "Create Calculated Field". As per the data type used, the calculation will be classified either as a dimension or measure.


Types of Calculations There are three types of calculations one can use in Tableau:


Basic Calculations A basic calculation allows users to transform data either on the data source level of detail or visualization level of detail. e.g. Splitting the strings(Data source level of detail), Aggregating columns(visualization level of detail)


Level of Detail(LOD) Calculations LOD calculations work on the same principle as Basic Calculation i.e. on data source level of detail or visualization level of detail but are more precise. There are three types of LOD Calculations: INCLUDE(more granular level)EXCLUDE(less granular level)FIXED(independent level) LOD Calculations aren't part of Tableau Desktop Specialist Certification so we aren't gonna dive deep into it.


Table Calculations A table calculation is only valid for fields in visualization. They don't consider any other field that is filtered out in visualization. These calculations could help create ranks, show running totals, percent of total, percentile, moving average, etc. There are two ways to create a Table Calculation: From Analysis From Column/Row Shelf Add some dimensions and measures on rows and column shelves. Right-click on the measure and choose "Quick Table Calculation" or "Add Table Calculation". Total table calculations supported by Tableau are: The scope of a table calculation could be: Table(Across) Table(Down) Table(Across then Down) Table(Down then Across) Pane(Down) Pane(Across then Down) Pane(Down then Across) Cell


Editing & Removing a Table Calculation To edit a Calculation, simply right-click and choose "Edit". To remove a calculation, right click on the measure in the view(on which calculation is applied) and choose "Clear Table Calculation".


Sample Certification Questions from this Topic Which function can return absolute value of a given number? a. Floorb. Ceilc. Roundd. Abs Solution: Abs


Want to learn more? How is YouTube using AI to recommend videos?Detecting COVID-19 Using Deep LearningThe Inescapable AI Algorithm: TikTokGPT-3 Explained to a 5-year-old.Tinder+AI: A perfect Matchmaking?An insider's guide to Cartoonization using Machine LearningHow Google made "Hum to Search?"One-line Magical code to perform EDA!Give me 5-minutes, I'll give you a DeepFake!


Sorting & Analytics Pane in Tableau: A Road to Tableau Desktop Specialist Certification

Tableau, Data Science, BI, Data Analytics Welcome to the tenth chapter, In this piece, we are going to learn about Sorting and Analytics Pane in


Table of Content Sorting Quick Sort(From Axis, Header, or Field Label) Authoring Sort(From Toolbar, Drag & Drop) Clearing SortAnalytics Pane Constant Line Average Line Median with Quartiles Box Plots Totals Average with 95% CL Median with 95% CL Trend Line Forecast Reference Line Reference Band Distribution BandSample Certification Questions from this Topic


Sorting Sorting is done on dimensions only. Nested Sorting -> A nested sort will assume each pane acts independently and sorts the row per pane. It is useful when we want to sort within a category of items. They don't convey the aggregated information about how the values compare overall.


Quick Sort From an Axis To sort the data from an axis, simply hover over the axis and choose the sort option. Sorting from an axis gives a nested sort by default. Observe the sort option near axis Earnings($ Million) and click on that to sort the view. Single-click to sort in ascending order, double-click to sort in descending order and triple-click to get the default order. From Header To sort the data by header, simply hover on the header and choose the sort option. From Field Label To sort from the field label, simply hover on the field label and choose the field on which you want to sort the view. Sorting from Field Label gives us non-nested sorting by default.


Authoring Sort From the toolbar We can either sort in ascending or descending order from the toolbar. Sorting from a Toolbar gives a nested sort by default By Dragging & Dropping To manually sort, we can drag the field and drop it anywhere we want. The heavy black line will indicate where the dragged field would be. Once we manually sort the data, the order won't change even if we refresh our data. Enabling any other type of sort (Field, alphabetic, or Nested) clears the manual sort we create. Specified Field Sort To sort the field by specified field, simply right-click on the field label. We can choose to sort in the following ways. Data Source Order - It will sort the data in the way, the data is originally sorted in our dataset.Alphabetic - It will alphabetically sort the fields, either A-Z or Z-A.Field - It will let you sort the data on basis of a particular field. The field doesn't need to be present in the view/viz.Manual - It will let you create a custom sorting order.


Clearing Sorts To clear the sorts, we have two options: Worksheet > Clear >Sorts 2. To clear specified sorts on the field, right-click on the field and choose "Clear Sort". We can also disable all the sorts, Worksheet>Show Sort Controls.


Analytics Pane We can add Analytics objects from Analytics Pane that could be availed from the view. Analytics Pane offers drag-and-drop functionality to add analytics objects such as box plots, constant lines, and average lines in the view. The scope of Analytics objects could be defined for either Table, pane, or a cell. Tableau offers the following Analytics objects: We can edit any Analytics object by right-clicking and choosing "Format".


Constant Line It lets you add a constant line to the viz. A constant line can be added for some specific measures, all measures, or only date dimensions. When we try to add a constant line, Tableau lets you enter a constant value.


Average Line Tableau lets you create an Average line either for the entire table, pane or cell. The average line could only be created for some specific measure or all measures.


Median with Quartiles A quartile segregates data into: Upper Quartile, Median, Lower Quartile. In Tableau, Median with Quartiles can be either added to Entire Table, Pane, or Cell. We can add Median with Quartiles either for some specific measures or all measures. The lower quartile is the region that contains 25% of total data points. The upper quartile contains 75% of the total data points. Median depicts the middle value of data points when sorted in ascending/descending order. The dark black line depicts the median and two shaded blocks representing Upper Quartile and Lower Quartile respectively.


Box Plot Box Plots are also known as Box-and-Whisker Plots. The scope of the box plot is always cell. Box plots are required to check if data have any outliers. The Box plot can be created for some specific measures or all measures. Box Plots can either be made through Analytics Pane or Show Me Pane. A box plot requires a Dimension on the Column shelf and a Measure on the Row shelf. Upper Whisker and Lower Whisker denote Maximum and Minimum Value respectively. Upper Hinge and Lower Hinge denote Upper Quartile(75% of data points lie here) and Lower Quartile(25% of data points lie here) respectively. Median denotes the middle value of data when sorted in ascending/descending order.


Totals This option allows us to add totals to the view, depending on the data points we can choose from Subtotals, Column Grand Totals, and Row Grand Totals. After choosing the required view, we get results like this.


Average with 95% CLI This adds an average line with a distribution band in the view. The distribution bands are configured to show a 95% confidence interval i.e. the average population will lie in this band 95% of the time. This could be either added for a specific measure or all the measures. The scope could be defined either for the table, pane, or a cell.


Median with 95% CLI This adds a median line with a distribution band in the view. The distribution bands are configured to show a 95% confidence interval i.e. the median population will lie in this band 95% of the time. This could be either added for a specific measure or all the measures. The scope could be defined either for the table, pane, or a cell.


Trend Lines To add a trend line, both row and column shelves must contain continuous data. A trend line requires 2 measures on opposing axes, or a date and a measure on opposing axes. We can add a trend line to the scatter plot. There are 5 types of trend lines in Tableau: Linear Trend Line Logarithmic Trend Line Exponential Trend Line Power Trend Line Polynomial Trend Line Right-click on the trend line to either edit or describe it.


Forecast Forecasting can be done for at least one date dimension and one measure in the view. Forecasting is only supported on Tableau Desktop and is not valid on multidimensional data. We can't use Forecasting in the view if it contains any of them: Table calculationsDisaggregated measuresPercent calculationsGrand Totals or SubtotalsDate values with aggregation set to Exact Date We can edit the forecast or ask for its description, by right-clicking on the forecasted line. When we increase our precision in forecasting, the range is increased.


Reference Line, Reference Band, Distribution Band, Box Plot We can create custom Reference Line, Reference bands, Distribution bands, and Box Plots. We can add a reference line to the scatter plot. A Reference Band can be based on two fixed points. A Reference Distribution plot can be along a continuous axis. The scope for Reference Line, Reference Band, Distribution Plot is either the Table, Pane, or a cell. The Box plot can only be defined on a cell.


Sample Certification Questions from this Topic Sorting from field label gives ______ sort by default.a. Nestedb. Non-Nestedc. Manuald. Data Source order Solution: Non-nested


References: [1] Tableau Help  Tableau Software [2] Personal Notes [3]Tableau Desktop Specialist Exam (New Pattern - 2021) - Apisero


Thanks for Reading! Feel free to give claps so I know how helpful this post was for you, and share it on your social networks, this would be very helpful for me. If you like this article and want to learn more about Machine Learning, Data Science, Python, BI. Please consider subscribing to my newsletter: Find me on Web: www.dakshtrehan.com Connect with me at LinkedIn: www.linkedin.com/in/dakshtrehan Read my Tech blogs: www.dakshtrehan.medium.com Connect with me at Instagram: www.instagram.com/_daksh_trehan_


Want to learn more? How is YouTube using AI to recommend videos?Detecting COVID-19 Using Deep LearningThe Inescapable AI Algorithm: TikTokGPT-3 Explained to a 5-year-old.Tinder+AI: A perfect Matchmaking?An insider's guide to Cartoonization using Machine LearningHow Google made "Hum to Search?"One-line Magical code to perform EDA!Give me 5-minutes, I'll give you a DeepFake!


Trends in AI - May 2022

Research Every month we analyze the most recent research literature and select a varied set of 10 papers you should know of. This time around covering topics such as Adversarial Attacks, Bias, Diffusion Models, Information Retrieval and much more.


Trends in AI - May 2022

1. Video Diffusion Models  Project page By Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet.  Why -> If you're following this blog series (as you should) you'll have heard of diffusion models. These models have been kicking ass for the past year; surpassing GANs as the coolest kid in image generation town, and recently being a key ingredient in the sauce that's made DALL·E 2 undeniably impressive. Well, now diffusion models upgrade from images to videos. Could this be the direction DALL·E 3 will set laser eyes on?  Key insights -> Diffusion is a generative modeling paradigm that allows you to generate data (image, sound, text) from a noise sample by iteratively reversing a diffusion process (i.e. adding differentiable noise at each step, this video contains a thorough explanation). This work is the first one to apply Diffusion Models successfully to generate videos. The architecture of the neural network is a standard U-NET¹ - a common architecture for modeling vision that combines convolutional and attention layers adapted to include the frames or temporal dimension. This is trained on short videos of only 16 frames and small resolution (64x64) and they use classifier-free guidance to achieve text-conditional generation. The process is very much analogous to that of diffusion models on images; but to extend the method to longer videos, the novel gradient conditioning method, which lets them interpolate or extrapolate new frames of the video as a conditional diffusion process. The resulting videos are not quite photorealistic, yet are still a substantial step forward: check them out yourself! Another relevant work in image generation is VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance, which would've been a top highlight if it weren't for DALLE·2⁷ stealing its thunder.


Trends in AI - May 2022

2. Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language  Project Website By Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence.  Why -> Unimodality is sometimes quoted as a weakness of generative Language Models (LMs from now on), there's only so much you can know about the world by looking at just text. But maybe it doesn't need to be trained on images to answer about images, maybe it can just interact with another model that has seen images.  Key insights -> This paper is about having foundation models² trained on different modalities (video, image, audio, text...) interact via a structured dialog to enhance the zero and few-shot multimodal capabilities of these. The motivation behind this idea is that models that learn from different data modalities will learn different and complementary information about the world which can be leveraged in a zero-shot setting, without the need of finetuning all the models together. The interaction happens in a dialog, hence the name of the framework "Socratic Models", and broadly speaking it goes a bit like follows. Imagine you want to answer complex questions about a video, a Visual Language Model (VLM) can generate captions for each frame of a video containing information like place and objects in the video. Then an LM can be prompted to generate a first-person (egocentric) summary of the scene and answer complex questions about it. This basic interaction through text can be extended to other modalities such as audio and images. A lot of the results in this paper are hard to compare apples to apples with existing methods because there aren't really any benchmarks yet designed to measure the capabilities of such a system. So this is more of a "hey this thing is possible" paper, not so much "this beats existing methods by a parge margin" paper. Still, this presents a potentially impactful direction of research, especially given the current trend of APIzation of Machine Learning. Deepmind also recently released  Flamingo: a Visual Language Model for Few-Shot Learning (along with an explainer blog post), which presents their zero and few-shot performance and would be a natural match for Socratic Models.


Trends in AI - May 2022

3. Planting Undetectable Backdoors in Machine Learning Models By Shafi Goldwasser, Michael P. Kim, Vinod Vaikuntanathan, and Or Zamir.  Why -> Learning ML algorithms from scratch can be hard, which is why outsourcing to a third party the implementation and training of a model is growing in popularity. What could go wrong?  Key insights -> This paper is quite theoretical in nature but the gist of it is quite simple. The authors prove that a third party can train and provide a classification model that performs undetectably well for regular data, but that has a computationally undetectable backdoor through which an adversary can add an undetectable perturbation to any input (a digital signature) to change the output to any desired misclassification. Imagine someone sells your bank a credit scoring model that outputs a binary prediction as to whether a loan should be granted or not. The bank hosts and runs the model, they can see everything it does. Still, they cannot detect it has a backdoor that lets any adversary applicant introduce a digital signature, which will ensure the output is positive. Looking at more mundane examples, even public pre-trained models which are extremely popular these days could contain undetectable backdoors enabling adversaries to manipulate inference! One can imagine that this also casts some doubt on out-of-the-box ad-hoc explainability methods, which could contain such backdoors as well making them untrustworthy or unreliable. The paper lays out the formal settings of the problem and investigates various levels of security, it's quite fascinating. It's still unclear to which degree all this is feasible; however, these kinds of fundamental security concerns for ML models need to be seriously taken into account as models grow in complexity and their intrinsic interpretability is.


Trends in AI - May 2022

4. The Distributed Information Bottleneck reveals the explanatory structure of complex systems  Code By Kieran A. Murphy and Dani S. Bassett.  Why -> The Information Bottleneck principle³ championed by Naftali Tishby is a powerful lens through which one can understand learning in Neural Networks (NNs). What can this perspective elucidate about complex data?  Key insights -> Representation learning can be understood from the lens of Information Theory. Broadly speaking, representations are subject to an inescapable tradeoff between accuracy and complexity; and learning objectives can be defined in terms of mutual information involving input, output, and latent representations. Tishby formalized this principle in the late 1990s and later applied it to improve our understanding of Deep NNs which is still a topic of debate today. At the core of this method is the idea that constraints are required to learn anything meaningful about data because to learn one needs to forget. Now when it comes to this paper, proposes to decompose complex interactions into various bottlenecks to more easily model complex interactions. It's interesting because this paper largely stems from a physics background so it's motivated by having models learn intelligible interactions an input presents. In practice, this means a bunch of VAEs⁴ running in parallel for different partitions of an input. The authors showcase how this method works in reverse engineering logical gates, decomposing images, and modeling condensed matter in physics. This is another case of a constraints-are-a-powerful-inductive-bias work, and as such it's not really expected to beat existing models in most in-domain benchmarks, yet it's still a fascinating direction for robust and interpretable learning.


Trends in AI - May 2022

5. Autoregressive Search Engines: Generating Substrings as Document Identifiers By Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Wen-tau Yih, Sebastian Riedel, and Fabio Petroni.  Why -> Language Models + Information Retrieval has been a hot topic from what we've seen in 2022. Conceptual barriers are being broken at the intersection of these two domains and the next generation of general-purpose search engines is emerging from it.  Key insights -> Recently, we discussed the Differentiable Search Index⁵ (DSI) paper in our blog series and in our podcast: having an LM directly output document ids given a query, which worked surprisingly well. In some sense, this paper builds a somewhat related system that leverages LMs for retrieval, overcoming some of the limitations that DSI presented such as scalability, comfortably establishing a new state-of-the-art performance on benchmarks like the KILT benchmark⁶. In a regular index, document features are precomputed and stored explicitly, like word counts or dense embeddings, such that given a query, a score can be computed as a proxy for passage relevance. In this case, an autoregressive text-to-text LM (aka GPT-like) learns distinctive n-grams for passages as their identifiers. At inference, given a query, the LM outputs the most likely n-grams in a constrained search, such that each n-gram gets a likelihood score that can be interpreted as a measure for passage relevance, resulting in a scalable retrieval. Well, there are actually quite a few more details to make this work, but it does work quite well achieving state-of-the-art on KILT. After a few years of Neural Information Retrieval based on either cross-encoders or dense retrieval, the possibility of departing from these paradigms is exciting!


Trends in AI - May 2022

6. What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? By Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel.  Why -> Recent history tells us that training large-scale LMs can yield transformative insights such as the emergence of impressive zero-shot performance in language tasks (e.g. GPT-3). But these insights are somewhat limited when only part of the picture is made public, which is where BigScience shines.  Key insights -> BigScience is a community-driven effort spearheaded by Huggingface for training a large-scale multilingual language model that is resulting in several publications detailing its design, iterations, and training from an open-source spirit so the community can learn from the nitty-gritty details of what it really takes to train a large LM to the tiniest detail. The insights from this line of research have been published in recent months and this is the latest edition investigating the question of choice of architecture + learning objective. The authors compare the combinations of architecture and training objectives: causal/non-causal decoder-only or encoder-decoder, and autoregressive vs. masked language modeling. TL;DR for zero and few-shot performance, the usual decoder-only autoregressive Language Modeling is the best choice of architecture and training objective.


Trends in AI - May 2022

7. OPT: Open Pre-trained Transformer Language Models  Code  Blog post By Susan Zhang, Stephen Roller, Naman Goyal et al.  Why -> F̶a̶c̶e̶b̶o̶o̶k̶ Meta fully open sources their version of GPT-3 ahead of BigScience, which is training a similar but multilingual model out in the open and is expected to be released sometime in the upcoming months.  Key insights -> To be honest, there's not that much to say about it that hasn't been said already, this is pretty much a GPT-3 clone with the big appeal that it's the biggest pre-trained Language Model to be made fully public. You'll find the most interesting tidbits by browsing around the GitHub repository that contains the code, and instructions to deploy, train and run this model, along with the detailed reports and behind the scenes of training such a model.


Trends in AI - May 2022

8. The Effects of Regularization and Data Augmentation are Class Dependent By Randall Balestriero, Leon Bottou, and Yann LeCun.  Why -> Aggregated performance metrics are useful because they let us compare how various models perform by looking at a single quantitative score such as average performance. While this practice is widespread and useful, it might obscure qualitative behavior that's highly relevant to the task at hand. In other words, two models with matching average performance might still present dramatically different failure modes, so it's important to develop a holistic understanding of how a model behaves especially if it'll be deployed and have an impact on tangible real-world decisions.  Key insights -> Bias in ML models is often assumed to be caused by the training data; however it's been shown that it can emerge as a consequence of other choices that might seem innocuous a priori such as the choice of loss/metric, pruning, data augmentation, etc. This paper shows how certain regularization techniques can hurt performance dramatically in a small group of classes while remaining almost undetectable at the aggregate performance metrics such as average accuracy. This paper explores this phenomenon by training an image classifier with different data augmentation techniques such as random cropping or color jittering affects per-class performance. For instance, heavy color jittering affects negatively the accuracy of detecting a basketball, while zebra detection remains intact; or random cropping is a good augmentation for "nematodes" but terrible for lions when overdone. After an in-depth empirical and theoretical analysis, the paper ends with a call for novel bias-free regularization methods. Results like these are tangible evidence of how easy it is to fool ourselves into thinking we understand how well an algorithm performs; a reminder that looking deeper than the surface-level metrics is essential particularly when models will be deployed out there. In addition, this paper is also a call for designing bias-free regularization methods.


Trends in AI - May 2022

9. A high-resolution canopy height model of the Earth By Nico Lang, Walter Jetz, Konrad Schindler, and Jan Dirk Wegner.  Why -> This paper is just kinda random and cool and interesting.  Key insights -> Canopy height is pretty much how tall vegetation is, which can be calculated as the elevation of the top of trees minus the elevation of the ground they're on. It turns out that canopy height throughout the world is a salient quantity to the global carbon cycle, so modeling it more accurately at a global scale is key to the study of environmental phenomena like deforestation, climate change, or biodiversity modeling. But accurately mapping how tall trees are all around the world is hard. The Global Ecosystem Dynamics Investigation (GEDI) is a mission designed to collect LiDAR data to measure vertical forest structures globally, but it's constrained to sparse measurements that only account for 4% of the earth's surface, and moreover, continuous measurement is not possible which makes studying evolution impossible. In contrast, optical satellite imagery such as that coming from Sentinel-2 is readily available. This work proposes to fuse these two data sources with Deep Learning to develop a probabilistic model of the earth's canopy height. Other recent works that will interest the environmentally-mindful folks are Machine Learning and Deep Learning - A review for Ecologists and The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink (Google is hopeful about their own future).


Trends in AI - May 2022

10. Datasets & Benchmarks Given the amount of interesting dataset and benchmark works this month, we've decided to group them into this TL;DR list Benchmarking Generalization via In-Context Instructions on 1,600+ Language Tasks, NATURAL-INSTRUCTIONS v2 -> Aims to measure the generalization of models to a variety of unseen tasks given a natural language instruction. Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality -> Like the Winograd schema challenge but for image-language pairs: ambiguous questions that require world knowledge to disambiguate. MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages -> A new diverse multilingual NLU benchmark.


Create Indonesian Recipe Generator by Fine-tuning T5, BART, and GPT-2

Repository and Demo I will provide the best model and also the code to train it. Feel free to try and download it :).  Huggingface Space (demo): Space Huggingface Model (download the model): Model Repository (to train the model): GitHub repository


Introduction In the past, I created an Indonesia recipe generator in my medium post by using a seq2seq Deep Learning approach such as Gated Recurrent Unit (GRU) and Transformers. I want to revisit my past work and improve it. So, this post will post some improvements from the previous one. The model in my previous post was raw and not pre-trained. So, it didn't have any prior knowledge to use in the training process. One of the improvements that can increase the quality of the model that I mentioned in my previous blog is using a pre-trained model. Many state-of-the-art research works implement it to improve the quality of the model from the non-pre-trained one. Moreover, currently, it's the era of the pre-trained model in Deep Learning. Nowadays, many pre-trained models appear and have outstanding results when the model has trained again (we call it fine-tune) on the target dataset. Therefore, it's intriguing to try it for my recipe generator project. In this experiment, I will use off-the-shelf publicly available pre-trained models. Since the data is in the Indonesian language, I need to use models pre-trained with the Indonesian data. They also need to handle sequence generation problems since the problem that I want to tackle is a text generation problem. I searched and only found T5, BART, and GPT models. Thus, I decided to experiment by using these models.


Data, Preprocessing, Exploration Data Analysis Since this is the continuation of my previous project, I use the same data that I used previously. Feel free to read more detail about it through my medium post below.


Method In this section, I will describe the models that I've tried and mentioned above. All of the models that I used are transformer-based models. I will briefly describe them.


Bart BART is a transformer-based model that is pre-trained through learning a corrupted input. It has an encoder-decoder architecture like the Transformer model with a few modifications such as the replacement of the activation function. There are several corrupted scenarios that the author of the BART tried. The released final system is a model which is trained through sentence shuffling and token masking. The idea is to make BART learn perturbation and have the capability of doing causal language modeling. To apply it to the system, I fine-tuned the model to the data. Here is an illustration of how pre-trained BART was built. In this experiment, I used IndoBART as the pre-trained model, which is released in this paper. The model is pre-trained through the Indo4B dataset, Common Crawl, and Wikipedia. The data has Indonesian, Sundanese, and Javanese languages. It is available publicly in the Huggingface model.


T5 T5 is also a transformer-based model that is pre-trained with a corrupted input. T5 is pre-trained by doing token masking. different from the BART which uses the data for causal language model training, T5 uses the training data as a seq2seq problem. The data may contain Translation, Question Answering, and Classification problems. The model is fed through learning those tasks with the addition of learning a corrupted input. Here is an illustration of how the model is pre-trained. I used the T5 model which is available in the HuggingFace. It is pre-trained by using the Indonesian mC4 dataset.


GPT GPT-2 is an auto-regressive pre-trained model that is pre-trained through causal language modeling that has no perturbation in the input (unlike BART and T5). Then, the pre-trained model is fine-tuned to our data. I used the IndoGPT model that is also released together with IndoBART in the same paper. The model is also pre-trained with the same data as IndoBART. Since the model is not encoder-decoder architecture, we need to reshape our input and make it a language modeling problem.


Setup I will split this section into code technical setup, model setup, and hyperparameter setup.


Code Technical Setup To make the training script, I used Pytorch as the deep learning framework. I wrap them with Pytorch Lightning6. I used the implementation of Model Checkpoint, Early Stopping, and 16-bit Precision from the Pytorch Lightning. For metric calculation, I used BLEU Score. BLEU Score is a popular metric for sequence-to-sequence problems. I use off-the-shelf BLEU score implementation from the sacrebleu Python package.


Model Setup I applied several modifications to the input of the model. For the architecture, I used an off-the-shelf implementation that Huggingface provided. For GPT, since it needs one input, I contacted the food name and recipe into an input with a special symbol >>>. T5 has seq2seq architecture, so I did a small modification to the input. From what I've read, T5 is pre-trained with a 'prompting' style of input. For example: input: summarize: <ARTICLE>. So, I follow it and change the data to become like that. Below is how I present the input-output of the model I didn't do any changes in the BART model, so I provide the input and the output as-is.


Hyperparameter Setup I used Adam as the optimizer technique. The learning rate varies depending on the architecture. I handpicked several learning-rate values based on several resources and I tried some of these values. I picked 1e-4, 1e-5, and 1e-4 as the learning rate of the model of GPT, BART, and T5 respectively. I used early stopping criteria to avoid model overfitting. It will stop training if the validation loss doesn't increase for 5 epochs. To pick the best model, I used the model that has the lowest validation loss. I used AdamW as the optimizer of the model. To make the training faster, I used Automatic Mixed Precision (AMP) that Pytorch provided. Unfortunately, T5 can't use AMP. So, I didn't use AMP when I fine-tuned the T5 model. Following my past article, to make a fair comparison, I used the Greedy decoder as the decoding strategy to predict the output for each of the models. You can see the details about how a greedy decoder works in my past blog.


Experiment Results Below is the result of my experiment. With my setup, IndoBART outperforms other models. T5, IndoBART, and IndoGPT have higher BLEU scores than the transformer vanilla. It indicates that a pre-trained seq2seq model may help to increase the performance of the model. All of the models that are trained on Indobenchmark's data outperform the model that is trained on the C4 model (T5). It's interesting to see the potential of each pre-trained model.


Analysis Please visit my blog for the best experience to see this section.