What is NoSQL? NoSQL means Not Only SQL but more than SQL. They are non-relational databases that do not follow a row-column format. Both structured and unstructured types of data can be stored. It is stored in the "key-value" format rather than the traditional row column. The tables are referred to as document objects in NoSQL databases. E.g., Redis, MongoDB, Firebase, Cassandra, CouchDB With the large amounts of data generated every millisecond in real-time, we need physical space as well as memory to support and manage the application and its data for further analysis.


Navigating the World of Data Engineering: A Beginner's Guide.

HDFS: Distributed file system for large data sets.A major component of Apache HadoopScales Hadoop clusters into hundreds of nodes.Follows the principle of "write once and read many times" More on HDFS: https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html


Apache Hive: A warehouse system that is fault-tolerant and distributed.Central store of information for making data-driven decisions.Used language similar to SQL to allow users to read, write and manage data in the form of queries. More on Hive: https://cwiki.apache.org/confluence/display/Hive/


Data Processing - Cleaning, manipulation, optimization The processing of data includes cleaning the data by removing the unnecessary data and keeping the user data as well as manipulating the data for optimization. The manipulation of data includes aggregation and joining methods in the data processing. The data engineer needs to understand the abstractions. In short, processing is a technique that converts raw data into meaningful information. The following can be used as a template for data engineers to process data.


Apache Spark Data processing framework that combines data and AI.Faster because it uses the least recently used (LRU) algorithm.Used for real-time and stream processing, feature selection, and building ML pipelines across programming languages like Python, Java, and Scala.Performs transformation and actions at large scale for ML and graph algorithms. More on Spark: https://spark.apache.org/docs/3.3.1/


Scheduling - Batch and streams Scheduling is a process of planning data processing jobs within specific intervals of time. It considers the time taken to process and complete the task for the next steps. It resolves the dependency requirements of jobs. Scheduling can be called the "glue" that holds the whole system together. Scheduling organizes and manages every job to work together in harmony. It gives the tasks a specific order and resolves the running dependencies. Scheduling can be done manually, at a specific time, or with a sensor. Sensor scheduling is a task that automatically runs after a specific condition is fulfilled. Manual: Updating the database table manually.Specific time: Updating the database table at 7 pm.Sensor: Updating the database table if a new entry was added or removed. It can be done in batches as well as in streams. In batches, a group of records is managed together at intervals and is cheaper while in streams individual records are sent in real-time immediately. Popular Tools: Apache Airflow, Oozie, Luigi


Parallel Computing Dealing with lots and lots of data every second, a lot of memory and power consumption takes place. As a solution to these problems, parallel computing comes into the picture. As the name suggests, the smaller subtasks are split and they work individually in parallel. These subtasks are distributed over different computer systems. These jobs work together to finish the task. The task needs to be large and must contain several processing units. The famous parallel computation framework is Apache Hadoop.


Apache Hadoop: Performs computation on large data sets across clusters.Used for batch processing.Written in Java.Handles failures in the application layer. More on Hadoop: https://hadoop.apache.org/


Cloud Computing and providers The term "cloud" refers to the data almost being stored in the air like a cloud. Technically, they are cloud services that allow us to use data computing services with maximum reliability. The physical servers are bought and need a huge amount of space along with electrical and maintenance costs. The constant supply of power is enough for the tasks and sometimes can go to waste. The servers on the cloud can be rented and do not require space. Many cloud providers follow a "pay-as-you-go" model where you only pay for the resources we need and which are used. It is reliable for data replication for databases. Tools: AWS, Azure, Google Data being the center of our everyday business lives, its collection, organization, and management are equally important to make our lives more efficient. Data engineering is vital in today's world for making future predictions, risk prevention and management, developing new marketing as well as business strategies, and many more applications. Now you know what data engineering is and what skills and tools you need to use to become a data engineer. FOLLOW US FOR THE SAME FUN TO LEARN DATA SCIENCE BLOGS AND ARTICLES: LINKEDIN: https://www.linkedin.com/company/dsmcs/ INSTAGRAM: https://www.instagram.com/datasciencemeetscybersecurity/?hl=en GITHUB: https://github.com/Vidhi1290 TWITTER: https://twitter.com/VidhiWaghela MEDIUM: https://medium.com/@datasciencemeetscybersecurity- WEBSITE: https://www.datasciencemeetscybersecurity.com/ - TEAM DATA SCIENCE MEETS CYBER SECURITY


N-Dimensional DICOM Volumes With ImageIO

What is a DICOM file? DICOM (Digital Imaging and Communications in Medicine) files are the standard format for human medical imaging. DICOM files also contain essential metadata with attributes like patient name, ID, and image pixel data. This format ensures that the medical image data can never be separated from a patient's metadata by mistake and also that we always have the information we need to properly display the images it contains. DICOM files with multiple image dataset elements are referred to as "volumes" and are common in a wide range of fields, including radiology, orthopedics, neurology, oncology, veterinary medicine, and more. DICOM files also support various modalities like CT scans, MRIs, and ultrasounds.


Reading a single DICOM file Imageio's imread function takes in a DICOM file and loads it as an image object, which is a type of NumPy array. Once we've converted the DICOM file to an imageio image object, we can plot it with matplotlib.pyplot, just as we would any other image array.


Reading DICOM metadata One of the most important characteristics of the DICOM file is its structure, which binds the image to its corresponding metadata. Imageio loads available DICOM metadata into a dictionary that is accessible through the meta attribute. You can also access specific metadata by indexing the meta dictionary with any of the available keys: We'll be taking a look into a few important pieces of metadata in the next section.


Reading a DICOM volume There are various ways to read a DICOM volume, but in this tutorial, we'll use ImageIO's volread() function. This function can load multidimensional datasets directly from a folder of images and aggregate the metadata accordingly. To plot high-dimensional data, all we have to do is slice it along the desired plane. Plotting slices sequentially can create a "fly-through" effect that helps you visualize the volume as a whole. Below we plot our chest CT scans sequentially:


Knee CT We can even make an interactive visualization that steps through the CT scan sequentially using ipywidgets. For this example, we'll use a volume of knee CT scans that is somewhat larger than the toy dataset we've been using so far. After using the interactive tool to walk through the full scan, we can isolate a few slices of interest for further evaluation. Let's say we think we see something interesting around slices 1520. We can display only those slices with the code below:


Slicing different planes DICOM image data can be 3D or even 4D, but how can we slice the image arrays to view different planes of this information? To answer this question, we'll first review a few helpful terms.


N-dimensional planes There are many ways to slice a 3D volume into 2D images, but when looking at human anatomy, the three main views are referred to as the axial, coronal, and sagittal planes. However, because most datasets don't have equal sampling rates across each dimensions, they also often don't have equal aspect ratios along each dimension. In these cases, we'll need to stretch the pixels to account for the differences. But how much should we stretch the pixels and in which directions? In order to calculate this, we'll have to dive back in to some of our DICOM metadata.


Attribute calculations Field of view is the amount of physical space covered by an image. We can calculate the FOV using two other properties, whose values are stored in our DICOM metadata. For this example we'll use a cranial CT scan. Follow along with the full code here, or download the data here. Array shape: the number of data elements on each axis (metadata key: shape), along each axis.Sampling resolution: the amount of physical space covered by each pixel (metadata key: sampling) along each axis. We can calculate the FOV along each axis by multiplying the shape and sampling attributes for each axis. Note that the FOV often varies across dimensions (as shown above). This means that the amount of space covered by our image is often not equal across planes, or, in other words, our FOV is often not a perfect cube. In order to slice different planes of our images, we also need to understand the aspect ratio. The pixel aspect ratio determines how much we need to stretch the pixels in each direction in order to get an appropriately scaled FOV. We can calculate the pixel aspect ratio using the same sampling attribute we used above, which is conveniently stored in the DICOM metadata. Notice that the field of view is different in the coronal and sagittal planes than it is in the axial plane, but the scale and proportions of the skull remain consistent. If we hadn't adjusted the aspect ratio, our coronal and sagittal views would appear stretched. Below we plot a sagittal image with the original unadjusted axial aspect ratio (left), and another image with the corrected sagittal aspect ratio (right).


Wrap up Thanks for making it all the way to the end of this article, and I hope you found it useful! Feel free to leave me a comment if you have any questions, and stay tuned for more content! Feel free to check out some of my other content below!


Identifying Traditional Nigerian Textiles using Artificial Intelligence on Android Devices ( Part 1 )

Approach Below are the steps I followed to build the model: Data collectionData cleaning and preparationModel architectureModel buildingMaking predictions In the following tutorial, I'll be showing you how to easily deploy the model on a Flutter Android app. So, let's move on.


Data collection When I decided to start this project, I couldn't find any online datasets that contained various Nigerian textiles and their labels. That meant only one thing, I have never done something like that before. At this point, I had two options. To drop the project and pick a simpler one that had an available datasetTo collect the dataset myself I decided to go on with the project since I loved the idea and wanted to see it work. I manually gathered Google images of the four classes, which are Akara, Aso oke, Atiku, and Lace. I downloaded 50 samples from each, but something unfortunate happened - all the images I collected got deleted! It happened as a result of a mistake in my script. You can imagine how I must have felt, but that didn't deter me. I adjusted the error, and I summoned up the courage to collect them again. I also had to make sure that I downloaded only good photos of the different textiles I was interested in. In total, I downloaded about 200 images. It's a minimal dataset, but it's a good way to start. I then uploaded the images to Google Drive because I wanted to use Google Collab to train the model.


Data preparation The first thing I did was import the necessary libraries. I need to mount the data since the dataset is on my Google Drive. What I did at this stage was to put my data into subdirectories of training and validation so that an image data generator would auto-label them for me.


Model architecture I used the transfer learning method to build the model. We use the method when we have a limited volume of data available. We can also call the pre-trained model the "base model". I made use of "MobileNetV2" as my base model because it allows you to build and run neural networks that can run in "low-compute" environments. It has been well-optimized to run on edge devices like phones, microcontrollers, microprocessors, etc., which suits my use case well. The mobile net model makes use of a depth-wise separable convolution which makes it suitable for low-compute environments. To learn more about the mobilenetv2 architecture, you can read the research paper here I used a combination of the base model and "fully connected" TensorFlow layers.


Model building To build the model, the first thing I did was download the pre-trained model and its weight using the Keras API. I then got the top layer's name because they are responsible for classification. When we begin training from these earlier layers, the model can be easily adapted to our use case. We discovered that the total number of layers of the mobile network is 152. I then decided to start my fine-tuning from the 120th layer. To achieve this, I had to make all the layers of the neural network trainable by the neural network by setting "layer. trainable" to true. I then freeze all layers before the 120th layer by looping through them and setting the "layer. trainable" to false. In building this model, I made use of the Keras functional API. The network contains a base model, Maxpool, Dropout, and Dense layers. The dropout layer does not allow the activation functions from the previous layers to rely on any feature, so it will have to spread out its weight, thereby helping in preventing overfitting. I then went on to instantiate my model, compile it and show the model's summary. To compile your model, you need to specify your loss function, your optimizing function, and your metrics. The loss function helps the neural network to calculate how far its predictions are from the target value on each training instance. We made use of the "categorical cross-entropy " loss from the Keras API because we are dealing with a multi-classification problem. If it were to be a binary classification problem, we are going to make use of the "binary cross-entropy". There are varieties of other loss functions you can use. To learn more about them, check this blog post which it is well explained.


Making Predictions It's time to make some predictions! To do this, we need to: Extract the image.Get predictions from the model.Get the class with the highest prediction.Visualize the output and see if it gets the image correctly. Here's the function to predict the image: Let's test the model with all the food classes and see how well it performs. We can see that the model got only one wrong out of four. The model couldn't really differentiate between Aso oke and Atiku textiles probably because they have a similar line pattern. The model can still be improved by adding more classes of different images.


Segformer: An Efficient Transformers Design for Semantic Segmentation

How do Segformers work? The design was first introduced by Enxie et al. in 2021. The goal is to have a simple, efficient, yet powerful semantic segmentation framework with Transformers and a lightweight MLP decoder. Some key points about Segformers: Not using positional encoding.Adapts an Efficient self-Attention mechanism from Pyramid Vision Transformers (PVT)A lightweight full-MLP decoder. For ease of understanding, I will go through the Segformers' architecture. First is the encoder, and then the decoder.


Encoder Segformers encoder outputs feature maps from 4 different scales. For each scale, the feature map's size is the input image's size reduced by k=4, 8, 16, and 32, and we get four feature maps of size (H / k, W / k, C) where C is the embedded dimension. This is because Transformers, due to the self-attention mechanism, tend to learn local features in early layers and global features in later layers. Then, Segformers use these outputs to utilize local and global information in segmentation. Each block of the Segformers is a sequence of smaller blocks that have: An Efficient self-attention layerA Mix-FFN (feed-forward network)An Overlapped Path merging layer.


Efficient Self-Attention (2nd key point) This is the key to avoiding the costly computation of Transformers on image data. Segformers adopt the "sequence reduction process" proposed by Wang et al. 2021. In the original self-attention mechanism of Vision Transformers, each head of the "multi-head attention" takes Q, K, V with the same dimension (N, C) where C is our embedding size, N = H * W is the length of the sequence. Then the following formula will have the complexity O(N²). The SRA (the efficient Self-Attention) reduces this complexity by projecting K and V to lower dimensionality with a reduction ratio R. As a result, the time complexity for two matrix multiplications is reduced by R times. Hence, the time complexity is O(N² / R). The reduction is implemented by a Reshape and a Linear projection on K and V: In practice, I find people using 2D Convolutional with kernel_size = stride = sqrt(R) on previous layers' outputs (H/k, W/k, C). This reduces each size by sqrt(R) and the sequence's size by R.


Mix-FFN (1st key point) NLP Transformers need "Positional Encoding" to improve their performances by showing them where they are. Segformers say: "Hey, it's something of a burden to me instead. And I'm going to remove it." Indeed, when Vision Transformers perform a task on image data, it receives "fixed positional information." It is fine when the image size is the same as the trained image data. "But what if it is bigger? Yes, the model will be lost". The 'positional information' needs to be interpolated to match the bigger image and hence reduce the model's accuracy. But still, location information is important. So Segformers introduce Mix-FFN to remove Positional Encoding but still have location information. Mix-FFN considers the effect of zero padding to leak location information, which is shown in the 4th section in How much position information do convolutional neural networks encode? Formulated Mix-FFN:


Overlapped Patch Merging Transformers and their Self-Attention mechanism are designed for NLP tasks. Its inputs are sequences of N-embedded vectors with embedded size C. Normal Vision Transformers need to divide images into separated patches to perform attention to images and then combine them again into feature maps. Enxie argues that using overlapped patches merging helps preserve local information around those patches. This will improve the model receptive field for better segmentation.


Full-MLP Decoder (3rd key point) Unlike other Vision Decoders using traditional CNNs, Segformer uses only MLP layers to avoid costly computation in generating segmentation maps. I don't have much to talk about the decoder. I suggest readers to read the receptive field analysis in the paper to understand the power of this decoder. According to the analysis, full-MLP has a larger receptive field by utilizing feature maps from four different scales. This results in better segmentations than traditional CNN-based decoders.


Conclusion To sum up, Segformer is an efficient design for semantic segmentation on large-resolution images. It avoids the costly computations of traditional Vision Transformers with an efficient self-attention mechanism and a lightweight full-MLP decoder. Also, the proposed Mix-FFN to replace Positional Encoding allows Segformer to maintain its performance on larger images when trained on smaller images. The overlapped patch merging is also a key feature to help Segformer improves its segmentations quality. The figure above shows how Segformers results compare to other State-of-the-arts. There are 6 configurations of Segformers from b0 to b5, where b0 is the most lightweight, and b5 has the best segmentation quality.


Thank you! Thank you for reading my blog. I hope this will be helpful. Some claps or comments will motivate me to write more and improve my writing. I also have a Kaggle notebook to finetune the Segformer for the downstream tasks with Hugging Face packages. (Link to notebook) Hope this helps you to train your own Segformers.


Shadow Deployment of ML Models With Amazon SageMaker

Shadow testing features  Run multiple model versions in parallel with one serving live traffic. Validate the new model version (shadow variant) without impact. Production variant accepts the request and serves the prediction response. Shadow variant will accept a user-defined portion of the requests. If required, we can capture the response of the shadow variant in S3, but it will not serve prediction responses to consumers. Once we are comfortable with the shadow variant, then we can promote that shadow variant as a Production variant and serve predictions from that model.


Shadow or challenger deployment use cases Evaluate the model performance metrics such as model latency, overhead latency, model errors, invocation 5xx errors, etc.Evaluate the model evaluation metrics such as accuracy, f1 score, precision, recall, MSE, etc.Evaluate the changes to model-serving infrastructure, like installing new dependencies, upgrading the model-serving framework and container versions, etc. If the new or updated model performs better than the existing production model wrt the above use cases, then we can promote a new model as a production model. In AWS re:Invent 2022, they demonstrated the capabilities of shadow testing by using the SageMaker console and SageMaker inference APIs for the first use case. We often improve the existing production models by collecting feedback from customers, retraining a model with new samples of data, or making use of model monitoring (model and data drift) components. So I would like to demonstrate how we can utilize the shadow testing feature in the second use case. In my previous post, I discussed how to fine-tune a HuggingFace BERT model on the disaster tweets classification dataset using on-demand and spot instances, deploy that model on a real-time endpoint, and update that endpoint. Let's deploy a HuggingFace BERT model as a Production variant that is fine-tuned with two epochs and consider a shadow variant that is fine-tuned with three epochs.


Deploy tweet-classifier-v2 model (as shadow variant) Let's deploy a tweet-classifier-v2 model as a shadow variant which is fine-tuned with three epochs. We capture the 100% request and the response of both production and shadow variants using Data Capture options by providing S3 URI. As per the settings below, SageMaker routes the 50% (InitialVariantWeight=0.5) copy of the inference requests received by the production variant to the shadow variant. Verify the production and shadow variants settings in the AWS console.


View production variant captured data from S3 Notice that SageMaker generates an eventId for each invocation which helps in analysis. Let's convert the production variant captured data into a pandas data frame.


View shadow variant captured data from S3 Notice that SageMaker generates an eventId and capture invocationSource as ShadowExperiment which helps in identifying the captured data from the shadow variant and production variant.


Compare the model evaluation metrics Captured data will help us in Model monitoring, like detecting data and model drifts. There are a lot of ways in which we can gain insights from captured data. Let's compare the probability scores from both production and shadow variants and check if the shadow variant results are better than the Production variant or not. We can observe that the shadow variant (BERT model fine-tuned with 3 epochs) results are better than the existing Production variant (BERT model fine-tuned with 2 epochs); hence we can promote the shadow variant as a production variant.


References Introducing Amazon SageMaker support for shadow testingMinimize the production impact of ML model updates with Amazon SageMaker shadow testing


Acknowledgments Special thanks to the AWS SageMaker team (Raghu Ramesha, Qiyun Zhao, Qingwei Li, and Tarun Sairam) for their contribution to introducing the SageMaker support for shadow testing.


A Brief History of Neural Nets

References Neural Networks: History https://cs.stanford.edu/people/eroberts/courses/soco/projects/neural-networks/History/history1.htmlMelanie Lefkowitz, "Professor's perceptron paved the way for AI - 60 years too soon", Cornell Chronichle (2019)Asifullah Khan, Anabia Sohail, Umme Zahoora, and Aqsa Saeed Qureshi, "A Survey of the Recent Architectures of Deep Convolutional Neural Networks", Artificial Intelligence Review (2020), DOI: https://doi.org/10.1007/s10462-020-09825-6Max Pechyonkin, "Understanding Hinton's Capsule Networks. Part I: Intuition", Medium (2017)Stable Diffusion Web Application: https://huggingface.co/spaces/stabilityai/stable-diffusionChatGPT web application: https://chat.openai.com/chat


Image Editing from Text Instructions: InstructPix2Pix

Watch the video We know that AI can generate images; now, let's edit them! This new model called InstructPix2Pix does exactly that; it edits an image following a text-based instruction given by the user.


References ►Brooks et al., 2022: InstructPix2Pix, https://arxiv.org/pdf/2211.09800.pdf► Website: https://www.timothybrooks.com/instruct-pix2pix/► Code: https://github.com/timothybrooks/instruct-pix2pix► Demo to try it free: https://huggingface.co/spaces/timbrooks/instruct-pix2pix


Trends in AI - 2023 Round-up

Community Twitter has long been the biggest online space where AI research people share and discuss their work publicly. But Elon Musk's infamous acquisition of the company has sent it onto shaky ground. Growing instability, unpredictable policy changes, and Musk's divisive political stances have produced an outcry to change over to other places such as Mastodon. For now, most of the action still remains on the bluebird site, and a full-on overnight exodus on political grounds remains unlikely, but we can't completely rule out the possibility of some sort of company meltdown within the next year. Another battlefront for the Deep Learning community has been that of frameworks. The adoption of PyTorch has been above TensorFlow for a couple of years now and is the most widely loved and used framework for Autograd and Neural Networks. In 2023, PyTorch v2.0 will be released, with a compiler and speedups as the headline feature. Not even Google is betting on a comeback of TensorFlow, and the JAX/FLAX ecosystem - already a favorite in Google Brain and many other researchers - is still not mature enough to go mainstream. Finally, the industry and academia divide continues to grow as (1) big tech has much more compute resources available and (2) much of the current blockbuster research relies on the close collaboration of dozens of world-class engineers, which are not available to the average PhD student. This means that academic research is shifting towards better examining, understanding, and extending existing models, and devising new benchmarks and theoretical advancements.


Trends in AI - 2023 Round-up

Language Models If LLMs were already the main characters in the AI landscape a year ago, the situation now is even closer to a monologue. Sutton's bitter lesson continues to age like fine wine. With ChatGPT, LLMs went mainstream - even my non-AI non-tech friends were asking about it - and we expect 2023 to be the year where this tech truly reaches the masses. Microsoft - already considering to expand its stake up to 49% of OpenAI - and Google won't want to miss out, so it'll be a clash of titans. Scale. Models have barely grown in terms of parameters in the past couple of years - contrary to much of the public AI discourse! The best existing available dense LLMs are still in the 200B parameter range because they were underoptimized and so much could still to be discovered and improved in that regime. However, we expect that to change this year with (1) Google making public use of their FLAN family of models and (2) OpenAI and contenders venturing into the trillion scale parameter count with the much anticipated GPT-4 if all optimization challenges are overcome. These might not power the bulk of LLMs as a Service because of costs but will become the next flagship tech to make the headlines.Optimization. The era of training on static text data is over. Current LLMs are not really just Large Language Models, their optimization includes curated, interactive, and continual data/text and formal languages like code. We expect developments in how LLMs are optimized that leverage more complex RL environments (e.g. LLMs as agents), LLMs trained within a formal environments loop to learn better symbol manipulation (e.g. Minerva v2), LLMs to generate more and increasingly better data to train themselves on, and methods to distill more performance on models that run cheaply on modest hardware, cause every FLOP counts in the economics of LLMs as a Service.Progress in Language Models will continue to trickle down to other AI fields like Computer Vision, Information Retrieval, and Reinforcement Learning (as has already happened in 2022).Code + LLMs. GitHub Copilot has been around for more than a year, and it's slowly changing the way people write code. Google shared earlier in 2022 that 3% of its code was already written by LLMs, and we expect code completion LLMs to get much better, slowly shifting how people write code.


Trends in AI - 2023 Round-up

Reinforcement Learning and Robotics Arguably, RL has not advanced substantially from a fundamental perspective in the last year. Instead, progress was constituted by the application of increasingly complex agents that combine Computer Vision, Text, Language Models, Data Curation... such as CICERO, Video PreTraining (VPT), MineDojo, or GATO. Largely driven by success with scaling up Imitation Learning or offline RL, with just a sprinkle of the good-old online RL agent-environment-reward loop. We expect the development of more multimodal complex agents that take actions under incomplete information, leveraging modular components based on large neural networks and large pretraining data. In 2023 we expect the symbiosis between LLMs and RL to grow even further: training LLMs in an RL setting, and using LLMs as part of RL agents (e.g. as planners for a policy, strong priors). Finally, Zero-few shot and extreme efficiency will be key for progress in robots that interact in the real world, and we expect the trend in ML modularization (the ability to just plug in pretrained modules), few-shot abilities, and causal representation learning to help in that regard in 2023. But we'd be surprised if there's a massive breakthrough in the space before extreme sample efficiency is solved in silico for traditional RL.


Trends in AI - 2023 Round-up

Computer Vision Diffusion Models and text-to-image were the stars of the show when it comes to 2022 CV. Our perception of what can be achieved by generating images is very different from what we thought a year ago. Yet, image understanding is far from solved. What are the keys that will get us closer? Causal Representation Learning (often related to Object-centric representation learning) is a growing field of interest that studies the learning of the causal relationships between elements beyond their statistical correlations. A key blocker for progress has been a lack of strong standardized benchmarking, and we expect 2023 to bring a shift in the benchmarking culture of CV, shifting focus to Out of Domain Generalization, Robustness, and efficiency, and away from in-domain image classification, tracking, segmentation...More multimodal models combining text, audio, and actions with vision like we've seen with Video Pretraining Transformer MineDojo.Diffusion Models have taken over generative text-to-image AI, and are being used for other applications such as molecule docking and drug design. Generative video and 3D scenes are and will be the next natural step for these applications, but we expect coherent long video generation to take longer. Modeling high-frequency data (tokens/images) is harder than gathering large-scale low-frequency data (e.g. novel-scale narrative structure). There's not enough static data to solve this problem by brute force, thus the need for better optimization techniques for large models.


Trends in AI - 2023 Round-up

Information Retrieval Finally, the topic close to our hearts. The biggest problem of Neural IR in the past couple of years has been to translate the success in academic benchmarks - where BM25 is routinely beaten - into real-world settings and widespread adoption. The keys for this to happen: No need human relevance annotations. This has already been one of the aspects where IR advanced the most in 2022, with proposals such as InPars (using LMs to generate annotations), LaPraDor (Unsupervised Contrastive Learning), and others.Convenience. Current models might perform well on benchmarks, but they don't just work. We expect advancements in convenience in the whole development lifecycle of neural IR models that will increase adoption.Conversational AI. Retrieval Augmented Language Models and powerful models like ChatGPT have recently revived interest in the space, as many now see real feasibility. While standardized evaluation remains challenging, we expect the interest in the space to grow. Besides research, 2023 could be a year of disruption in the space of consumer web search, and just a paradigm shift of what people come to expect from search engines. Microsoft's collaboration with OpenAI and the recent stellar success of ChatGPT has many speculating about the possibility of a 180° turn for Bing with the adoption of true complex web-scale question answering powered by language models that just work. Google now sees its main business challenged and this might be the year of disruption where Google needs to step up its game.


This AI newsletter is all you need #31

What happened this week in AI by Louis Last week has been a busy with new Generative AI developments, particularly with the announcement of ChatGPT's paid version, which sparked a conversation among users. At the beginning of the month, OpenAI shared a waitlist for users to join their Professional tier with perks on their Discord channel. The upgrade is not yet visible to all users, so it's likely that it will be released only to certain users or in specific locations. ChatGPT had over a million users within the first six days of its launch, proving that it is here to stay. While the monetization of ChatGPT was inevitable, it could be argued that it is too early to ask users to pay for it.


This AI newsletter is all you need #31

Announcing our new Video Livestream Content platform for AI content creators. After the success of our new livestream Events format in our 36,000-member Learn AI Together Discord community, we are very excited to announce our new open platform for AI video and audio content contributors. At Towards AI we have helped 2,000 different contributors publish blogs focussed on Artificial Intelligence and Machine Learning and to reach our audience of over 360,000 followers on social media. This has helped contributors grow their reputation, reach and audience in the AI community. We are now planning to do the same for livestream video contributors. Please reach out and pitch us your idea if you'd like to present any AI-related content to our community, whether it is a class, tutorial, paper, reading group, panel discussion, competition, etc. We would love to host it live on our Learn AI Discord. The content will be recorded, 100% owned by you and available for you to distribute on your own Youtube channel or website afterward. So if you are looking for an audience for your high-quality AI video (or audio) content it would be a perfect match. We will include in the Events calendar section of this newsletter (85,000 direct subscribers, 360,000 followers on socials) and include in the Events Calendar of our Learn AI Together Discord (36,000 members).


This AI newsletter is all you need #31

Hottest News Andrew Ng predicts the next 10 years in AIAndrew Ng, a prominent figure in AI, predicts the future of AI by taking a less radical approach. In this report, Andrew shares his understanding of what it takes to make AI work beyond big tech. He also discusses his "data-centric approach" to AI and how it relates to his work with Landing AI (a startup working on facilitating the adoption of AI in manufacturing) and the current state of AI.The Ethics of Artificial Intelligence: Examining the Implications of AI on SocietyArtificial intelligence (AI) is a rapidly growing field that develops computer systems that can perform tasks that require human intelligence, such as understanding natural language, recognizing images, and making decisions. As AI becomes more prevalent, it's important to consider the ethical implications of this technology, including how to ensure it is used for the benefit of humanity, how to address job displacement, and how to protect privacy.Meet Claude: Anthropic's Rival to ChatGPTAn AI startup called Anthropic, co-founded by former employees of OpenAI, has begun testing a new AI assistant named Claude, similar to ChatGPT. The article presents informal comparison findings between Claude and ChatGPT, showing that Claude is a formidable competitor to ChatGPT with improvements in several areas.AI and the future of work: 5 experts on what ChatGPT, DALL-E and other AI tools mean for artists and knowledge workersThe Conversation gathered insights from five AI researchers about the potential impact of large language models on artists and knowledge workers. The experts acknowledged that the technology is still developing and raises several issues including misinformation and plagiarism, which have an effect on human workers. Read the opinions of all the experts on AI, the future of jobs and the need for governance and credibility.Diversity, Equity and Inclusion in Artificial Intelligence - Let's Evolve the Narrative!Digital transformation has the potential to improve inclusion and equal opportunities for diverse communities, but there is a need to address gaps in representation and the risks of bias in AI development and evaluation, product design, marketing, access to AI and ML training, and the narrative on what a career in this field actually 'looks like'.


This AI newsletter is all you need #31

Three 5-minute reads/videos to keep you learning What's unsolved in generative AI?Much has already been written about the emerging generative AI technology and large language models (LLMs). While generative AI has taken the technology world by storm, other important aspects require further experimentation and testing. This article delves into the core terminology and explores unsolved elements in generative AI, such as prompting, governance, and computing.Curated Resources and Trustworthy Experts: The Key Ingredients for Finding Accurate Answers to Technical Questions in the FutureWith new tools like ChatGPT, the ability to distinguish between credible and unreliable sources remains challenging and crucial. This article discusses factors that prevent tools like ChatGPT from replacing traditional search engines and expert knowledge. It also examines the limitations of AI tools in generating responses for technical answers and predicts the necessary ingredients for finding answers in the future.This AI can clone your voice! VALL-E (explained)Last year, we witnessed the emergence of generative AI for both images and text. Within the first week of 2023, researchers have created a new system for audio data called VALL-E. It can imitate someone's voice with only a 3-second recording, achieving higher similarity and speech naturalness than ever before.Generative AI and long-term memory for LLMsIn this video, YouTuber James Briggs shares in-depth information about generative AI. He covers various aspects, such as the meaning of generative AI, the options for improving the performance of large language models (LLMs), the concept of long-term memory in LLMs, and the use of an OP stack for retrieval-augmented GQA.Caretta (GitHub Repo)Caretta is a lightweight, standalone tool that instantly generates a visual network map of the services running in a Kubernetes cluster. It utilizes eBPF to efficiently map all service network interactions and Grafana to query and visualize the collected data. Enjoy these papers and news summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #31

Upcoming Community Events The Learn AI Together Discord community hosts weekly AI seminars to help the community learn from industry experts, ask questions, and get a deeper insight into the latest research in AI. Join us for free, interactive video sessions hosted live on Discord weekly by attending our upcoming events. Towards AI x Whitebox Kaggle competitionTowards AI has partnered with a new FinTech startup, Whitebox, by Arav Kumar, a member of our community, to host an event where you can display your AI talent. In this upcoming Kaggle competition, you'll be able to enter your best stock trading strategies and our constituents will help you fund it.If you're interested, please sign up here and we'll reach out to you via email shortly. The competition will be hosted from February 1st - 8th on Kaggle and the winners can opt in to having their strategy connected to the Alpaca API and funded! Attend Whitebox's Q&A livestream on our discord this weekend to learn more about the competition! Ask your questions in our form.Date & Time: 28th January, 6 pm ESTThe Neural Network Architecture Seminar (#3)This is the third session of a (free) nine-part series on Neural Networks Architectures presented by Pablo Duboue (DrDub), covering Structure Learning Networks. Find the link to the seminar here or add it to your calendar here.Date & Time: 25th January, 11 pm ESTIf you missed the first part of the series, find last week's event recordings here.LAIT's Reading Group: The reasons why ML projects fail to deliver valueLearn AI Together's weekly reading group offers informative presentations and discussions on the latest developments in AI. It is a great (free) event to learn, ask questions, and interact with community members. Join the upcoming reading group discussion here.Date & Time: 28th January, 6 pm ESTIf you missed the last session, find last week's event recordings here.


This AI newsletter is all you need #31

Featured Community post from the Discord sparsh.g#3081 created a fun project that allows you to train your AI to play the Asteroids game using neural networks and a genetic algorithm that follows the principles of natural selection to evolve. The project allows you to save the best AI, load it later, or play the game yourself. Try it here and support a fellow community member. Share your feedback and experience in the thread linked here.


This AI newsletter is all you need #31

Article of the week How ChatGPT Helps You To Automate Machine Learning? by Gencay I. Machine Learning is an iterative process that helps developers & Data Scientists write an algorithm to make predictions and allow businesses or individuals to make decisions accordingly. ChatGPT, as many of you already know, is the ChatBot that will help humans avoid doing google research and find answers to their questions. This article shows the different uses of ChatGPT, which will help in Machine Learning.


This AI newsletter is all you need #31

Our must-read articles AI Anyone Can Understand: Part 10 - Naive Bayes by Andrew Austin In-Depth Tokenization Methods of 14 NLP libraries with Python Example by Amit Chauhan Bentoml vs. Fastapi: The Best ML Model Deployment Framework and Why It's Bentoml by BEXGBoost If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.


This AI newsletter is all you need #31

Job offers Senior Machine Learning Engineer @Tubi (Remote) Clinical Informatics and Data Science Intern @Freenome (Remote) Senior Engineering Manager - Data Intelligence @Chainalysis (Aarhus, London) Principal Technical Program Management, AI @Axon (Remote) Head of Data Science @Founders Factory (London/ Hybrid Remote) Senior Data Engineer @pulseData (Remote) Front-End Engineering Intern @percipient.ai (Santa Clara, USA) Interested in sharing a job opportunity here? Contact sponsors@towardsai.net. If you are preparing your next machine learning interview, don't hesitate to check out our leading interview preparation website, confetti!


How do the text2image Models Work?

Diffusions All the popular text2image models are diffusions. So, we will start with an explanation of what diffusion is. Diffusions in neural networks is a method of image generation. And we will start discussing this method in the context of generation without any conditions. Without text.The diffusion process in the context of image generation is two processes: the forward and the backward. The forward process corrupts the image; the backward process restores it. We will start with the forward process. It corrupts the image into white noise. And this is an important detail. As a result, we want to not just corrupt the image and make it uninterpretable, but we want to get a normal independent distribution of pixels. Why do we need this? To sample from this normal distribution. In other words, our forward process simplifies the distribution of images, and in the end, it becomes so simple that we can easily generate these "corrupted" images. We will corrupt the image according to the following formula: This formula means that at each step, we will multiply the image by a number less than one and add noise. Thus, after many steps, we will make the white noise from the image. To understand the process, I like the image below. It shows well how at each step of the process, we simplify the distribution of images to something very clear. Now for the most interesting part. We've generated a noisy image. It needs to be denoised. The denoising will be done by a trainable neural network. Here, we need to remember how we added noise: we multiplied the image by a constant and added noise. We know the constants. We know the result. Therefore, to restore the image, we need the noise. Specifically, we will predict the noise with the neural network. Now we need to choose the neural network. We will be predicting noise the same size as the image. For such tasks, the Unet is often used. And that is what we will be using. Unet is a symmetrical architecture. The image on input, image on output. In our case, the input will be not only the image but also the time. To add it, some use a small MLP to get embedding from the time, trigonometric transformations, or both. The resulting features are fed through multiplication or concatenation into the feature maps of the Unet. As a result, we get these algorithms. Here, t is the step number (time), x_i are images, \epsilon_i are noise, and \epsilon_{\theta} is our network. Pay attention to sampling. The sampling described above is basic and very clear. We literally do the whole process backward. However, sampling algorithms can be different. And they can run on top of a trained model without additional training. The idea is that we can "cut corners" during the reverse process. Do not take all the steps, but only part, and adjust the constants in the formulas. However, since we have agreed not to use math, we will not describe any math behind it.


How do the text2image Models Work?

Stable Diffusion As we have covered the general principles of diffusions, it's time to take a closer look at specific methods. We will be discussing Stable Diffusion, one of the most talked-about text2image models. This model is distinguished by its openness. There is an article, source code, and weights available for it. The most important thing, perhaps, is the weights because their availability has generated many interesting techniques over the already trained model. The first enhancement is that the diffusion process runs in the latent space. The thing is that Unet is a heavy architecture. Ьay is not in terms of the number of weights but in terms of the number of operations. It works in the original resolution of the input image and takes a lot of time. Especially if you consider that in the diffusion process, the network needs to be run many times. This is where the idea comes from - the diffusion process needs to be run not on images but on their representations in lower resolution. In other words, not in the space of images, but in the hidden, latent space. To convert the image into such a space, an autoencoder is used. This is a neural network of two subnetworks. The encoder compresses the image, and the decoder, which decompresses it back. The compressed image will be our latent representation. The second modification is the conditioning method. Specifically, how we will give the diffusion information about the text. The feature of Stable Diffusion is that the mechanism is modular, and information can be transferred not only about the text but also about anything. How does it work? In two stages. In the first stage, features are extracted from the condition using a pre-trained network. In the case of text, CLIP ViT-L/14 is used. The output is feature vectors for each text token. The second stage is the injection of these features into the diffusion process. This is done using a cross-attention mechanism. With its help, we will pass information about the text to the intermediate layers of the Unet. Cross-attention was invented for neural networks that work with text. In short, the idea is that as input to the cross-attention, Q, K, and V are given. In our case, Q, V are token embeddings from the text multiplied by trainable projecting matrices. K has also projected features, but this time from the Unet. What formally happens next is well seen in the picture. But in terms of interpretation, attention is responsible for calculating the attention mask (a tensor of numbers between 0 and 1) based on K and Q, and then multiplying V by this mask. These are the most striking details of Stable Diffusion. If you want more details, it is better to go to the paper, but we will stop here. As a result, we have an algorithm that can generate many meanings in high-quality images. But we will go further and see what else can be done with it. And it can be done a lot, and first of all, because of the modularity. Since stable diffusion consists of many parts, and modules, producing intermediate results, by replacing these results, you can control generation.


How do the text2image Models Work?

Prompt-to-Prompt Image Editing Let's start with the task of editing the input text. If you just edit the text and send it for generations, you never know what will come out. Most likely something completely new. But what if we want to get the same image but slightly modified? Modified exactly where the text has changed? To do this, let's remember how cross-attention works. Cross-attention creates masks between the features of the Unet and the embeddings of the tokens (words) in the request. If you look at these masks, it will unexpectedly turn out that they are meaningful and represent the object generated for the corresponding word. And they also change over time in a meaningful way. From a blurry mush spread all over the picture to a recognizable image. The idea is that for text editing, let's use the masks of the original text on the first reverse diffusion iterations and modified ones on the last. And this idea works. The later you start changing the masks from the original text, the better the original image will be preserved.


How do the text2image Models Work?

Dreambooth Now let's imagine that we want to generate an image of a person, not of any person, but of ourselves. And even in different styles. Unfortunately, the network doesn't know who you are, and there are no labeled photos of yourself in the datasets. So we will fine-tune it. We'll invent a new word 'w', gather photos of ourselves, and fine-tune the network to restore ourselves when the query is 'w person'. And then generate ourselves in different styles. Or our dog. Or anything.


How do the text2image Models Work?

Vision Decoding And finally, we will learn how to read minds. Or, more precisely, to restore what the subject sees from a fMRI image. fMRI is almost the same as MRI. A huge machine you lie in and wait while your brain is scanned. Like MRI, it measures oxygen, but it does it faster and at a lower resolution. The result is a 3D tensor, where the amount of oxygen is recorded in each voxel. We will compress this tensor with an autoencoder, put it in StableDifusion instead of text, and train.It will work with an accuracy of 22%, meaning that in 22% of cases, the network will generate the same object that the patient saw. And that's a lot for such a task! That's it. There is still a lot to talk about. How to generate music with the same technology, and how to make videos. But that's for another time.


3 Commands to Secure Your ML Models from Malicious Pickles

First of all, what is a pickle Basically, it is a Python object. I don't know much about pickle, so it's hard for me to explain what it is and how they could hurt our computer. I found some links to references and put them at the bottom of the page as references. You can check them out later. Before that, I asked ChatGPT to help me figure out what a "pickle" is in Python. Why does using pickle in the ML model contain security risks?


How to keep yourself safe from pickle The basic principle is that only download models from reliable sources. Hugging Face has a built-in pickle scan, which will show if the model contains a pickle or not. And the suggestion that was given in the documentation is Don't use pickle. I understand; you just can't help, right? I want to use the model that downloads from the internet but adds a bit more safety, so I searched several options for pickle scanning. When you use Automatic1111 WebUI, a safety check is built in. But DiffusionBee, which is a great, stable GUI app for Mac that makes it easy to start making AI art, doesn't have a pickle scan built-in, so I have to do it myself. There are several resources you can find online to help you detect pickles: Stable Diffusion Pickle ScannerStable-Diffusion-Pickle-Scanner-GUI And I really depend on GUI, so I like the second tool. Here's where the problem comes in. Although it is open-source and available on GitHub, it only includes a Windows.exe file. Thus, I can't simply download and execute it on my Mac. However, because the developer made it open-source, we might try to execute it. The only problem left now is - I know nothing about programming. Luckily, the magical ChatGPT might know. Let's ask if it has any clue about it? First, I got these explanations, which were good but didn't help me: Then I asked how to execute these files and got pretty decent answers. okay, it appears that glimmer of hope appears. I have conda but I don't know what the env_name is. Let's keep asking to see if ChatGPT could also help me with this, and it does.


Now we got all the pieces to run the application! According to the conda.yaml, the env_name is sdpsgui. Here are the most important 3 commands we need: I believe you can nail it by now, but if you are not familiar with a terminal like me, here are the steps, and we could do it together.


Let's give it a spin. First, we go to Stable-Diffusion-Pickle-Scanner-GUI and hit the code button to download the ZIP file. Unzip them, and open your terminal, go to where your unzip folder is. For example, mine is under /Download/Stable-Diffusion-Pickle-Scanner-GUI-0.1.6If you never use a terminal before, the way to go to your folder is to type cd + folder_name So I have to go to the download folder first: Then go to the folder where the Stable Diffusion Pickle Scanner GUI is: Once you are in the right place, your terminal might looks like this: and you can run the 3 important commands now (type the command after % and hit enter). First, we create a new conda environment: Then we activate the environment with the following: Finally, let's open the GUI with the following: TADA!We successfully open the app on Mac. If you think this is useful, please give the developers a GitHub star or buy them a coffee to thank them. https://github.com/diStyApps/Stable-Diffusion-Pickle-Scanner-GUIhttps://github.com/mmaitre314/picklescan


Reference Reddit article: Keep yourself safe when downloading models, Pickle malware scanner GUI for Stable DiffusionStable-Diffusion-Pickle-Scanner-GUIOriginal Pickle Scan Repo: https://github.com/mmaitre314/picklescan which you can also download it from pip install: https://pypi.org/project/picklescan/0.0.7/Stable Diffusion Pickle ScannerHugging Face Pickle ScanningPython official doc of pickle - Python object serializationEmbrace The Red: Machine Learning Attack Series: Backdooring Pickle FilesColdwaterQ: Backdooring Pickles: A decade only made things worseNever a dill moment: Exploiting machine learning pickle filesFickling on GitHubTensorFlow Remote Code Execution with Malicious Model


This AI newsletter is all you need #30

What happened this week in AI by Louis Last week, we successfully hosted not one, but two events directly on our Discord server - a first for us. The events were a great start to 2023, and we're excited to host many more in the future. To keep track of upcoming events, we've added a section on our Discord community where you can find our upcoming (free) events. More information is in our new event section below! I also wanted to mention that we would love it if you reached out to us if you'd like to host an event on our server. We'll share and host it for you, providing a live audience of AI enthusiasts, students, and professionals. You'll also have the freedom to record and use the content afterward.


This AI newsletter is all you need #30

Hottest News Microsoft eyes $10 billion bet on ChatGPTMicrosoft is reportedly in negotiations to invest $10 billion in the company behind ChatGPT, the highly popular app that has captivated both casual users and artificial intelligence experts since its latest software release. The funding, which would also involve other venture firms, would value OpenAI, the creator of ChatGPT, at $29 billion, including the new investment.Generative AI: a game-changer that society and industry need to be ready forGenerative AI is not a new concept and the machine-learning techniques that underlie it have progressed over the last decade. However, with the recent release of models such as Stable Diffusion and ChatGPT, generative AI has become a highly discussed topic among technologists, investors, policymakers, and society as a whole. Yet, as we continue to explore the potential of these systems, it is essential to address questions such as the risks involved, governance, and future implications to prepare for the emerging capabilities of generative AI.In the Reign of Generative AI, All Hail... the Fact-CheckerGenerative AI, whether it is used for content and images or simplifying our work through automation, is a technology that is here to stay. As we continue to experiment with it and push its boundaries, it is essential to keep in mind that machine learning is not perfect. This is not only evident in the occasional bizarre phrases or images it produces but also in its capacity to get facts incorrect. The role of fact-checking, which has been relatively insignificant for the past few decades, may become a vital practice in content creation in the years to come.


This AI newsletter is all you need #30

Three 5-minute reads/videos to keep you learning How to use ChatGPT to boost your writingChatGPT has become a viral sensation and has reached a million users. However, people are still experimenting with the best ways to utilize it to its full potential. This article explores the proper way to present a prompt, considerations to keep in mind when crafting a prompt, and techniques for getting the most out of ChatGPT.Introducing Simple ML for SheetsIn this Twitter thread, @Sumanth_077 presents a step-by-step guide on how to construct machine learning models directly within Google Sheets. The guide includes clear, visual instructions and references, introducing a simple approach to machine learning for sheets. It also highlights the capabilities of Sheets and how to achieve them with a few simple steps.Prompt Engineering 101: Introduction and resourcesAs we continue to experiment with new generative AI systems and strive to achieve optimal results, it is crucial to comprehend the impact that prompts can have on the outcome. This guide delves into the concept of a prompt, its fundamental components, and the process of prompt engineering. Enjoy these papers and news summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #30

Upcoming Community Events The Learn AI Together Discord community hosts weekly AI seminars to help the community learn from industry experts, ask questions, and get a deeper insight into the latest research in AI. Join us for free, interactive video sessions hosted live on Discord every week by attending our upcoming events. The Neural Network Architecture Seminar (#2)This is the second session of a (free) nine-part series on Neural Networks Architectures presented by Pablo Duboue (DrDub). Find the link to the seminar here or add it to your calendar here.Date & Time: January 17th, 11 pm ESTIf you missed the first part of the series, find last week's event recordings here.LAIT's Reading GroupLearn AI Together's weekly reading group offers informative presentations and discussions on the latest developments in AI. It is a great (free) event to learn, ask questions, and interact with community members. Join the next reading group discussion here.Date & Time: January 20th, 6 pm EST If you missed the last session, find last week's event recordings here. Add our Google calendar to see all our free AI events!


This AI newsletter is all you need #30

Featured Community post from the Discord jUMAD1 has released his second notebook, "Evaluate Shopper Clusters with clusteval" on Kaggle. This notebook delves into the use of the "clusteval" library, as well as dataset exploration, minimal preprocessing, evaluating created clusters using dendrogram and silhouette plots, and providing marketing suggestions based on profiling results and analysis. Give it a read here and support a fellow community member. Share your questions and feedback in the thread linked here.


This AI newsletter is all you need #30

Article of the week The Art and Science of Regularization in Machine Learning: A Comprehensive Guide by Data Science meets Cyber Security Regularization is a technique used to prevent overfitting in machine learning models. The author begins by addressing the fundamentals and then delves deeper into solutions with code and graphical examples. This is a comprehensive, one-stop guide for tuning the ideal ML model.


This AI newsletter is all you need #30

Our must-read articles Paper Review: InstructOR One Embedder, Any Task by Building Blocks Ultimate MLOps Learning Roadmap with Free Learning Resources In 2023 by Youssef Hosni If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.


This AI newsletter is all you need #30

Job offers Senior Machine Learning Scientist @Freenome (Remote) Domain Expert, Editor (Fixed-term contract) @Dataminr (Remote) Sr. Research Scientist - Computer Vision @Axon (Seattle, WA/Remote) Backend Software Engineer @Anderson Optimization (Remote) Machine Learning Engineer @Anduril (Sydney, Australia) Implementation Engineer @Gather AI (Pittsburgh, PA, USA)


LLMs Encode Clinical Knowledge: A Quick Review

Outline IntroductionContributionsLimitationsConclusionReferences


Introduction In the field of medicine, language is an enabler of key interactions for and between clinicians, researchers, and patients. This provides opportunities for leveraging LLMs for modeling properties of textual data in the medical domain. There is evidence that LLMs can act as implicit knowledge bases. The weights of these networks store information, resulting in information that is pliable and hence can be operated upon in a representation space. This phenomenon equips LLMs with an ability to form associations between stored information to produce meaningful insights. The unfortunate news is that this associative capability can lead to hallucinations as information stored in weights is unreliable. Hence, current AI models for applications to medicine and healthcare lack the ability to address significant gaps in effectively leveraging language as a tool for mediating real world clinical workflows. For instance, LLMs are found to have the potential to mirror misinformation,bias, and stereotypes in the corpus. With the profound advances made by LLMs, AI systems are undergoing innovative repurposing and helping address limitations posed by predominantly single-task AI systems.


Contribution Key contributions made in this study can be summarised across the following three axes; Dataset benchmarkFramework for human evaluationModeling Firstly, this paper introduces a dataset benchmark for medical question answering called MultiMedQA. This benchmark is a collection of six open-question-answering datasets - MedQA [jin2021disease], MedMCQA [pal2022medmcqa], PubMedQA [jin2019pubmedqa], LiveQA [abacha2017overview], MedicationQA [abacha2019bridging], and MMLU clinical topics [hendrycks2020measuring]). MedQA mirrors the US Medical License Exam (USMLE) style of questions. Furthermore, this benchmark is augmented by HealthSearchQA, the seventh dataset comprising curated commonly searched consumer health queries in English. Secondly, a robust human-centric evaluation framework is proposed to address some of the current limitations with automated metrics for assessing long-form answer generation, such as the bilingual evaluation understanding metric (BLEU). Clinicians and lay users (non-experts) are captured in the evaluation of the models' generative output. The clinician's evaluation is run along twelve different evaluation axes whilst lay users are evaluated along two unique axes. These include - how well the models' output agrees with scientific consensus; the possibility and the likelihood of harm; evidence of comprehension, reasoning, and retrieval ability; the presence of inappropriate, incorrect, or missing content; the possibility of bias in the answer; answer captures user intent and helpfulness of the answer. Thirdly, this paper highlighted how well LLMs encode clinical knowledge with major architectural modifications. The authors built on the Pathway Language Model (PaLM) and Flan-PaLM family of LLMs. PaLM is a transformer model architecture composed of a decoder-only setup, with key features such as - SwiGLU activation function to substitute for standard activation functions(ReLU, Swish, GeLU), parallel transformer layers, multi-query attention mechanism, rotary position embeddings (ROPE) as a substitute for absolute or relative position embeddings, shared-input output embeddings, zero bias kernels, and layer normalizations, and use of a SentencePiece vocabulary. Leveraging the PaLM baseline and instruction prompt tuning paradigms, the authors demonstrated the Flan-PaLM variant to gain superior performance across a suite of evaluation tasks over the baseline. Moreover, given the key limitations of domain adaptation methods and end-to-end finetuning of the model using copious amounts of in-domain data, the authors successfully investigated prompting and prompt-tuning to aid Flan-PaLM in adapting to the medical domain. The instruction prompt tuning technique designed in this study incorporates soft-prompt learned by prompt-tuning as an initial prefix that is shared across multiple medical datasets, followed by a task-specific human-engineered prompt alongside the original question and/or context.


Limitations Although promising, the dataset benchmark fails to cover multiple languages and excludes a larger variety of medical and scientific domains, hence partially reflecting real-world clinical workflows. Secondly, although Flan-PaLM was able to reach state-of-the-art performance on several medical questions and answering benchmarks, there are important gaps to be bridged in order for it to reach an expert clinician level on many clinically important axes. Important future directions proposed to help address this challenge include - establishing solid grounds for the models' responses in authoritative medical sources and accounting for the time-varying nature of medical consensus, the ability to effectively quantify and communicate uncertainty to a generic user-in-the-loop, and multilingual support for responses. Secondly, it is critical to conduct exhaustive work to improve the human evaluation framework. The pilot rating framework is not very exhaustive as it fails to capture important variations across diverse population groups. Also, the pool of clinicians and lay-users assessing the model responses is limited. Lastly, the evaluation framework failed to investigate the impact of variation in the clinician rater's medical specialty, demographics, and geography. Moreover, fairness and equity are underexplored in this study, especially the lacking understanding of how perturbations to demographic identifiers in prompts influence the model outputs. Also, the safety-critical and complex requirements of the medical domain pose an important question as to how the approach of sampling clinicians to participate in identifying the best-demonstration prompts examples and crafting few-shot prompts impacts the overall behavior of LLMs. In conclusion, I am fascinated by the performance of LLMs demonstrated in this rigorous study. Not only does it exemplify a successful application and evaluation of LLMs in the medical context, but also, it demonstrates exciting directions for future research and improvements. Thank you for reading :)


References [1] Large Language Models (LLMs) encode clinical knowledge [2] Galactica: A Large Language Model for Science [3] PaLM : Scaling Language Modeling with Pathways


Using ChatGPT to understand Microsoft's VALL-E - Neural Codec Language Model Paper

Motivation In my previous article, I discussed how ChatGPT could assist in designing your text prompt. In this article, I would like to share how the ChatGPT chatbot can be extremely useful when we read scientific papers. So let's read the Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers paper recently released on arxiv. I used ChatGPT to define field-specific terms such as phonemes and to provide a more thorough understanding of the current advancements in Zero-shot TTS (Text-to-Speech) synthesis. I must admit that it's quite fun to have such a highly intelligent personal assistant at my disposal.


Few definitions What is a Phoneme?


Few words about the paper The paper introduces a new neural audio codec model called VALL-E. A neural audio codec model is a machine learning model that uses neural networks to compress and decompress audio data. The goal is to reduce the bitrate of audio data while maintaining the quality of the sound as much as possible. These models typically use techniques such as lossy compression, which discards some of the audio data that is considered less important to the overall quality of the sound. The compressed audio can then be decompressed by a corresponding decoder model to reconstruct the original audio data. The models have lots of applications but are still in development. For instance, they can be used in speech synthesis and reduce the amount of data that needs to be stored on PCs, smartphones, etc. VALL-E was pre-trained using 60,000 hours of English speech and employs zero-shot prompting to generate natural speech with high-quality speaker similarity. VALL-E generates discrete audio codec codes based on both phoneme and acoustic prompts. VALL-E uses the follwing pipeline: It allows speech editing, zero-shot TTS, and content creation by working in conjunction with other AI generative models. It requires only 3 seconds of a person's voice to be able to learn the voice and use it to read the text prompt. It is impressive how VALL-E can maintain speaker emotion and the acoustic environment. For now, it works only in English. I invite you to check out their impressive demo. See below the overview of VALL-E. The paper list the key accomplishments as follows: VALL-E is the first TTS framework with strong in-context learning abilities similar to GPT-3. It allows for prompt-based zero-shot TTS without the need for additional structural engineering or pre-designed acoustic features and fine-tuning as in previous methods.Creation of a generalized TTS system in the speaker dimension by utilizing a large amount of semi-supervised data, demonstrating that the simple scaling up of semi-supervised data has been underestimated for TTS.VALL-E is able to produce diverse outputs with the same input text while preserving the acoustic environment and the speaker's emotion of the acoustic prompt.VALL-E generates natural speech with high speaker similarity through prompting in the zero-shot scenario. Evaluation results indicate that VALL-E outperforms the state-of-the-art zero-shot TTS system on LibriSpeech and VCTK. For a more in-depth understanding, I suggest reading the paper. Please note that the model is not publicly available at this time. However, I think that it is possible that similar models will be released on the Hugging Face platform by the end of the year. I hope that you enjoyed the article.


Thank you for reading! Don't forget to subscribe if you want to get my future stories in your inbox. If you enjoy reading my story and want to support me as a writer, consider signing up to become a Medium member and gain access to thousands of Data Engineering and Data Science articles. Find me on LinkedIn and Twitter! See my collection of MLops articles


Handling Mislabeled Tabular Data to Improve Your XGBoost Model

Why Data-Centric? This article highlights data-centric AI techniques (using cleanlab) to improve the accuracy of an XGBoost classifier (reducing prediction errors by 70% on the noisy dataset considered here!). These techniques optimize the dataset itself rather than altering the model's architecture or hyperparameters. As a result, it is possible to achieve further improvements in accuracy by fine-tuning the model with the newly enhanced data. Enhancements to the dataset are model-agnostic and therefore are transferable to other modeling and analytical endeavors, as opposed to being specific to a particular type of model. Previous research shows many popular datasets contain incorrect or mislabeled examples. Similar to other modalities, tabular data is also vulnerable to label noise. This data table is typically organized in a row, and column format, stored in a CSV file or SQL database, and can include numerical, boolean, and/or text values (such as sensor readings, bank loan details, car sales data, etc.). While errors in tabular data may not be as immediately apparent, it is crucial to address this issue in order to ensure the reliability and accuracy of machine learning models trained on this type of data. Because data-centric techniques focus on improvements to the data, in this article, we will not change any code related to model architecture, hyperparameters, or training! We will see a reduction in error strictly come from increasing the quality of our data which leaves room for additional optimizations on the modeling side.


Take a Snapshot At a high level, we will: Establish a baseline accuracy of XGBoost model trained on the original data.Use find_label_issues() to highlight hundreds of mislabeled data points.Remove the data with automatically-flagged label issues from the dataset, and then retrain the exact same XGBoost model. This simple step reduces the error in model predictions by 36%! The raw difference in accuracy values between the two XGBoost models is 8%.Manually correct the label issues of all examples found by find_label_issues(), which reduces the error in model predictions by 70% from the baseline, identical XGBoost model! To run the code demonstrated in this article, here's the full notebook.


Examine the Data You can download the dataset here. Let's take a look at our student grades tabular dataset. The data includes three exam scores (numerical features), a written note (categorical feature with missing values), and a (noisy) letter grade (categorical label). Our aim is to train a model to classify the grade for each student based on the other features, but 20% of the grade labels in this dataset are actually incorrect. We have access to the true letter grade each student should've received, which we use for evaluating both the underlying accuracy of model predictions and how well cleanlab detects which data are mislabeled. These true grades are only reserved for model evaluation and are manually validated, gold-standard labels. They are not present in any of the training procedures. We utilize a 75/25 split for our train/test data. In your noisily-labeled datasets, there will typically be no such ground truth, and therefore addressing label issues is even more important to facilitate proper model evaluation.


Train and Evaluate XGBoost Classifier Now that we've seen what can be achieved with data-centric AI techniques let's take a look at how we get there. For our model of choice, we will use XGBoost, an implementation of gradient-boosting decision trees (GBDT), which are commonly used with tabular data. If our tabular data consisted solely of numerical and boolean values, we could potentially utilize a simpler model such as a nearest-neighbor or logistic regression. However, our data includes a notes column, which we will treat as a categorical feature. Fortunately, XGBoost (>v1.6) is able to handle mixed data types (numerical and categorical) by setting the enable_categoricalparameter to true, thereby simplifying the modeling process. Using the default hyperparameters, our baseline XGBoost model demonstrates an accuracy of 79.2% when trained on the noisy labels and predicting the test set. It appears that the presence of 20% label noise is significantly disrupting the model's ability to accurately predict the labels on such a trivial task.


Find Label Issues In order to use cleanlab, we need to obtain out-of-sample predicted probabilities for all of our training data in order to provide the find_label_issues() method with the necessary input. Getting the predicted probabilities can be achieved through the use of our XGBClassifier model with cross-validation, which can be implemented easily using the cross_val_predictfunction from scikit-learn. In just a few lines of code, we get a list of possible label issues! A few of the top results are shown below. Let's take a look at a few of the label issues automatically identified in our dataset. Take a look at row 1, where the student got grades of 91, 89, and 81, which should result in a 'B' yet was accidentally labeled as an 'F'. In row 2, the student had great participation resulting in an addition of 10 points to the overall average, receiving exam grades of 90, 74, and 95 (averages to 86.3, overall 96.3 with the bonus), which should result in an 'A' yet was accidentally labeled as an 'F'. Note: find_label_issuesis able to determine that the given label is incorrect, without ever seeing the ground truth label letter_grade.


How'd We Do? Let's go a step further and see how find_label_issues()did at automatically identify which data points are mislabeled. If we take the intersection of the label errors identified by find_label_issues() and the true label errors, we see that we are able to identify 83% of the label errors correctly (based on predictions from a model that is only 79% accurate).


Retraining for a More Robust Model Now that we have the indices of potential label errors let's remove them from our data, retrain our model, and see what performance improvement we can gain. Keep in mind our baseline model from above, trained on the original data using the noisy_letter_grade as the prediction label, achieved an accuracy of 79%. Let's use a very simple method to handle these label errors and just drop them entirely from the data and retrain our exact same XGBClassifier. After removing the suspected label issues, our model's new accuracy is now 87%, which means we reduced the error-rate of the model by 36%. Note: throughout this entire process, we never changed any code related to model architecture/hyperparameters, training, or data preprocessing! This improvement is strictly coming from increasing the quality of our data which leaves room for additional optimizations on the modeling side.


Fixing the Label Errors Instead of just dropping the potential label issues, the smarter (yet more time-intensive) way to increase our data quality would be to correct the automatically-identified label issues by hand. This simultaneously removes a noisy data point and adds an accurate one. I reviewed the potential label errors identified by find_label_issues() and made adjustments to the labels as needed. We then re-train the exact same XGBoost model on the dataset improved by hand. After evaluation on the unmodified testing data, the resulting predictions achieve 94% accuracy, which is a 70% reduction in error from our original model and data. Our model trained on the dataset auto-filtered using the find_label_issues() method, this is a 52% reduction in error.


Conclusion For the student grades dataset, we found that simply dropping identified label errors and retraining the model resulted in a 36% reduction in prediction error on our classification problem (with accuracy improving from 79% to 87%). Going one step further, we manually reviewed and corrected the automatically-identified label issues, resulting in a 70% reduction in prediction error (with accuracy improving from 79% to 94%). By using open-source libraries for data-centric AI like cleanlab to ensure the integrity of your data, you can mitigate costly labeling errors and boost the performance of your models.


This AI newsletter is all you need #29

What happened this week in AI by Louis Our Learn AI Together Discord community has grown to 35,000 members and we are excited to see the engagement in our new AI Technical Questions forum format where members of our community and team are there to try to help with any AI questions, theory or ops. Building on this we have several exciting new features in the pipeline for the Community this year starting with Community Events. Given the success of the graduate seminar on Neural Networks Architectures that Pablo Duboue (DrDub) taught last year in Argentina, he has decided to reiterate the seminar this year, this time in English in a 9 part series through the Towards AI Discord server. We are excited to host it and hope you will join us and learn with us! Add our Google calendar to see all our free AI events! (More information in our community section below!) If you are interested in hosting an Event or live video seminar with us in our server - please reach out!


This AI newsletter is all you need #29

Hottest News Why ChatGPT is not a threat to Google SearchThe New York Times reports that Google's management has declared a "code red" in response to the potential disruption ChatGPT may cause. However, it may be premature to say that ChatGPT will surpass Google. The technology still has a long way to go, and even when it matures, Google Search may be well-positioned to benefit from language model technologies.2023 tech predictions: AI and machine learning will come into their own for securityAs AI technologies continue to advance, there are several predictions about their impact on security and its impact on the economy. Some security and technology professionals predict that artificial intelligence and machine learning will have mass applications for security and detection in the near future, while others expect new technologies and services to be budget-conscious.The Fourth Age Of ProgrammingThe emergence of software that can write, draw and speak as humans has often been seen as a sign that machines are about to revolutionize our society and completely alter the value and nature of labor. It is crucial to understand the current capabilities of machine learning code generation, the different eras of programming, and how it has evolved to this point.


This AI newsletter is all you need #29

Three 5-minute reads/videos to keep you learning Introduction to Graph Machine LearningThis is an introductory guide on the concept of graphs, their uses, and effective ways to represent them. It also covers the representation of graphs using machine learning techniques, such as graph neural networks and graph transformers, and provides additional learning resources on the topic.How to translate a website into different languages with GPT-3GPT-3 has demonstrated potential for use in a wide range of fields and with the right prompts and adequate preparation, it can be used to translate text into multiple languages. While manual tweaks and adjustments may be necessary to optimize performance, this article provides a step-by-step guide to streamline the process and achieve high-quality results.The End of ProgrammingIn this paper, Matt Welsh discusses the evolution of programming as a field. He argues that the traditional concept of "writing a program" is on the verge of obsolescence. In fact, he predicts that for all but highly specialized applications, most current software will be replaced by AI systems that are trained rather than programmed. The paper outlines the progression of this transition. Enjoy these papers and news summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #29

Neural Networks Architectures seminar series by Pablo Duboue (DrDub) Format: There will be 9 introductory lectures of about 30' to 40' of presentation followed by a discussion with the attendees. The lectures take place on Tuesdays at 8 pm California time. That is 11 pm New York time, 9:30 am India time. The first lecture is on January 10th and will then be held weekly. The recording of the lectures will be available as a YouTube playlist. The chronogram and content follow this (updated page coming soon). Target audience: Deep learning researchers and practitioners interested in improving their understanding of neural network architectures across the board. This is not an introductory course nor it deals with implementation. Follow the event and get notified when we are live: Some questions people might ask: - Is there any cost associated with this? No, but if you enjoy the lectures please present a paper yourself. The goal is to learn together. - There will be certificates awarded to attendees? No. - Will the lectures cover architecture so and so? Most probably not but take a look at the chronogram. Your best bet is to prepare a paper on that architecture for the seminar and we can discuss it all together. - Will coming to this seminar help me with "extremely precise needs arising from requirements at work or university"? Most certainly not. And in the off chance it does, you'll be sitting through tons of material that is not relevant. The discord server has very good question forums, your best bet is to post there. - Where can I find the event? It will be hosted on the server. Check out the Event feature at the top left of your screen! Any more questions ping @DrDub on the server.


This AI newsletter is all you need #29

Featured Community post from the Discord A new weekly reading group hosted by Nex#5992 and Big Bux Chungus#1729 only on Learn AI Together! Papers, approaches, libraries, good practices, and more will be presented weekly every Friday 6 pm EST! Follow the announcements on the #ai-reading-group-text channel and join the first event here. Please reach out in the #ai-reading-group-text channel if you'd like to present anything to the weekly reading group! (~30 min presentations)


This AI newsletter is all you need #29

Towards AI Article of the week How To Create an End-2-End Text Paraphrase App by Stavros Theocharis The author explains how to construct an app that can perform the function of paraphrasing in two distinct ways, each of which has its own advantages. The first approach is to make use of the FastAPI, while the second approach is to make use of Streamlit.


This AI newsletter is all you need #29

Other must-read Towards AI articles A Comprehensive Guide to Building Multi-lingual Neural Machine Translation using Keras by Fraol Batole YOLOV5(m): Implementation From Scratch With PyTorch by Alessandro Mondin Topic Modeling for E-commerce Reviews using BERTopic by Eugenia Anello If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.


This AI newsletter is all you need #29

Job offers Software Developer Intern @AnswerRocket (Remote) Backend Software Engineer @Anderson Optimization ( Full-time Remote) Software Engineering Intern @Inworld.AI (Vancouver, Canada) Software Engineer in Test @Welovesupermom Pte Ltd (Full-time Remote) Head of Support & Continuous Engineering @Covariant (USA) Senior Data Scientist @Narvar (Remote)


10 Underrated Software Patterns Every ML Engineer Should Know

#1. Factory Method Problem: When you start your new shiny application, you construct objects randomly across your code. But as you expected, it all became a huge mess. Also, your code is invaded by if statements and disjoint interfaces that are a horror to maintain. That's why you need a better way to build new objects. Solution: Using the factory method pattern, you can refactor all your creational code in a single method. By doing that, you eliminate duplication and force the code to use a single interface. Therefore, you can quickly swap different implementations of the interface through inheritance. ML use-case: From my experience, the factory method pattern is the most useful, common, and powerful SWE good practice I use in my ML projects. It is a simple concept used to construct your models, pipelines, and evaluators in a clean and maintainable way. Using this method, you eliminate duplication by having a centralized way to construct all these resources (we all know how messy the initialization of models can be). Thus based on your configuration file, you easily inject the objects of your choice.


#2. Abstract Factory Problem: Let's imagine your code supports a UI for both Windows and macOS. It is implemented by adding an if-else everywhere in the code where it constructs a UI element. Besides being repetitive and hard to maintain, if you want to add support for, let's say, Linux, it will be a nightmare. Solution: By using the abstract factory pattern, you refactor all the creational code for one family of classes (e.g., all the buttons, panels, and text for Windows or macOS) in a single class. The application will create the UI elements based on the abstract factory without knowing which concrete family it is. ML use-case: One of the most useful SWE good practices that can be followed in ML is the creational patterns. Frequently, ML models and data pipelines are complex to configure and tightly coupled. If you don't use the proper data pipeline for your model, your algorithm won't work. Thus, you can use an abstract factory to ship a family of objects, where one family consists of all the data processing steps, your models, your configurations, etc. This goes hand-in-hand with all the good practices followed in MLOps. Usually, your code starts with a factory method, and as it gets bigger, you refactor it to use an abstract factory.


#3. Strategy Problem: You designed a map that helps people give directions. The first iteration supports generating routes by car from point A to point B. Everything is working great, and the users love your app. Thus you decided to add support for pedestrians and bikes. But because your app is a monolith, you found it hard to incorporate another algorithm. Solution: The GUI side of the app remains the same regardless of the type of route (car, bike, walk). The only different thing is the algorithm that computes the path from A to B. Thus by using the Strategy pattern, you refactor every algorithm in its own class (with its own interface) and abstract everything else away. The rest of the application will use only an interface that returns a route. Ultimately, it will be straightforward for you to swap between different methods (strategies) or add new ones. ML use-case: Any ML application is packed with various algorithms, and it is valuable to know how to abstract these components. Thus, using the Strategy pattern in your system, you can easily swap between different models, feature engineering pipelines, evaluators, etc. Imagine how powerful this is when doing hyperparameter tuning. You can transform everything into a configurable hyperparameter.


#4. Template Method Problem: Similar to the issue presented in the Strategy pattern, you end with a monolith class that contains hard-to-maintain and duplicate code. For example, you have a reader class that supports CSV, Docx, and Excel formats. You extended the logic through if-else. Therefore, you ended up with a God class. Solution: Using the template method pattern, you can turn the monolithic class into a series of steps representing your "template" algorithms. Afterward, for every specific implementation, you subclass the template and code the particular implementation. In our reader example, the template would have a read(), process(), and save() method. Let's say that only the input may vary. Therefore, we can live the common part of the algorithm in the parent/template class and override only the read() method for our particular inputs. ML use case: Most often, you end up using different inputs, pipelines, encoding methods, and models. Therefore, with this pattern, you can easily reuse the overall structure of the algorithm and override in the subclasses only particular parts of the logic. This is one pattern that I find myself using often. Also, it goes hand in hand with the strategy pattern. Using those two patterns, you can write flexible classes that fit together like legos. Interesting note: The Factory Method is a particular case for the Template Method.


#5. State Problem: Often, you end up having a class polluted with massive conditionals that alter how the class behaves according to the current values of the class's fields. Such a code takes a lot of work to read and maintain. Solution: Extract all state-specific code into a set of distinct classes. Every state class will know what the next possible states are. As a result, you can add new states or change existing ones independently, reducing the maintenance cost. This pattern follows the logic of a finite-state machine. ML use case: As in most complex software systems, you have a context class that behaves differently depending on various scenarios. Unfortunately, we are not at the point where AI takes all the decisions into a piece of software. Usually, a system uses ML only on a subset of its most complex components. But, the holistic system is responsible for processing the model's input and output in specific ways. For example, we want to implement a chatbot, where for specific steps (states) of the conversation, we want the bot to behave differently (e.g., in the beginning, actually chatting with an AI bot, and after a few questions, the conversation is taken by a real human being, etc.).


6. Visitor Problem: You just finished your awesome mobile game. Unfortunately, you realized that you ended up with a God class with 50 methods for interpreting information from the game's dynamics and 100 methods for taking various actions based on that. Solution: You can quickly refactor the "interpreting information" and "taking actions" into Visitor classes. In this specific scenario, we will have an InterpretingInformationVisitor and a TakingActionsVisitor. Thus, you will remove most of the code from the main class and move it away into different hierarchies of classes. After, you can inject the visitor class into the main class, which will know what method to call from the visitor. ML use case: We all know how some data-related classes (e.g., the preprocessing pipeline) have 10k+ lines of code. That is incredibly hard to manage. Here the visitor pattern kicks in. It helps you decouple your main class from other hierarchies of classes that are specialized in specific tasks. Such as exporting data to different file formats, reading data from various sources, etc. In this manner, we avoid huge classes and class hierarchies through composition. In the end, your data class will focus on its leading role and delegate auxiliary behavior to its visitors. It results in flexible and readable code.


7. Mediator Problem: You built a virtual traffic manager that consists of cars, pedestrians, traffic lights, and bikes. Unfortunately, your manager asked you to add support for electric scooters. By the current design, all your classes depend on and interact with each other. Thus, you have to update all your classes to add this new class. Which is terrible. Solution: Using the Mediator pattern, we eliminate the dependencies between pedestrians, traffic lights, and others and let only a single class called the Mediator handle the interaction between them. Thus, all the classes will depend only on the Mediator class. Now you can easily add additional structures and update only the Mediator class. ML use case: Its role is to eliminate dependencies between various components. This pattern is essential in data pipelines. Where a specific transformation shouldn't depend on other transformations. Using Mediator, you can reuse the transformation like an atomic function, which is mediated by a Mediator class. The Scikit-Learn Pipeline object is an implementation of the Mediator pattern. The transformations aren't aware of other transformations. Only the pipeline object knows how to compose and call them in a specific order. Therefore, this assures reusability between the transformations.