How to Use Hugging Face Pipelines?

What is the Transformers library? Transformers is a library that provides APIs and tools to easily download and train state-of-the-art pre-trained models. You may ask what pre-trained models are. Let me explain. A pre-trained model is actually a saved pre-trained network that was previously trained on a large dataset. Using pre-trained models, you can save the time and resources needed to train a model from scratch. Nice, we looked at what the Transformers library is. Let's carry out some tasks to show how to use this library.


How to Use Hugging Face Pipelines?

Transformer Applications Transformers has great functions to handle various NLP tasks. The easiest way to tackle NLP tasks is to use the pipeline function. It connects a model with its necessary pre-processing and post-processing steps. This allows you to directly input any text and get an answer. To use Transformers, you need to install it with the following command: To show how to utilize the pipeline function, let's import it from transformers. Cool, we can now perform the NLP tasks with this object. Let's start with sentiment analysis.


How to Use Hugging Face Pipelines?

Sentiment Analysis Sentiment analysis is one of the most used NLP tasks. It is the process of detecting positive or negative sentiments in text. To show how to do this task, let's create a text. Awesome, we now have a text. Let's find out the sentiment of this text. To do this, first, we instantiate a pipeline by calling the pipeline function. Next, we give the name of the task we are interested in. Nice, we are ready to analyze our text using this object. As you can see, our pipeline predicted the label and showed the score. The label is positive, and the score is 0.99. It turns out that the model is very confident that the text has a positive sentiment. Great, we have finished our sentiment analysis. It is simple, right? Let's take a step back and think about what happened. This pipeline first selected a pretrained model that has been fine-tuned for sentiment analysis. Next, when creating the classifier object, the model was downloaded. Note that when passing some text to a pipeline, the text is preprocessed into a format the model can understand. In this analysis, we used a pipeline for sentiment analysis. You can also use it for other tasks. Some of the pipelines that have been developed recently are Sentiment-analysis; we just learned how to perform this pipeline, summarization, named entity recognition, question-answering, text generation, translation, feature extraction, zero-shot-classification, etc. Let's have a look at a few of these. The pipeline we're going to talk about now is zero-hit classification.


How to Use Hugging Face Pipelines?

Zero-Shot Classification Imagine you want to categorize unlabeled text. This is where the zero-shot classification pipeline comes in. It helps you label text. So, you don't have to depend on the labels of the pretrained model. Let's take a look at how to use this pipeline. First, we're going to instantiate by calling the pipeline function. Now let's create a text to classify. Let's define candidate labels. Cool, we created our text and labels. Now, let's predict the label of this sentence. To do this, we're going to use the classifier object. As you can see, the text is about education. Here we didn't fine-tune the model on our data. Our pipeline directly returned probability scores. This is why this pipeline is called zero-shot. Let's move on and take a look at the text generation task.


How to Use Hugging Face Pipelines?

Text Generation Tools like ChatGPT are great for generating text, but sometimes you might want to generate text about a topic. The goal of text generation is to generate meaningful sentences. Our model gets a prompt and auto-completes it. Let's see how to perform a pipeline. First, we instantiate the pipelines with text-generation. Let's go ahead and create a prompt. Now let's pass this prompt to our object. As you can see, a text was generated according to our sentence. Note that this text is randomly generated. So it's normal if you don't obtain the same results as here. In this example, we used the default model. You can also select a specific model from the hub. To find a suitable model for your task, go to the model Hub and click on the corresponding tag on the left. Here you can see the supported models for your task. Cool, let's try a model. First, we're going to create a pipeline. Let's pass our task and model name to it. Cool, we instantiated an object. Let's create a text with a maximum length of 30 using our previous prompt. As you can see, a text was created with the model we determined. Let's go ahead and take a look at the named entity recognition task.


How to Use Hugging Face Pipelines?

Named Entity Recognition (NER) NER is one of the most popular data preprocessing tasks. In NLP, real-world objects like products, places, and people are called named entities, and extracting them from text is called named entity recognition. Let's show how this task is done with an example. First, let's create an object from the pipeline. Here we passed grouped_entities=True to regroup together the parts of the sentence. For example, we would like to group "Google" and "Cloud" as one organization. Now let's create an example sentence. Now, let's pass this text to our object. As you can see, our model correctly identified the entities in our text. Nice, let's move on and the question-answering task.


How to Use Hugging Face Pipelines?

Question-Answering In question-answering, we give the model a piece of text called context and a question. The model answers the question according to the text. Let's illustrate this with an example. First, let's create an object from the question-answering pipeline. Now let's use this object. As you can see, our pipeline extracted information from the context. Cool, we learned how to perform various NLP tasks with the pipeline. You can also use pipeline for other tasks, such as summarization and translation.


How to Use Hugging Face Pipelines?

Wrap-Up Transformers is a library in Hugging Face that provides APIs and tools. You can perform NLP tasks using this library. The easiest way to do this is to use Hugging Face pipelines. Pipelines provide an easy-to-use API that connects a model with its necessary pre-processing and post-processing steps. So, you can easily carry out various NLP tasks using pipeline objects. That's it. Thanks for reading. I hope you enjoy it. Let me know if you have any questions. Follow me on Medium, so you don't miss the latest content. We also create content about AI on other social media platforms. Don't forget to follow us on YouTube  Twitter  Instagram


How to Use Hugging Face Pipelines?

Resources Hugging Face DocumentationNatural Language Processing with Transformers


Galactica: AI Of Science By Science For Science

Architecture Similar to GPT, this architecture has casual language modeling and generates one token at a time. Therefore, it is a Transformer architecture in a decoder-only setup. Some additional plugins added to this existing architecture are: GeLU ActivationContext WindowNo BiasesLearned Positional EmbeddingsVocabulary You may refer to the paper for details.


Dataset We have heard people saying, "Nothing's best than learning from mother nature itself". For a science-loving model that took this saying to heart, it took a massive corpus of 106 billion tokens from various scientific sources. It was a combined dataset ranging from NLP, citations, step-by-step reasoning, and mathematics to protein structure, chemical names, and formulae. They called it the NatureBook. It has not been released publically.


Proposed Capabilities Galactica came up with great hopes for the scientific community, although it couldn't keep up for long. Although I couldn't collect some detailed information by using the demo, which is not available now, here is what the creators proposed that could make Galactica different from its counterparts like OPT, GPT-3, and BLOOM: Mathematical Reasoning In terms of mathematical understanding and computation logic, Galactica is highly capable compared to its counterparts. It can translate whole mathematical equations to simple English explanations and simplify huge & complex mathematical equations. Galactica performs strongly compared to larger base models on this task. When coming to solving numerical problems in physics, Galactica could do it to a great extent, even generating Python code for a calculation that it cannot solve reliably in a forward pass, using a working memory token. A smaller demo in HuggingFace gave the following result: 2. Chemical Understanding Galactica stands distinguished at IUPAC name predictions and identifies functional groups easily (I really wanted that in high school). This was achieved by self-supervised learning. It can annotate protein sequences and generate molecule structures using chemical names. It achieved remarkable results in drug discovery tasks. It can understand and work on SMILES (Simplified Molecular Input Line Entry System) formulae with appropriate reference tokens like [START_I_SMILES] and [START_AMINO] in the prompts, as shown below. 3. Predict Citations and LateX Along with writing a publication, it is important to cite sources too. The ability to predict citations is an ultimate test for "organizing science". Galactica was trained using three datasets to achieve this- PWC Citations, Extended Citations and Contextual Citations. It was framed more like a text-generation task using the reference token [START_REF] for the prompt. Navigating thousands of sources of literature like this is hard, and Galactica outperformed all other programs as expected. Same for the case of LaTeX equations, it outperformed larger language models BLOOM and GPT-3 by a large margin, with a better ability to memorize and recall equations to solve problems. 4. Programming Languages and Code While Galactica might not be up to ChatGPT's standards, it has a good grasp of how to explain and create code. Here is a peek into language-to-language translation and code-to-human language conversion from the demo on the website: 5. Text Generation and Writings This was one of the capabilities that gave the initial hype to Galactica. This article's cover picture also draws inspiration from it!! It could author research journals and other literature, complete with citations and appropriate paragraph divisions. Meta Chief AI Scientist Yann LeCunn put it like this- " This tool is to paper writing as driving assistance is to driving. It won't write papers automatically for you, but it will greatly reduce your cognitive load while you write them". It is astonishing when we realize that Galactica helped write its own research paper. A public announcement of one's existence, if we could exaggerate. Borrowing exact words from the paper itself, it goes like this.


What happened later? Galactica came with a lot of promises. But to put it straight, it didn't live up to the expectations. Galactica demo had to be closed within three days of public release due to misuse, as they put it. The notable problem with this new LLM(Large Language Model) by Meta was that it could not differentiate fact and fiction. Criticisms arose on Galactica, calling it "wrong", "erroneous", "prejudiced," and "statistical nonsense". Check out the Twitter thread by Michael Black, Director of Max Planck Institute for Intelligent Systems. Some demos are available as HuggingFace spaces (they may not be the official ones). Although the demo was recalled, the website is still open, and the model has been published as open source, welcoming all sorts of improvements and works over it. As that seed could grow into a tree of state-of-the-art models, providing it to the open-source community was a wise move.


Conclusion The road to superintelligent systems may not be an easy one. As far as we are concerned right now, counting on Moore's Law is still a good bet. Meta AI didn't actually fail, but their work triggered extensive research on LLMs. We cannot deny the fact that LLMs are the future of search engines, let Google deny it or deal with it. Microsoft has already taken the first steps to boost its search engine Bing with a massive investment in OpenAI. Those who have used ChatGPT must have realized that instead of getting the information we want from a store-and-retrieve system, we are in need of something that can reason and think with us. Something that could combine diverse sources and present us with a logical result would give us a feeling of satisfaction instead of scrolling through websites for hours. Not to admit that I used ChatGPT successfully to pass a class test. LLMs have displayed immense potential by specializing in knowledge tasks that were thought to be unique to humans. We seem to be on the right track but there are still questions regarding reliability and responsible AI that needs to be addressed. Monetizing useful tools like Dall-E and ChatGPT are some other matters that I could leave for the readers to ponder. But the open-source community never fails to delight us with surprises like StableDiffusion when we wanted it the most after DALL-E 2 was monetized. ChatGPT is also a milestone that deserves immense appreciation. It was tried and used by millions of techies and non-techies around the globe which even jammed OpenAI servers at times. Let us look forward to what's coming next. As Nick Bostrom put it," Machine intelligence is the last invention that humanity will ever need to make. Machines will then be better at inventing than we are." Don't forget to check out the original paper. Images and data in this article were taken from the demo and original papers in the references.


References [1]. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, Robert Stojnic. Galactica: A Large Language Model for Science arXiv:2211.09085 [2]. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin. Attention Is All You Need arXiv:1706.03762


Scaling AI with Ray

Introduction to Ray Ray is an open-source unified compute framework making it easy to scale AI and Python workloads, from reinforcement learning to deep learning to model tuning and serving. Below is Ray's latest architecture. It mainly has three components: Ray Core, Ray AI Runtime, and Storage and Tracking. Ray Core provides a small number of core primitives (i.e., tasks, actors, objects) for building and scaling distributed applications. Ray AI Runtime (AIR) is a scalable and unified toolkit for ML applications. AIR enables simple scaling of individual workloads, end-to-end workflows, and popular ecosystem frameworks, all in just Python. AIR builds on Ray's best-in-class libraries for Preprocessing, Training, Tuning, Scoring, Serving, and Reinforcement Learning to bring together an ecosystem of integrations. Ray enables seamless scaling of workloads from a laptop to a large cluster. A Ray cluster consists of a single head node and any number of connected worker nodes. The number of worker nodes may be autoscaled with application demand as specified by Ray cluster configuration. The head node runs the autoscaler. We can submit jobs for execution on the Ray cluster or interactively use the cluster by connecting to the head node and running ray.init. It's simple to start and run Ray. The following will illustrate how to install it. Install Ray Install Ray libraries Furthermore, Ray can run at scale on Kubernetes and cloud VMs.


Scaling AI with Ray

Scale LLM and RL with Ray ChatGPT is a significant AI milestone with rapid growth and unprecedented impact. It is built on OpenAI's GPT-3 family of large language models (LLM) employing Ray. Greg Brockman, CTO and cofounder of OpenAI, said, At OpenAI, we are tackling some of the world's most complex and demanding computational problems. Ray powers our solutions to the thorniest of these problems and allows us to iterate at scale much faster than we could before. It takes about 25 days to train GPT-3 on 240 ml.p4d.24xlarge instances of the SageMaker training platform. The challenge is not just processing but also memory. Wu Tao 2.0 appears to need more than 1000 GPUs only to store its parameters. Training ChatGPT, including large language models like GPT-3 requires substantial computational resources and is estimated to be in the tens of millions of dollars. By empowering ChatGPT, we can see the scalability of Ray. Ray tries to tackle challenging ML problems. It supports training and serving reinforcement learning models from the beginning. Let's code in Python to see how to train a large-scale reinforcement learning model and serve it using Ray Serve. Step 1: Install dependencies for reinforcement learning policy models. Step 2: Define training, serving, evaluating, and querying a large-scale reinforcement learning policy model. Step 3: Now train the model, serve it using Ray Serve, evaluate the served model, and finally shut down Ray Serve.


Scaling AI with Ray

Ray Nostalgia and Trend Ray was initiated as a research project at RISELab of UCB. RISELab is the successor of AMPLab, where Spark was born. Professor Ion Stoica is the soul of Spark and Ray. He initiated to found Databricks with Spark and Anyscale with Ray as their core products. I was privileged to work with RISELab fellows in its early stage and witnessed Ray come into being. Above is Ray's project post in 2017. We can see it was elegantly simple but scalably powerful for AI applications. Ray is a stellar ship, proliferating. It is one of the fastest-growing open sources, as shown by the number of Github stars below. Ray is emerging in AI engineering and is an essential tool to scale LLM and RL. Ray is positioned for the massive AI opportunities ahead.


Learn Everything About The OpenAI Python Library & 5 Remarkable Things Chatgpt Can Do With Hands-On Examples In Python!

1. Introduction In recent years, OpenAI's ChatGPT has taken the world of Artificial Intelligence by storm. As a highly advanced language model, ChatGPT has the ability to generate human-like text and perform various NLP tasks with remarkable accuracy. In this article, we will explore 5 of the most impressive things ChatGPT can do, along with hands-on examples in Python. From text generation to question-answering, we will showcase the versatility and capabilities of this cutting-edge technology. Buckle up, and get ready to be amazed. Let's get started!


2. What Is It? In a nutshell, ChatGPT is an advanced language model developed by OpenAI. It is part of the GPT (Generative Pretrained Transformer) family of language models and is one of the largest and most powerful models of its kind. It has been trained on a massive corpus of text data, including books, articles, and websites, allowing it to generate human-like text and perform a wide range of NLP tasks with remarkable accuracy. At its core, ChatGPT is a deep learning model that has been trained to predict the next word in a given sentence based on the previous words. It uses a transformer architecture, which allows it to handle sequential data and maintain context in its predictions. This makes it well-suited for a variety of NLP tasks, including text generation, question-answering, sentiment analysis, and more.


3. How Was It Trained? ChatGPT was trained using a massive corpus of text data, including books, articles, and websites, totaling over 40GB of text. This data was used to train the model on the task of predicting the next word in a given sentence based on the previous words. The training process involved inputting large chunks of text into the model, allowing it to learn the relationships between words and the patterns in language. The model was then fine-tuned using smaller, more specific datasets to improve its performance on specific NLP tasks (see a graphical representation below). The training process was highly computational, requiring powerful hardware and a lot of time. However, this investment in training has paid off, as the resulting model is able to generate high-quality text and perform a wide range of NLP tasks with remarkable accuracy. The size of the model and the diversity of the text data it was trained on have allowed it to gain a broad understanding of language, making it well-equipped to handle a wide range of topics and situations. Furthermore, because the model is pretrained, it can be fine-tuned for specific tasks and domains, allowing it to be adapted to a wide range of applications and use cases.


4. The OpenAI Python Library Not many people know about it, but you can actually use ChatGPT from Python using the openai library. First, let's install it: and now let's also create an openai API key that is needed: Go to: https://platform.openai.com/account/api-keys and log in.Click on "Create new secret key"Copy the API key.


5. The 5 Remarkable Things Chatgpt Can Do With Hands-On Examples In Python Here are just 5 of the many remarkable things we can do with this library.


1. Text Generation ChatGPT can generate high-quality text that is often indistinguishable from text written by a human. For example: This prints: Cool!


2. Question Answering ChatGPT can answer questions with remarkable accuracy, providing relevant information based on the text it has been trained on. Example: ChatGPT got it right!


3. Sentiment Analysis ChatGPT can analyze the sentiment of a given text, determining whether it is positive, negative, or neutral. Example: Nice!


4. Summarization ChatGPT can summarize long texts, condensing the information into a more compact form. Example: Super nice reply!


5. Text Completion ChatGPT can complete sentences or paragraphs based on the input it receives, making it a valuable tool for writing and editing. Example: Putting all the pieces together: This is impressive!


6. Conclusions & Closing Remarks In conclusion, the 5 remarkable things that ChatGPT can do, including text generation, question answering, sentiment analysis, summarization, and text completion, have a wide range of practical applications. Whether it's for writing articles, generating summaries, or answering questions, ChatGPT can save time and effort by providing high-quality and relevant information quickly. Furthermore, its ability to complete sentences or paragraphs and analyze sentiment makes it an effective tool for writing and editing. With the increasing demand for AI solutions in various industries, the applications of ChatGPT are only going to grow. These examples in Python serve as a starting point for anyone looking to harness the power of this cutting-edge technology.


Stay tuned & Support this effort. If you liked and found this article useful, follow me! Questions? Post them as a comment, and I will reply as soon as possible.


Behind the Glory: The Dark Sides of AI Models That Big Tech Will Not Tell You

Mining for AI: Natural Resource Extraction To understand what AI is made from, we need to leave Silicon Valley and go to the place where the stuff for the AI industry is made. The term "artificial intelligence" may evoke the ideas of algorithms and data, but it is powered by the rare earth's minerals and resources that make up the computing components. The cloud, which consists of vast machines, is arguably the backbone of the AI industry. And it is built from resources like rocks, lithium brine, and crude oil. It converts water and electricity into computing power. From the lithium mines in Nevada to the extraction of "gray gold" (white lithium crystal) in San Francisco - once a hub for gold mining, the AI industry relies on extracting minerals from the earth's crust. Building natural language processing and computer vision models that run on the computational infrastructures of Amazon Web Services or Microsoft's Azure is energy-intensive.


The Myth of Clean Tech: Cloud Data Centers The data center has been a critical component of improvements in computing. These centers, being among the largest energy consumers globally [3], require water and electricity from sources such as coal, gas, nuclear or renewable energy to power their vast computers. China's data center industry gets 73% of its power from coal, emitting roughly 99 million tons of CO2 in 2018 [4]. In the Netherlands, for example, Microsoft's second data center in Middenmeer was reported to have used 84 million liters of water, way higher than the expected 1220 million liters [5], during a time of severe drought in the country. Addressing water-intensive and energy-intensive infrastructure is a major issue. Certainly, the industry has made significant efforts to make data centers more energy efficient and to increase the use of renewable energy. Some corporations, such as Apple and Google, are responding to concerns about the energy consumption of large-scale computation by claiming to be carbon neutral - which obviously means they offset their carbon emissions by purchasing carbon credits [1]. Microsoft supported the use of renewable energy at its data center in Dublin by using lithium batteries, which provide backup power for the data center in case the demand exceeds the power supply generated by the wind turbines [6]. However, this is far from a perfect solution to carbon dioxide emissions. The mining, production, and transport of the lithium battery itself have a negative impact on the environment, not to mention, lithium batteries have a limited lifespan; once degraded, they are discarded as waste [1].


The Carbon Footprint of AI AI industry has a meaningful carbon footprint. As estimated, the total greenhouse gas emission of the world's computational infrastructure has matched that of the aviation industry at its height, and it is increasing at a faster rate [7]. In 2018, OpenAI released an analysis showing that since 2012, the amount of computing used in the largest AI training runs has been increasing exponentially, with a doubling time of 34 months [8]. This represents an increase by a factor of 10 each year, mainly due to researchers building hundreds of versions of a given model by experimenting with different neural architectures and hyperparameters to find the optimal design and repeatedly finding ways to use more chips in parallel [8]. In a study published in 2019, Emma Strubell and her team at the University of Massachusetts Amherst conducted research on the carbon footprint of NLP models. They sketch out potential estimates by running AI models over hundreds of thousands of computational hours. While the average human is responsible for an estimated 5 tons of Co2e per year, Strubell's team estimated that training a single deep learning model can generate up to 284 tons of Co2 emissions - roughly equal to the total lifetime carbon footprint of five cars [9].


The Cost of Larger Models: Diminishing Returns Diminishing return is a growing concern in the AI industry, as stated in the 2019 paper from the Allen Institute. The study provides detailed data documenting the diminishing returns of image recognition models since the pace of model growth, and training size far exceeds the increase in model performance. For instance, in the figure below (on the right), an improved version of the Resnet152 (60.2M parameters), known as ResNeXt(83.5M parameters), requires an increase of roughly 35% in computational resources (measured by total floating-point operations - FPO ) to achieve only a 0.5% accuracy improvement [10]. In 2020 researchers at OpenAI [24] published a paper where they investigated how the performance of large transformer and LSTM models scale as a function of model size (number of parameters) and training dataset size (number of tokens). As shown in the above figure (on the left), the model performance can be saturated both by the size of the training dataset and by the size of the model. It seems that currently, the investment in more resources, e.g., training data and computation, still pays off in terms of downstream performance, although at an increasingly lower rate [9]. However, the AI industry's desire to build much larger models for incremental performance improvements may come at a high cost to the planet. Finding the saturation point is an important question to be looked into.


The Human-fueled AI The story of AI is a story about labor. Large language models like ChatGPT are impressive, but they are not magic - they rely on massive human labor, which often goes unacknowledged. This unseen labor helps build, maintain, and test AI systems and takes many forms, from the mining sector, where resources are extracted to create the core infrastructure of AI systems, to the software side, where labor is needed to manually label training data and review toxic content.


Setting the Low Rate for Crowworkers Workers do repetitive tasks such as labeling data or reviewing contents that are essential to sustaining AI systems but rarely receive credit for making the systems work and are even poorly compensated in some places[1]. A study from the United Nations International Labour Organization surveyed 3,500 crowd workers from 75 countries who offered their labor on popular task platforms like Amazon Mechanical Turk, Figure Eight and Microworkers. The report found that a substantial number of people earned below their local minimum wage even though the majority of respondents were highly educated [11].


Psychological Impact of Content Moderation and Labelling Data Those who do content moderation and labeling work, such as assessing violent videos and hate speech, can suffer from lasting forms of psychological trauma [12]. The internet contains a significant amount of toxic content such as child sexual abuse, murder, torture, self-harm, etc. This makes it difficult to purify the training data. Even a team of hundreds of people would have taken years to go through the giant datasets and label them manually. Hence the development of an AI-powered safety mechanism that can detect toxic language is crucial in removing harmful data and making AI tools safer [13]. Take chatGPT as an example. In order to train AI to recognize and remove toxic content, a labeled database of toxic content is required. These labeled examples are used to train an AI tool to detect those forms of toxicity. That detector will be built into ChatGPT to check whether the toxic text is included in its training data and filter it out before it reaches the users. To get labeled data, OpenAI outsourced the labeling to Sama - a San Francisco-based firm that employs workers in Kenya to label data for Silicon Valley clients like Google and Meta. However, the process of labeling such data was emotionally draining for Sama's employees, who had to handle disturbing text and images on a daily basis. One worker reported suffering from recurring nightmares after reading a graphic description of sexual abuse involving a child and an animal [13]. Due to the traumatic nature of the work, Sama ultimately ended its contract with OpenAI eight months earlier than planned in February 2022 [13].


Those who own the data own the future It can't be denied that data has become a driving force in the success of AI. Back in those days, data was remarkably hard to find. You couldn't even get a million words in computer-readable text easily. The internet, however, changed everything. As more people began to write online and upload photos to social media platforms, the digital era has created large-scale labeled data. Suddenly, training sets could reach a size that we could never have imagined. As an example, on an average day in 2019, approximately 350 million photographs were uploaded to Facebook, and 500 million tweets were sent [14]. This abundance of data has put tech giants in a powerful position - as they have access to a never-ending pipeline of images and texts that can be used to train AI models. This has been made easier by people happily labeling their photos with names and locations, resulting in more accurate, labeled data for computer vision and language models.


Bias in Data Let's discuss the elephant in the room: AI safety aims to align AI systems with human values. However, research has shown that large AI models have various forms of bias, such as stereotypical associations or negative sentiments toward specific groups [15]. This is largely due to the characteristics of the training data. Because the size does not guarantee the diversity The Internet is vast and diverse, and accordingly, it is easy to think that very large datasets, such as Common Crawl, which includes petabytes of data collected over 8 years of web crawling, must therefore be a good representative of the ways different people view the world. However, let us start with who is contributing to the texts on the Internet. The access to the internet itself is not evenly distributed, resulting in Internet data overrepresenting younger users and those from developed countries [15]. Furthermore, the data sources for large language models, such as Reddit and Wikipedia don't represent the views of the general population. For instance, GPT-2's training data is sourced by scraping data from Reddit, and Pew Internet Research's 2016 survey reveals that 67% of Reddit users in the United States are men, and 64% are between the ages of 18 and 29 [15]. AI bias comes from humans. Training data reflects existing human worldviews and social bias because the language we produce reflects our own biases. A human knows and agrees that it is unethical to discriminate against black people and women, but then, when a black woman applies for a job, the human subconsciously discriminates against her and decides not to hire her. Similarly, research has shown that language models like BERT and GPT-3 associate phrases related to disabilities with negative sentiment and overrepresent topics like gun violence, homelessness, and drug addiction in texts discussing mental illness due to the characteristics in its training data [16,17].


Listen to the algorithms - AI bias is the real killer The use of biased data can lead to unfair algorithms. A case exhibiting this was the Dutch childcare benefits scandal that led to the resignation of the Dutch government in 2021. The Dutch tax authority used a classification algorithm to detect fraudulent behaviors. This algorithm used an imbalanced data set because most of the observations were parents with an immigration background. Second, the model used the Dutch/non-Dutch nationality indicator as a predictor for the fraudulent risk model [18], resulting in the model negatively discriminating against people with non-Dutch nationality and wrongfully accusing 26000 parents of fraud [19]. In another example, a group of researchers developed an automatic system for classifying crimes using a crime dataset from the Los Angeles Police Department that includes thousands of crimes labeled by police as gang-related [1]. These researchers used a neural network that employs only four pieces of information, including weapons, suspects, neighborhood, and location, to predict gang-related violence. However, the police database is heavily skewed because not every crime is recorded. For instance, crimes that are more likely to be recorded are those areas heavily patrolled by police. Therefore, locations that are often checked by the police are over-represented in the data, leading to machine learning predictions that amplify existing bias. Project Maven, funded by the Pentagon, aims to build AI war machines that comb through drone footage and detect objects such as vehicles and humans as they move between locations [1]. This raises a fundamental ethical question surrounding the use of AI in warfare. Lucy Suchman argues that the problems with AI in warfare go far beyond whether the "killing was correct". Questions such as who built the training sets and which data was used, how objects are labeled as a threat, and what kinds of activities trigger a drone attack are crucial. Attaching life-or-death issues to potentially unstable and political algorithms is a risky gamble. In 2021, a US drone strike in Afghanistan wrongly killed 10 civilians who were originally believed to be associated with Islamic State. The Pentagon acknowledges the drone strike was a "tragic mistake", blaming the confirmation bias [20]. Imagine you are a soldier fighting in the war or just a normal individual who wrongfully got killed by your country's drones due to the mistake of AI. How terrifying that sounds.


The Drama of Privacy The collection of people's data to build AI systems raises clear privacy concerns. A noteworthy instance is a partnership between the Royal Free National Health Service Foundation Trust in Britain and Google's subsidiary DeepMind, where the trust agreed to share the health records of 1.6 million patients. But when the agreement with DeepMind was investigated, the company was found to have violated data protection laws by not sufficiently informing patients [22]. The advancement of technology should not come at the cost of compromising fundamental privacy rights.


The End of Consent The AI industry marks a shift away from consent-driven data collection. Those responsible for collecting the data believe that the internet's contents were theirs for the taking. Neither the people in the photos nor those who write the texts have a say in how the data they create is used. For example, a professor at the University of Colorado was reported to have installed a camera on the main walkway of the campus and secretly captured photos of more than 1700 students to train his own facial recognition algorithm for government-backed research [23]. Another example is Microsoft's dataset MS-Celeb, which scraped roughly 10 million photos of 100,000 humans from the internet in 2016. At the time, the dataset was described as the largest publicly available facial dataset in the world. Microsoft later took down the data set after a report from the FTs found that many of the people represented in the dataset were not aware of it and did not consent to have their photos used [24]. The success of AI relies on getting ever-larger datasets, but why would we accept this practice, despite the foundational problem like data privacy and consent?


Final thoughts In this article, I discussed a few hidden costs and consequences related to the use of large AI models. Firstly, the environmental impact is significant due to the high consumption of energy and water to power AI systems. How to make AI's industry green is critical for climate action. Secondly, there is the issue of human labor involved in the development and maintenance of AI systems. Despite being crucial to the functioning of AI systems, crowd workers are often underpaid and may suffer from lasting psychological trauma. This highlights the need for better compensation and support for these workers. Finally, there are ethical concerns surrounding the abundance of data created in the digital era, which AI relies on to function. Unfair algorithms and violation of privacy are examples that need to be addressed to ensure the fair use of AI technology.


Bibliography [1] Crawford, K. (2021). The atlas of AI: Power, politics, and the planetary costs of artificial intelligence. Yale University Press. [2] Hu, Tung-Hui. A Prehistory of the Cloud. MIT press, 2015. [3] Cook, Gary, David Pomerantz, Kassie Rohrbach, Brian Johnson, and J. Smyth. "Clicking clean: A guide to building the green Internet." Greenpeace Inc., Washington, DC (2015). [4 ] Datacenter Microsoft Wieringermeer slurpte vorig jaar 84 miljoen liter drinkwater. https://www.noordhollandsdagblad.nl/cnt/dmf20220810_68483787 [5] Microsoft datacenter batteries to support growth of renewables on the power grid. https://news.microsoft.com/source/features/sustainability/ireland-wind-farm-datacenter-ups/ [6] Cloud sustainability. https://cloud.google.com/sustainability/ [7] Masanet, E. (2018). How to stop data centres from gobbling up the worlds electricity (vol 561, pg 163, 2018). Nature, 561(7722). [8] AI and Compute. https://openai.com/blog/ai-and-compute/ [9] Strubell, E., Ganesh, A., & McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. arXiv preprint arXiv:1906.02243. [10] Schwartz, R., Dodge, J., Smith, N. A., & Etzioni, O. (2020). Green ai. Communications of the ACM, 63(12), 5463. [11] Berg, Janine, Marianne Furrer, Ellie Harmon, Uma Rani, and M. Six Silberman. "Digital labour platforms and the future of work." Towards Decent Work in the Online World. Rapport de l'OIT (2018). [12] Gillespie, Tarleton. Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media. Yale University Press, 2018. [13] Exclusive: OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic. https://time.com/6247678/openai-chatgpt-kenya-workers [14] "Facebook by the Numbers (2019)"; and "Advertising on Twitter." [15] Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021, March). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency (pp. 610623). https://dl.acm.org/doi/pdf/10.1145/3442188.3445922 [16] Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020). Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462. https://doi.org/10.18653/v1/2020.findings-emnlp.301 [17] Hutchinson, B., Prabhakaran, V., Denton, E., Webster, K., Zhong, Y., & Denuyl, S. (2020). Social biases in NLP models as barriers for persons with disabilities. arXiv preprint arXiv:2005.00813. https://doi.org/10.18653/v1/2020.acl-main.487 [18] The dutch benefits scandal: a cautionary tale for algorithmic enforcement. EU Law Enforcement. https://eulawenforcement.com/?p=7941 [19] Belastingdienst werkte ouders die recht hadden op kinderopvangtoeslag bewust tegen. https://www.trouw.nl/nieuws/belastingdienst-werkte-ouders-die-recht-hadden-op-kinderopvangtoeslag-bewust-tegen~bf13daf9/#:~:text=De%20Belastingdienst%20heeft%20bewust%20aangestuurd,kort%20ontkend%20door%20de%20Belastingdienst. [20] Pentagon acknowledges Aug. 29 drone strike in Afghanistan was a tragic mistake that killed 10 civilians. https://www.nytimes.com/2021/09/17/us/politics/pentagon-drone-strike-afghanistan.html [21] Bergen, M. (2018). Pentagon Drone Program Is Using Google AI [22] Revell, T. (2017). Google DeepMind's NHS data deal 'failed to comply'with law. New Scientist. [23] Hernandez, E., & Ehern, E. (2019). CU Colorado Springs students secretly photographed for government-backed facial-recognition research. The Denver Post. [24] Microsoft quietly deletes largest public face recognition data set. https://www.ft.com/content/7d3e0d6a-87a0-11e9-a028-86cea8523dc2 [25] Kaplan, Jared, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. "Scaling laws for neural language models." arXiv preprint arXiv:2001.08361 (2020).


Understand AI Interpretability & Explainability

Introduction It has become an urgent expectation to interpret and demystify black-box algorithms to increase their adoption. A growing number of organizations have begun to adopt Artificial Intelligence (AI) and Machine Learning (ML) for their crucial business decision-making procedures. This is in an effort to increase the adoption of these technologies. AI and ML are progressively being employed for the purpose of defining our day-to-day activities in a variety of domains, including finance, healthcare, education, recruiting, transportation, and supply chain, to name a few. But because AI and ML models play quite an important role, business stakeholders and consumers are becoming increasingly concerned about the models' inability to be transparent and interpretable. This is because black-box algorithms are highly susceptible to human bias. Model explainability is essential for high-stakes domains such as healthcare, finance, the legal system, and other critical industrial operations. In modern healthcare facilities, for example, data analysis on patient health is performed with the use of models that take into account a diverse range of factors. These algorithms are able to provide very precise predictions about the chance that a patient will suffer from a certain health condition. It is no longer just a suggestion but rather a choice that is guided by evidence, despite the fact that a doctor may or may not ultimately make the diagnosis. To interpret a machine learning model's decisions, one needs to find meaning in the model's output. You can also trace the output back to its original source and the transformation that created it.


Machine learning interpretation I'll give you a simple example to help you understand the concept of interpretation. So, simple linear regression is considered one of the basic models: yhat is the dependent variableβ0 and β1 are considered the constant, and the coefficient accordinglyx1 is the independent variable Let's introduce 2 new variables in the positions of yhat and x1. These are the income and education: Let's also suppose a simple example that education is measured in years and income in $. So, supposing a new regression model with bo=500 and b1=100: This means that for each additional year of education, the person gets $100 more on average. So, for example, someone who has been in school for six years will earn $1100 (as a prediction of the above model). One of the many ways that this model of linear regression may be described is by describing how the model itself operates, which is also one of the possible explanations. Having said that, this is just one perspective on the situation. Due to the fact that the actual results and the projected outcomes for the training set do not correspond with one another, the model is not infallible. The disparity (ε_i) between the two is referred to as the error, although it is also sometimes called the residuals. When assessing an error in a model, there are several different angles from which one might approach the problem. You may make use of an error function such as "mean absolute error" in order to quantify the degree to which the values that were actually obtained deviate from those that were predicted. When performing linear regression, the assumptions that someone has to test are homoscedasticity, linearity, normalcy, independence, and a lack of multicollinearity (if there is more than one feature). These assumptions make sure that the best model is being used for the problem at hand. When we have a better understanding of how a model operates and under what conditions it does so, we are better able to explain why it might generate particular predictions and when it doesn't. In the preceding example, what would happen if we wanted to explain why an individual with 6 years of education is anticipated to make $1100 but, in fact, earns only $800? This is an issue that often occurs. This is why we use the phrase "on average". Therefore, it should be anticipated that there will be circumstances in which the model will not be as reliable. What else would happen if we utilized this model to forecast values that have not yet been seen? For illustration's sake, let's pretend that it was trained on data ranging from 0 to 10 years of education and that we now have a case with 12 years of education. The earlier model takes on a straightforward form. In order to achieve a higher level of accuracy, we will need to include a greater number of variables. Some of them are mentioned in the bibliography for such problems, such as job experience, age, industry, etc. Would it be reasonable to add others? Perhaps the individual's gender or location? If the data that we obtained was solely based on males, then we do not know how accurate the results are for females (this is an example of selection bias), and what if the income had more to do with social standards and characteristics that are not included as variables (this is an example of omitted variable bias)? Another thing that needs to be done thereafter is to explain whatever features of the model (or variables) have an effect on the performance of the model (feature importance). We must be careful not to make things worse. Why?


Interpretability In most cases, researchers would use the words interpretability and explainability interchangeably. Despite the fact that these terms are quite closely linked to one another, there are certain studies that pinpoint their distinctions and separate these two ideas. Nevertheless, a number of efforts have been made [2] in order to define not only these terms but also relevant topics such as comprehensibility. There is currently no tangible mathematical concept for interpretability or explainability, nor have they been analyzed by some metric. None of these definitions make use of mathematical formality or rigor [3]. Doshi-Velez and Kim, in their work [4], describe interpretability as "the ability to explain or to convey in intelligible words to a human", which is one of the definitions of interpretability that is considered to be among the most widely accepted ones. Another prominent definition of interpretability was provided by Miller T. (2019), where he specifies it as "the extent to which a human is able to grasp the reason for a choice" [5]. Even if they make sense intuitively, these definitions are neither mathematically rigorous nor formal. Based on Masis S. (2021), I will try to keep up with a specific differentiation between them [6]. The term "interpretability" refers to the degree to which human beings are able to comprehend the inputs and outputs, as well as the causes and effects, of a machine learning model. If you are able to describe the conclusion that a model generates in a way that can be understood by humans, then you will be able to say that the model has a high degree of interpretability. To phrase it another way, the question is: why does a particular input into a model result in a specific output? When it comes to the data that is being input, what are the requirements and constraints? Where do we stand currently in terms of the confidence levels of the predictions? Or, why does one character have a significantly greater influence than the others? In the context of interpretability, the particulars of a model's operation are only relevant if the model can demonstrate both that it is the proper model for the use case and that it can explain how its predictions were arrived at. In the first example, we may claim that there is a linear link between income and education; hence, it makes more sense to use linear regression rather than a non-linear approach. Statistical analysis will allow you to demonstrate this, provided that the variables in question do not contradict the presumptions of linear regression. Even though the data is on our side, you should still consult with the domain expertise are involved in the use case. From dataset selection through feature selection and engineering, as well as choices regarding model training and tuning, many decisions are taken, each of which has the potential to enhance model complexity and decrease its interpretability. Because of its intricacy, describing how it operates is a difficult task. The interpretability of machine learning is a highly active field of study, and as a result, there is still a great deal of controversy about its exact definition. The question of whether or not complete transparency is required for a machine learning model to be considered adequately interpretable has been brought up in the discussion. It is not necessary for the concept of interpretability to exclude opaque models, which are often complicated, as long as the choices taken do not impair the trustworthiness of the model. This kind of concession is what many in the industry term "post-hoc interpretability." In post-hoc machine learning interpretation, a person explains the model's logic on behalf of the machine learning model.


Do we need interpretability? The capacity to understand decisions is not always necessary for decision-making systems. There could be a few exceptions: when incorrect results have no significant consequences. For example, suppose we have a use case where we want to use a computer vision application to count the number of cars in a specific area. If the model sometimes underestimates the total number of cars, there is probably no discriminatory bias and the cost will be relatively low (this means that the possible loss of the prediction does not cost too much). Interpretability is crucial, for example, in self-driving cars, where it is needed to understand possible points of failure. In addition, it is important in cases like LLMs that may produce unethical results based on gender or race. It is possible for us to fill in the blanks in our knowledge of the issue if we explain the choices that a model makes. Given the excellent accuracy of our machine learning algorithms, we have a tendency to enhance our level of confidence to the extent that we believe we completely comprehend the problem. This is among the most critical challenges, and it contributes to one of the most serious problems. Before continuing with explainability, let's see what the black box and white box models are....


Black box and White box models Simply put, "black box" is another name for models that are opaque. A system is said to have the characteristics of a "black box" if it can only be seen from the outside in terms of its inputs and outputs, while the internal workings of the system remain hidden from view. A black-box model may be opened up in the context of machine learning; nevertheless, the workings of the model are not straightforward to comprehend. White box models are the reverse of the black box approach. They attain complete or very close to complete interpretive transparency, which is why they are often known as transparent. In reality, they are straightforward models such as linear regression.


Explainability Everything that interpretability entails is included in explainability as well. The distinction is that it extends farther on the criteria of interpretability than it does on the need for transparency. This is because it requires human-friendly explanations for the "insides" of a model as well as the process of training a model, in addition to merely model inference. This criterion could include varying degrees of the model, design, and algorithmic transparency, depending on the application that's being used. There are several parts of transparency: Model transparency: The ability to describe how a model is trained on a step-by-step basis. Design transparency: Having the ability to provide an explanation for decisions such as the model architecture and hyperparameters used. For instance, we may defend these decisions by referring to the scale of the training data or the kind of information it contains. Algorithmic transparency: The capability of explaining how automated optimizations work. Random searches like random hyperparameter optimization or stochastic gradient descent make the total pipeline non-transparent. Other concepts that can affect the trust in models and also their explainability are "statistical-grounded", "non-reproducibility," "overfitting," "the curse of dimensionality," "human cognition," and "Occam's razor".


Do we need explainability? The capacity to make decisions that are both trustworthy and ethical is the primary driver behind explainability. There are a number of other factors that contribute to explainability, including causality, transferability, and informativeness. As a result, there are a lot of different use cases in which complete or almost complete transparency is prized, and this is quite appropriate.


References [1] Abraham C., Sims R., Daultrey S., Buff A., Fealey A. (2019). How Digital Trust Drives Culture Change. MITSloan Management Review [2] Lipton Z.C. (2018). The mythos of model interpretability. Queue, 16, 3157 [3] Adadi A., Berrada M. (2018). Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI). IEEE Access, 6, 5213852160 [4] Doshi-Velez F., Kim B. (2017). Towards a rigorous science of interpretable machine learning. arXiv [5] Miller T. (2019). Explanation in artificial intelligence: Insights from the social sciences. Artif. Intell., 267, 138 [6] Masis S. (2021). Interpretable Machine Learning with Python. Packt Publishing. Birmingham, UK


How I built Supervised Skin Lesion Segmentation on HAM10000 Dataset

Data Set To explore Binary Segmentation, I used the popularly available HAM10000 Dataset of 10015 skin lesion images with the type of cancer and their corresponding masks. Masks are 2D images with pixel intensities of either 1 (high) or 0 (low). They are essentially the product of any binary segmentation task, as our goal is to divide the image into two major regions. The types of cancer in the dataset include: Actinic keratoses and intraepithelial carcinoma / Bowen's disease (AKIEC),basal cell carcinoma (BCC),benign keratosis-like lesions (solar lentigines / seborrheic keratoses and lichen-planus-like keratoses, BKL),dermatofibroma (DF),melanoma (MEL),melanocytic nevi (NV)vascular lesions (angiomas, angiokeratomas, pyogenic granulomas and hemorrhage, VASC). Some data exploration was done on the data set. It was found that the most common type of cancer was Melanocytic Nevi with a 67% share. Since my focus was on segmentation, I focused only on the images. The type of cancer could be used in other downstream tasks, such as cancer classification. Some EDA was done on the ground truth masks. Using the fraction of pixel intensities, I found the distribution of the size of the lesion compared to the image size. Note: A point to note here is that the zoom level of the images when they were captured is not known. Zoom level can immensely affect the analysis done here. Since we have no information about it in the data, we can revisit some outliers. This gives us an idea of whether the reason for the change in the fraction covered is severity or zoom level. Upon monitoring a few of the outliers, it was found that some images appear to be taken through a microscope, indicating a high zoom level, whereas some others appear to be taken using just a camera. To avoid this kind of deviation in the dataset, a more structured way of data collection should be followed, such as additional meta-data on how the image was captured and under what level of zoom. This will enhance the quality of the data set and any downstream tasks performed on it. This analysis is done under the assumption that all the images had the same zoom level or under the assumption that the zoom level does not affect our approach significantly. The distribution of the size of skin lesions looks skewed towards the left side, with the most frequent 'fraction of image covered by skin lesion' of ~1020%. Since our model will be built based on this data, if we are to test a different data set on our model, we will have to ensure a similar distribution of lesion sizes.


Approach A high-level idea for the approach used is as follows. The specifics and detailed descriptions are provided later on. Model - To perform the task of binary segmentation, a Convolution Neural Network is used.Input - The input to our CNN would be an RGB image from the dataset resized to 128x128.Output - The output would be a 2D mask image of the size 128x128.Loss function - Dice loss is used as the loss function for this model. It is a common metric used in binary segmentation.Tracking metric - Jaccard Index/Intersection over Union is another popular metric in binary segmentation.


Model The python library 'segmentation models pytorch' was used to easily create and handle the CNN model. 'segmentation models pytorch' allows us to create a model by combining encoder and decoder architectures and pre-trained weights from popular CNN architectures like ResNet, VGGNet, UNet, etc. The model which was used for the task had: U-net architecture: To exploit the strength of the encoder-decoder framework and skip connections from the encoder to the decoder. The encoder converts the image to a concise lower-dimensional shape by using a series of convolutional layers and max pooling layers. The decoder then converts the concise shape to the required output shape through a series of convolutional and upscaling layers.This framework is found to be effective because converting the information to a lower dimensional vector removes noise and helps better training on important features. The additional skip connections are found to increase the quality of the output.Resnet34 encoder and decoder with pre-trained weights on ImageNet: To help with good initialization. ImageNet is a popular image dataset used in competitions and for comparing models. By using pre-trained weights, we are ensuring the model's ability to identify important features to start with. This helps us use lesser data and lesser training time since most of the work is done and only finetuning to our task is needed.


Input and Output The images in the data set were of shape (3,450,600), indicating that they are color images with 3 channels - RGB, and have a height of 450 pixels and width of 600 pixels. These images were resized to (3,128,128). This is because the encoder-decoder architecture of CNNs easily handles sizes of power of 2. Higher-quality images are computationally expensive, but down-sampling too much will result in the loss of important features. So 128 was chosen to balance the trade-off between image quality and computational time. The 2D output masks and the ground truth masks too, were set to the same size (128,128).


Loss function and metrics The quality of results from a machine learning model is highly influenced by what metric we use to evaluate and assess the model. Different evaluation metrics and loss functions can be used to test our model performance. In any prediction task, for every data point, we have a ground truth which is the expected correct output, and a corresponding prediction that the model outputs. The purpose of an evaluation metric is to give us an idea of how close the prediction is to the ground truth. Now that we have a qualitative intuition behind what the purpose of a metric is, we use the prediction and ground truth to make up a quantitative formula. Let us suppose a classification task of 2 output 2 classes. So for every data point, the output can be either of the 2 classes, say 0 (Negative class) and 1 (Positive class). Now for every data point, we have a corresponding ground truth and a prediction. We can split the dataset into 4 parts: True Positives (TP): The data points where both ground truth and prediction were class 1.True Negatives (TN): The data points where both ground truth and prediction were class 0.False Positives (FP): The data points where ground truth was class 0 and prediction was class 1.False Negatives (FN): The data points where ground truth was class 1 and the prediction was class 0. By looking at these measures we see that we would aim to maximize TP and TN and minimize FP and FN. 'Accuracy' is a well-known metric and is used in a lot of prediction applications. Accuracy can be simply defined as the percentage of correct predictions out of all the predictions. Using the above-mentioned terms, the formula for accuracy is: It looks all plain and straightforward till now! But we now face 2 problems which we present and solve sequentially. The first problem is that the task at hand is binary segmentation and not binary classification. For every data point, the output is not a binary class but an image! We are tasked with comparing a ground truth image with a predicted image. An image is just an array of pixel-intensity values. In our case, the output images are masks in which pixel intensities can either be 0 or 1. So now we just have both ground truth and predicted values of an array of pixels whose values are either 0 or 1. This looks very similar to the binary classification stated above. Now we can just use TP, TN, FP, and FN over all the pixel values and calculate accuracy. The second problem we face is that of choice of metric. Accuracy can prove to be ineffective in some applications. For our task, we are interested in predicting the area of the affected skin, i.e., we focus more on getting the predictions of one of the classes right, the class being 1, let's say. In this case, some metrics give better insights. Jaccard Index or Intersection Over Union (IoU) is simply the ratio of the number of activated pixels in the intersection of ground truth G and predicted image S and the union. where . denotes the number of activated pixels. We can rewrite the same in terms of TP, TN, FP, and FN as: Since only TP is there in the numerator and not TN, only correct predictions of class 1 will increase the score. This metric is very intuitive because, ideally, we want to have as much intersection between ground truth and prediction as possible, and we want the union to be as close to the intersection as possible. A closely related metric to the Jaccard Index is the Dice Score. The Dice Score is a spatial overlap metric that is computed as follows for predicted image S and ground truth G. This can be rewritten as: Either of these metrics can be used for binary segmentation. We have used the Dice Loss for training the model. The model minimizes the Dice Loss, thereby maximizing the Dice Score. We have used Jaccard Index for final evaluation.


Training The python library 'PyTorch lightning' was used for training the model. 4000 images were used out of the complete data set. Out of these, 95% (3800) was used for training, and 2.5% (100) was used each for validating and testing. The following hyperparameters were chosen after tuning by trial and error: Batch size: In batch-wise training, the model is updated after every batch of data points based on the loss computed using the data points of the batch. The batch size was selected to be 16.Optimizer: Adam was used as an optimizer with an initial learning rate of 0.0001. Adam optimizer helps in better convergence by using momentum-based gradient descent and adaptive learning rate. It can handle sparse gradients on noisy problems.Epochs: This is the number of iterations the model trains through the whole dataset. The more the number of epochs, the better it fits the training data. It should be such that the model neither under fits nor overfits. The number of epochs was set to 20.


Results After training, the test set was evaluated using the trained model. Jaccard Index was calculated for each sample and averaged over the whole test set. The mean Jaccard Index was found to be 0.8828. Some results are visualized in figures 25.


Failure cases and shortcomings Though the model achieved a high Jaccard Index on the test data, the average score was pulled down due to a few specific scenarios. Some of them are mentioned in figures 69.


References https://www.kaggle.com/datasets/surajghuwalewala/ham1000-segmentation-and-classificationBasak, H., Kundu, R., Sarkar, R. (2022). MFSNet: A multi-focus segmentation network for skin lesion segmentation. Pattern Recognition, 128, 108673. https://doi.org/10.1016/j.patcog.2022.108673https://www.labellerr.com/blog/semantic-vs-instance-vs-panoptic-which-image-segmentation-technique-to-choose/Pham, H. N., Koay, C. Y., Chakraborty, T., Gupta, S., Tan, B. L., Wu, H., Vardhan, A., Nguyen, Q. H., Palaparthi, N. R., Nguyen, B. P., H. Chua, M. C. (2019). Lesion Segmentation and Automated Melanoma Detection using Deep Convolutional Neural Networks and XGBoost. 2019 International Conference on System Science and Engineering (ICSSE). https://doi.org/10.1109/icsse.2019.8823129


This AI newsletter is all you need #33

What happened this week in AI by Louis ChatGPT and generative AI are still hot topics this week. ChatGPT reportedly reached 100 million monthly active users in January, making it one of the fastest-growing apps in history. This success is leading to increased pressure on incumbent companies to integrate the latest LLM technology into their products. Open AI has launched several integrations with Microsoft as well as details of its subscription plan for ChatGPT. Google has also announced the upcoming release of its new ChatGPT competitor called "Bard" which is built on its LaMDA model. After its partnership with OpenAI, Microsoft is preparing to launch a version of its Bing search engine using ChatGPT, aiming to become more competitive with Google. ChatGPT has become more accessible through integration into Microsoft's release of Teams Premium. OpenAI is also planning to launch a mobile GPT app and test a video-generating AI. The rapid adoption of this technology also raises new questions and Microsoft, GitHub, and OpenAI are facing a proposed class action complaint that accuses them of scraping licensed code to build GitHub's AI-powered Copilot tool. The companies have asked the court to dismiss the complaint. OpenAI has launched a pilot subscription called ChatGPT Plus, to monetize its viral success. The service offers faster response times, access to ChatGPT during peak times, and priority access to new features and improvements. It starts at $20 per month. The early rumors were that the ChatGPT Plus feature would cost $42 per month, and many people believed it was too expensive. However, now that it's priced at $20 per month, do you think it's worth subscribing to? This issue is brought to you thanks to Qdrant: Qdrant open-source vector search engine launches managed cloud platform Qdrant - robust vector similarity search engine with advanced filtering support. It is written in Rust, which ensures stability and high performance proved by benchmarks. The managed cloud platform is now fully available for business use, allowing companies of any size to benefit from Qdrant's cutting-edge features without handling its deployment and maintenance. Qdrant cloud platform can be accessed through the website.


This AI newsletter is all you need #33

Hottest News 1.Is ChatGPT Coming for Your Job? ChatGPT has been a topic of discussion since its launch, with opinions divided between its potential benefits and perceived threats. While some experts see ChatGPT as a tool that could greatly enhance AI-human partnerships, many people are unsure of how to react to it. This article asked the robot itself, what impact the ChatGPT could have on the engineering profession. 2. Big Tech was moving cautiously on AI. Then came ChatGPT Small companies are driving AI to the masses, prompting Big Tech to respond. Three months before ChatGPT's launch in November, Meta, Facebook's parent company, introduced a similar chatbot, Blenderbot. However, Blenderbot failed to create the same excitement as ChatGPT. According to Yann LeCun, Chief Artificial Intelligence Scientist at Meta, the reason it was boring was that it was made safe. 3. Cool things that happened at Scale AI's Generative AI hackathon In this Twitter thread, @alexandr_wang shared a list of the most interesting projects, including the winning projects created by the ~300 hackers who attended Scale AI's Generative AI hackathon last week. 4. The AI literacy project TheGP is seeking to connect with talented hackers and quick prototypers who are actively contributing in the fields of engineering, product, or design as part of the AI Literacy project. They aim to share and publish insights gathered from various builders and provide further learning opportunities by delving into deeper discussions on topics such as accessibility & application of language models, toolkits they use, and enhancing productivity & creativity with LLMs. 5. Gaslighting and reality in AI 2022 was the year when generative AI left the lab and gained widespread recognition. Gary Marcus reflects on what has and hasn't changed in recent years and provides a series of examples demonstrating the ease with which generative AI can produce nonsensical results and its superficial understanding of reality.


This AI newsletter is all you need #33

Three 5-minute reads/videos to keep you learning 1.How to build a chatbot that answers questions about documentation and cites its sources The tutorial was initially hosted via a live stream on our Learn AI Discord. It teaches you how to build Buster, a chatbot that answers questions related to the Hugging Face transformers library while referencing its sources. It outlines the three critical components required to make Buster function: gathering data from the documentation, constructing a document retrieval system & ranking the most relevant sources, and generating text based on the user's question and providing the answer. 2. Guide and resources for prompt engineering This GitHub guide includes a collection of recent papers, educational resources, datasets, and tools relevant to prompt engineering. It also features a compilation of blog posts and books for further learning. 3. AI-enabled executives: how ChatGPT will sharpen strategic thinking The article discusses how ChatGPT can enhance strategic thinking and decision-making capabilities, such as anticipating and planning for the future, thinking critically and creatively about complex problems, and making effective decisions in uncertain situations. 4. When to Build vs. Buy Your Data Warehouse (5 Key Factors) Nishith Agarwal, the Head of Data and ML Platforms at Lyra Health and the creator of Apache Hudi, draws on his experiences at both Uber and Lyra Health to present five considerations that impact the decision to build or buy the data warehouse, data lake, and data lakehouse layers of a data stack. These considerations include cost, complexity, expertise, time to value, and competitive advantage. 5. How to build a simple semantic search engine In this Twitter thread, Cohere presents a step-by-step guide on how to construct a basic semantic search engine. The steps involved include obtaining a list of texts to search, embedding the archive of questions, converting the embeddings into indexes, and others. Enjoy these papers and news summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #33

Upcoming Community Events The Learn AI Together Discord community hosts weekly AI seminars to help the community learn from industry experts, ask questions, and get a deeper insight into the latest research in AI. Join us for free, interactive video sessions hosted live on Discord weekly by attending our upcoming events. Recurrent Networks: The Neural Network Architecture Seminar (#5) This week's session in the (free) nine-part Neural Networks Architectures series will be led by Pablo Duboue (DrDub) and focuses on Recurrent Networks. During this session, he will explore topics such as training RNNs by unrolling, internal memory access and update in LSTMs and GRUs, as well as Encoder/Decoder and attention in Encoder/Decoder systems. Find the link to the seminar here or add it to your calendar here. Date & Time: 7th February, 11 pm EST If you missed the first part of the series, find last week's event recordings here. 2. LAIT's Reading Group Learn AI Together's weekly reading group offers informative presentations and discussions on the latest developments in AI. It is a great (free) event to learn, ask questions, and interact with community members. Join the upcoming reading group discussion here. Date & Time: 11th February, 10pm EST Add our Google calendar to see all our free AI events! Please reach out and pitch us your idea if you'd like to present any AI-related content to our community, whether it is a class, tutorial, paper, reading group, panel discussion or competition. We would love to host it live on our 38,000-member Learn AI Discord community. The content will be recorded, 100% owned by you, and available for you to distribute on your own Youtube channel or website afterward.


This AI newsletter is all you need #33

Meme of the week! Meme shared by Louis B#1408


This AI newsletter is all you need #33

Featured Community post from the Discord jUMAD1#0227 recently updated a data science notebook on linear regression for WW2 temperatures. The notebook was part of the 2019 Machine Learning Scholarship by Indonesia's Ministry of Communication and Informatics. The update includes improved structure and code, a review of linear regression assumptions, and evaluation metrics. Find the notebook here and support a fellow community member! Join the conversation and share your feedback here.


This AI newsletter is all you need #33

AI poll of the week! Join the discussion on Discord.


This AI newsletter is all you need #33

Article of the week Breaking Down DeepMind's AlphaTensor by Adrienne Kline The first AI system for discovering novel, efficient, and mathematically proven algorithms for fundamental tasks such as matrix multiplication. This system sheds light on a 50-year-old open question in mathematics regarding the fastest method for multiplying matrices. This paper, released in 2022, is a fundamental breakthrough in machine learning and attempts to answer a translational research question with domain-agnostic applications and implications. The author has explained the paper in the simplest possible manner.


This AI newsletter is all you need #33

Our must-read articles Handwritten sentence recognition with TensorFlow by Rokas Liuberskis A Brief History of Neural Nets by Frauke Albrecht If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.


This AI newsletter is all you need #33

Job offers Software Engineer II, Backend @Freenome (Remote) Reserve Engineer @Cobalt Robotics (Seattle/Bellevue WA) Perception, System Integration/ML Deployment Engineer @Outrider (Golden, Colorado) Sr. Full Stack Engineer, Enterprise Platform @Labelbox (Remote) Senior Automation QA Test Engineer @Arturo (Hybrid/Denver, Colorado) If you are preparing your next machine learning interview, don't hesitate to check out our leading interview preparation website, confetti!


Paper Review: Multimodal Chain of Thought Reasoning

Learnings from the paper The authors of Multimodal Chain-of-Thought Reasoning in Language Models show the following: How LM's with less than 1 Billion parameters can effectively leverage CoT.How Visual Features can be combined with the text features produced by an LM to help with multimodal reasoning.A two-step process of generating the rationale followed by generating the answer is more effective than a one-stage process of producing the rationale and answer together.


CoT and models with fewer than 1 Billion Parameters It's been observed that smaller models tend to make up things more often than larger models do. This is often referred to as hallucination. This paper concluded that CoT-based prompting only helps when models have at least 100 billion parameters! The authors also observed a very high rate of hallucination when they trained the UnifiedQA (a T5 model trained on a bunch of Question Answering datasets) model to generate the rationales/reasoning. They randomly sample 50 cases (note that this is a very small sample size) where the model predicts the wrong answer to a question and observe that in 64% of the cases, the rationales contained hallucinations.


Dataset In this paper, the authors focus on creating a model that can help with multimodal question answering as opposed to text-only question answering. The dataset in focus is ScienceQA. The dataset can have images and text as a part of the context and also contain explanations for an answer so that models can be fine-tuned to generate CoT/rationales.


Use of Visual Features As we've seen that smaller LMs are prone to hallucinations when they generate their CoT/rationale, the authors speculate that having a modified architecture where the model can leverage the text features produced by an LM and Visual Features produced by an image model would be more equipped at producing rationales and answering the questions. The authors make use of visual features generated by the DETR model.


Multimodal CoT The authors propose having two stages: Use text (question + context) and visual features to produce a rationale.Use the produced original question + context + rationale produced from step 1+ visual features to produce an answer. Both steps 1 and 2 go through the same model architecture. This idea is summarized in the image below. Essentially there are two forward passes, each forward pass has a different input and output as explained above.


Architecture Overall, we need a model that can produce text features and visual features and leverage them to generate a textual response. However, it is important for there to be some interaction between the textual and visual features, essentially some sort of co-attention mechanism. This helps in encapsulating the information present in both modalities. To accomplish all of that, the authors choose a T5 model, which has an encoder-decoder architecture and, as stated above, the DETR model to produce visual features. The encoder of the T5 model takes care of producing textual features. However, instead of leveraging the textual features produced by the encoder, the decoder of the T5 model instead uses the output of a co-attention-styled interaction layer that the authors propose.


Interaction between Visual and Textual Features Assume that H_language is the output of the T5 encoder. X_vision is the output of DETR. The first step is to ensure that the visual features and the textual features have the same hidden size so that we can use an attention layer. Note: All code snippets are from the repo: W_h is essentially a linear layer, and H_vision corresponds to the final visual features. W_h helps in changing the size of the visual features to match that of the textual features. Next, we need to add an attention layer so that the visual and textual features can interact with each other. For this, the authors use a single-headed attention layer with H_language as the query vector and H_vision as the key and value vectors. Now we have embeddings that contain information from both the textual and visual features. The authors then leverage gated fusion to produce the final set of features that'll be sent to the decoder. Gated fusion has two steps: Obtain a vector of scores between 0 and 1 to determine the importance of each of the attention features.Use the score to fuse the text and attention features. W_l and W_v are essentially two linear layers. Finally, the fused features are passed to the decoder. That's pretty much the architecture that the authors follow! However, remember that there are two stages. The first stage is to produce the rationale/CoT. The second stage leverages the CoT produced in the first stage to produce the answer, as shown in the image above.


Results The authors use the UnifiedQA model's weights as the initialization point for the T5 model and fine-tune it on the ScienceQA dataset. They observe that their Multimodal CoT method outperforms all previous benchmarks, including GPT-3.5. Interestingly even the base model which has only 223 million parameters, outperforms GPT-3.5 and other Visual QA models! This highlights the power of having a multimodal architecture. The authors also show that their two-stage approach outperforms a single-staged approach.


Conclusion The biggest takeaway from this paper is how powerful multimodal features can be in solving problems that have visual and textual features. The authors show that with visual features, even small LMs can produce meaningful Chains of Thought/reasoning with much less hallucination, which brings to light the role vision models can play in evolving CoT-based learning techniques. From their experiments, we see that adding visual features that come at the cost of a few million parameters brings much greater value than scaling up a text-only model by billions of parameters. If you have any thoughts, drop a comment below. Until the next time, take care and be kind.


Introduction to Autoregressive Models

Problem Statement Not limited to Deep Learning, "distribution estimation" is a problem in many applications. The goal is to estimate the distribution P(x) underlying dataset D. Therefore, the estimated distribution should be close to the actual distribution and tractable. By saying tractable distribution, the distribution has a closed form and is computationally efficient to obtain.


Introduction to Autoregressive Models

Why do other methods become intractable? Before NADE, one of the most popular methods was using directed graphical models or Bayesian networks. The "Bag of words" (BOW) method uses Bayesian networks, a popular NLP tool. It tries to estimate the feature vector x of a sentence s with a vocabulary v. Then we have the estimated x* = [ '#times w in s' for w in v]. The feature vector x has size N = size of the vocabulary v. This method is quite suitable for text and document classification with small vocabulary sizes. When N increase, x will become too large and requires heavy computations. Also, the assumption on x is that words are independent of position. But they are not just binary and also different in contexts. For example: "Bob likes Alice" and "Alice likes Bob" are entirely different, but they ended up with the same feature vector. General Bayesian Network models like BOW require many assumptions and constraints to be tractable.


Introduction to Autoregressive Models

Autoregressive Models Autoregressive Models (ARM) are architectures that rely only on past values to predict current data values. For example, simple ARMs use lagged values p, AR(p) models, to indicate current value based on "p" past values: Where y_t is the current value of the time series, y_{t-1}, y_{t-2}, ..., y_{t-p} are the lagged values of the time series, c is a constant, a_1, a_2, ..., a_p are the coefficients of the model, and e_t is the error term. In other words, AR(p) parameterizes the distributions P(y_t  y_{t-1}, ..., y_{t-p}). And by that, we have: These AR models are often used in time-series applications. But they are not just that. Usually, we have to deal with data where we can embed them into a vector x of dimension d. Therefore, we can estimate the distribution of each dimension 'i' by the previous dimensions' values. In the next section, I will explore the architecture of Neural Autoregressive Models to compute such p(x) in a tractable and efficient way.


Introduction to Autoregressive Models

Neural Autoregressive Models [1] NADE is an Autoregressive Model that models the distribution p(x) if the input vector x of dimension D. For any order o (permutation of the integers 1 to D), we have: Where o_{<d} is the first d-1 dimensions in o and x_{o{<d}} is the corresponding sub-vector of x. The architecture of simple NADE: In details, the model parameterises each conditional distribution by a feed-forward neural network. We have: In this figure above, a hidden layer matrix W and bias c are shared by all hidden feature h_d. Then it is used to compute the probability of x_{o_d} via prediction layer V with bias b_{o_d}. The weight sharing scheme for W makes the parameter complexity of the model become O(HD), where H is the hidden vector size and D is the input vector size, rather than O(HD²). NOTE: The computational complexity for hidden vector h_d is also O(HD) because we can reuse the output of previous computation for h_{d-1} The model is trained under the following objective: And usually with SGD optimizer. The log term goes from 0 -> -infinity when P goes from 1 to 0. So, by minimizing this NNL loss, we push P closer to 1. In other words, it increases the accuracy of predicting real value x_{o_d} with x_{o_{<d}}.


Introduction to Autoregressive Models

Conclusion Our goal is to have a tractable estimator. Neural Autoregressive models like NADE are more robust to the size of inputs dimension, promising us an efficient computation for distribution estimator. However, we cannot say it is going to guarantee us a tractable estimator and a good estimator as well. AR models work on a strong assumption that the current values rely only on past values. For example, in sentence completion tasks in NLP, we may need to look both backward and forward to get the whole context of the sentences.


Introduction to Autoregressive Models

Applications and further reading. Some variants of deep Autoregressive Models: PixelCNNPixelCNN++WaveNet A Generative Model for Raw Audio.Transformers: Attention is all you need These models have been successful in many applications. Especially Transformers have already dominated the AI world for a while. Its success is not just limited to Natural Language Processing (NLP) but also Computer Vision (CV) and more. The article is only to introduce the concept of the Autoregressive Model. There are also many problems underlying this architecture, such as: How does NADE generalize to other data types?"Dimension orderless" training procedure for Deep NADE.Relation between I will leave the resources for readers who are interested in diving deeper into AR models.


Introduction to Autoregressive Models

References [1] Uria et al. Neural Autoregressive Distribution Estimation, Journal of Machine Learning Research 1 (2000) 148 Uria et al. A Deep and Tractable Density Estimator. International Conference on Machine Learning, Beijing, China, 2014 Aaron van den Oord el al. Pixel Recurrent Neural Networks Tim Saliman et al., 2017. PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications Aaron van den Oord et al. WaveNet A Generative Model for Raw Audio. Vaswani et al. Transformers: Attention is all you need


Navigating the World of Data Engineering: A Beginner's Guide.

Data or data? No matter how you read or pronounce it, data always tells you a story directly or indirectly. Data engineering can be interpreted as learning the moral of the story. Welcome to the mini tour of data engineering where we will discover how a data engineer is different from a data scientist and analyst. Processes like exploring, cleaning, and transforming the data that make the data as efficient as possible. If you ever wonder how predictions and forecasts are made based on the raw data collected, stored, and processed in different formats by website feedback, customer surveys, and media analytics, this blog is for you. Data engineering is a process that includes steps on how the data flows once the data is collected, ingested, and later managed into the storage system like rows and column tables. The first step is data collection and preparation. This step includes cleaning the data for better and more accurate results we need. Cleaning the data can be finding and removing the missing values or duplicate values, converting the values into the same data format of a row, etc. Now we have organized our data for further exploration and visualization. The visualization of the data is important as it gives us hidden insights and potential details about the dataset and its pattern, which we may miss out on without data visualization. These visualizations can be done using platforms like software tools (e.g., PowerBI, Tableau) and programming languages like R and Python in the form of bar graphs, scatter line plots, histograms, and much more. To learn more about visualizations, you can refer to one of our many blogs on data visualization for a glance. With the help of the insights, we make further decisions on how to experiment and optimize the data for further application of algorithms for developing prediction or forecast models.


What are ETL and data pipelines? A data pipeline is a process that ensures the flow of data efficiently by automation techniques and reducing errors. The ETL framework is popular for Extracting the data from its source, Transforming the extracted data into suitable and required data types and formats, and Loading the transformed data to another database or location. These data pipelines are built by data engineers.


Extract: Extraction is the first step that needs to be performed to get the required data ready for processing. The source of extraction of data can be files like text files, excel sheets, word documents, databases like relational as well as non-relational, and also the APIs. Text files include both unstructured and flat files (row and columns as record and attribute like .tsv and .csv). The JSON format is also used for extraction, which is the JavaScript Object Notation. It is semi-structured with data types like integer, string, null, boolean, and array as well as objects. The APIs deal with requesting responses from the web. For example, Twitter API provides us with a few user info, their tweets, activity, etc. There are application databases and analytical databases. Application databases include transactions, inserts, updates, changes, and online transactional processing (OLTP) and are row-oriented. OLTP allows the databases to be executed by large numbers of people on the internet in real-time. The analytical databases are column-oriented and follow the OLAP method. OLAP is an approach that performs analysis on huge volumes of data from a store, warehouse, database, etc. We'll learn more about this in detail ahead.


Transform: Transformation comes from an extraction of data from the source. It can be the selection of features, validation, splitting as well as joining the data features. These transformations are done to make data more efficient to use and work with, identify hidden patterns and trends, and make our work easier and simpler as much as possible. E.g., join() and split() methods.


Load: In short, loading is a process that loads the data from sources like analytical, applications, or MPP databases into your required system or another database. These databases can be both columns as well as row-oriented. This is done with the help of aggregating queries and performing online transactions. The automation techniques can be extracting the data from file formats and various sources, transformation. Combining different data from different sources, validation of data, and loading the data into the databases. As these techniques are executed by the machine itself, it reduces the chances of human error and requires lesser human attention. One of the major advantages of data pipelines is that it minimizes the time consumption of the data flow. The data pipelines follow the Extract, Transform, and Load (ETL) framework. Data pipelines play a crucial role in managing and handling data from one system to another. This data can be directly loaded and can be ready to use in the applications.


Difference between Data Engineer and Data Scientist You may have already guessed by now, that it is the job of a Data Engineer to collect, store, prepare, explore, visualize, and experiment with the data and its limits to utilize it as much as possible. Perfectly using the correct data for the right people in every possible way is one of the required skills by the data engineer. The data engineer is responsible for ingesting the data from various external as well as internal sources, optimization of data storage systems, and removing corrupt and unwanted data for others to further carry out tests and advance the development. This is done with the knowledge of how to process a large amount of data and its management with the required and appropriate skills of using the available machines, platforms, and tools for


The skills of Data Engineers: Ingestion and Storage of dataSetting up the databasesBuilding the data pipelineHas Strong software skills Data engineers make a Data Scientist's life easier. As we now know, the data workflow consists of collection & storage, preparation, exploring, and gaining insights through visualization and experimentation. The data engineers deal with the data collection and storage part of the workflow. The later steps are carried out by the Data Scientists.


The skills of a Data Scientist: Exploiting DataAccessing databasesUsing pipeline outputsHaving Strong analytical skills


The famous Big Data and its Vs Today's trending jargon is 'Big Data', which you have heard many times in this modern digital era. While you are reading this, within seconds, a huge amount of data is generated and processed around the whole world in many ways, like sending and receiving emails, playing media like videos and streaming music, and payment transactions. The word 'big' can mean a huge quantity concerning the type of data, accuracy, usefulness, size, and volume. Sensors, devices, social media, organizational data, communication channels, and multimedia play a crucial role in the growth, management, and development of big data.


The 5 V's of Big Data Volume: The size and amount of the data.Value: The quality and helpfulness of the data. Some of the data can be converted into valuable data.Variety: The range and diversity of data types like structured (row-column format), unstructured (images, videos) as well as semi-structured data (e.g. JSON, XML, YAML).Velocity: The speed of the data to ensure a smooth and continuous flow of data for storing, handling, and managing the data in a given period of time like within an hour, day, or a week.Veracity: It is the accuracy and honesty of the data. The truthfulness of the data showcases confidence in the results. More on Vs of Big data here: https://www.irjet.net/archives/V4/i9/IRJET-V4I957.pdf


Data Storage Let us first understand structured, unstructured, and semi-structured data.


Structured Data: About 20 percent of the data is structured in the real world. Structured data as the name suggests follows a structural format. For example, relational databases store the data in row and column model structures. This type of data is easy to organize and can be searched quickly when necessary. These rows and columns contain defined data types like numbers and characters. They are created and managed with the help of SQL queries.


Unstructured Data: This kind of data has no fixed structure and does not follow any model. This data cannot be stored in the row and column format. This kind of data includes images, videos, text, or sound. It is usually stored in data lakes as well as in data warehouses or databases. AI can be used to easily organize and find the required unstructured data. Semi-structured Data: What makes semi-structured data different from structured data is that even though both kinds of data have a consistent model, semi-structured data has a less-rigid format when it comes to sizes. Different observations can have different numbers of values. They can also be of different data types. Examples of semi-structured data are JSON, XML, and YAML which are used in NoSQL databases.


What is SQL? SQL is a Structured Query Language used in the industry for relational database management systems (RDBMS). With SQL, the data is stored in row and column format and can be accessed all at once, in groups, by applying filters, and also can be aggregated. It is easier to learn as it is written in the English language. Data engineers use SQL to create and manage databases. E.g., SQLite, MySQL, PostgreSQL, SQL server, MariaDB