6. Contextual Augmentation Adding or deleting context from the original text results in additional examples of text data, which is known as contextual augmentation. This can be accomplished by modifying the original content by adding or deleting sentences or paragraphs. If the source text is a news article, for instance, we can add or remove sentences to produce new versions of the same item. The nltk package for Python can be used for this. Here is an illustration of how to use nltk for contextual augmentation: In this case, we are altering the original text's context by utilizing the nltk package. The first n_sentences of the text are added to the context using the add_context function, whereas they are removed using the remove_context function. We can generate additional samples of text data that are identical to the original text but have distinct circumstances by employing contextual augmentation.


From Synonyms to GPT-3: The Ultimate Guide to Text Augmentation for Improving Minority Class Labels in NLP

7. Text Generation using Language Models Deep learning models called language models can produce text that is comparable to the input text. We can build a huge number of new examples of text data that are similar to the original text but with various word selections and sentence structures by utilizing language models to generate new text data. GPT-2 or GPT-3 libraries can be used for this. The following is an illustration of how to create text using GPT-3: In this instance, we create fresh text data that is identical to the supplied text using the OpenAI API. The temperature, max_tokens, and n parameters regulate the randomness and length of the created text, while the prompt parameter sets the prompt that will be used for text generation. We can build a huge number of new examples of text data that are similar to the original text but with various word selections and sentence structures by utilizing language models to generate new text data. Additionally, you can add the prompt in CHATGPT as well as in the above code as a prompt as - So that it will generate 5X more data for you.


So the question would arise in your mind, which method should you choose  It can be difficult to select the optimum strategy for text augmentation because results depend on the dataset and the particular problem. There are benefits and drawbacks to various strategies. While rapid, word substitution and synonym replacement could not result in meaningful improvements. Although they demand more resources, generative models like the GPT-3 offer sophisticated variants. Though not always, contextual augmentation can alter meaning. Although it can produce new content, back translation is not always accurate. The most effective strategy for increasing text variations for minority class designations is frequently combining different approaches. The best approach depends on the dataset, and it could take some trial and error to find it.


Some important stats for text augmentation and its effect on models  Machine learning model accuracy can be increased by up to 4% using data augmentation approaches, including text augmentation - Google.Word substitution and synonym replacement as text augmentation approaches can improve the performance of natural language processing models and boost the F1 score by up to 7% - University of California, Berkeley.Generative models like GPT-3 employed for text augmentation can increase machine learning model accuracy by up to 13% - Carnegie Mellon University.Contextual augmentation techniques like adding or removing sentences can improve the performance of machine learning models and raise the F1 score by up to 8% - Carnegie Mellon University.Back translation for text augmentation can increase machine learning model accuracy by up to 6% - IBM. In Conclusion, there is truly a need for text augmentation methodology to deal with minority classes so as to improve the model's performance. There may be other methods also, but combining them with this method would surely benefit a lot. Happy learning ! If you like the content, claps  are appreciated, and follow me  for more informative content


Understanding LangChain : PART 1

What is Langchain? LangChain is an innovative framework that is revolutionizing the way we develop applications powered by language models. By incorporating advanced principles, LangChain is redefining the limits of what can be achieved through traditional APIs. Furthermore, LangChain applications are agentic, enabling language models to interact and adapt to their environment with ease. Langchain constist of few modules. As its name suggests, CHAINING different modules together are the main purpose of Langchain. The idea here is to chain every module in one chain and, at last, use that chain to call all modules at once. The modules consist of the following: ModelPromptMemoryChainAgentsCallbackIndexes Let's start with,


Understanding LangChain : PART 1

Model: As discussed in the introduction, Models mostly cover Large Language models. A large language model of considerable size is a model that comprises a neural network with numerous parameters and is trained on vast quantities of unlabeled text. There are various LLMs by tech giants, like, BERT by GoogleGPT-3 by OpenAILaMDA by GooglePaLM by GoogleLLaMA by Meta AIGPT-4 by OpenAIand many more... With the help of the lang-chain, the interaction with large language models gets easier. The interface and functionality provided by Langchain help to integrate the power of LLMs easily into your work application.LangChain provides async support for LLMs by leveraging the asyncio library. There is also Async support [By releasing the thread that handles the request, the server can allocate it for other tasks until the response is prepared, which maximizes resource utilization.] provided by langchain for the case where you want to call multiple LLMs concurrently and it is known as network-bound. Currently, OpenAI, PromptLayerOpenAI, ChatOpenAI and Anthropic are supported, but async support for other LLMs is on the roadmap. You can use the agenerate method to call an OpenAI LLM asynchronously. You can also write a custom LLM wrapper than one that is supported in LangChain. I used OpenAI for my application and mostly used Davinci, Babbage, Curie, and Ada models for my problem statement. Each model has its own pros, token usage counts, and use cases. You can read more about these models here.


Understanding LangChain : PART 1

Prompts: We all know that prompt is the input that we give to any system to refine our answers to make them more accurate or more specific according to our use case. Many times you may want to get more structured information than just text back. Many new object detection, and classification algorithms which are based on contrastive pretraining and zero-shot learning included prompts as a valid input for results. Just to give examples, OpenAI's CLIP and META's Grounding DINO use prompts as input for predictions. In Langchain, we can set a prompt template according to the answer we want and then chain it to the main chain for output prediction. There is also a facility for the output parser to refine results. Output Parsers are responsible for (1) instructing the model how output should be formatted, and (2) parsing output into the desired formatting (including retrying if necessary). In Langchain, we can provide prompts as a template. Template refers to the blueprint of the prompt or the specific format in which we want an answer. LangChain offers pre-designed prompt templates that can generate prompts for different types of tasks. Nevertheless, in some situations, the default templates may not satisfy your requirements. With the default, we can use our custom prompt template.


Understanding LangChain : PART 1

Memory: Chains and Agents in LangChain operate in a stateless mode by default, meaning that they handle each incoming query independently. However, there are certain applications, like chatbots, where it is of great importance to retain previous interactions, both over the short and long term. This is where the concept of "Memory" comes into play. LangChain provides memory components in two forms. First, LangChain provides helper utilities for managing and manipulating previous chat messages, which are designed to be modular and useful regardless of their use case. Second, LangChain offers an easy way to integrate these utilities into chains. This makes them highly versatile and adaptable to any situation.


Understanding LangChain : PART 1

Chains: Chains provide a means to merge various components into a unified application. A chain can be created, for instance, that receives input from a user, formats it using a PromptTemplate, and subsequently transmits the formatted reply to an LLM. More intricate chains can be generated by integrating multiple chains with other components. LLMChain is considered to be among the most widely utilized methods for querying an LLM object. It formats the provided input key values, along with memory key values (if they exist), according to the prompt template, and subsequently sends the formatted string to LLM, which then produces an output that is returned. One may take a series of steps subsequent to invoking a language model, where a sequence of calls to the model can be made. This practice is especially valuable when there is a desire to utilize the output from one call as the input for another. In this series of chains, each individual chain has a single input and a single output, and the output of one step is used as input to the next.


Understanding LangChain : PART 1

Agents: Certain applications may necessitate not only a pre-determined sequence of LLM/other tool calls but also an uncertain sequence that is dependent on the user's input. These kinds of sequences include an "agent" that has access to a range of tools. Based on the user input, the agent may determine which of these tools, if any, should be called. According to documentation, the High-level pseudocode of agents looks something like:


Understanding LangChain : PART 1

Example: Let's summarize everything in one snapshot attached below. Understanding all modules and chaining is important to build pipeline applications for large language models using Lang-Chain. This was just a simple introduction to LangChain, in the next part of this series, we will work on real pipelines like making pdfGPT, making conversational bots, answering over documents, and other applications. This application-based work will make these concepts more clear. The documentation of LangChain is good to understand but I have added my own thoughts to make it more clear. You can find LangChain's documentation here. [Edit: I added the next parts of LangChain series below so that you don't need to search for it in my profile]


Understanding LangChain : PART 1

If you have found this article insightful It is a proven fact that "Generosity makes you a happier person"; therefore, Give claps to the article if you liked it. If you found this article insightful, follow me on Linkedin and medium. You can also subscribe to get notified when I publish articles. Let's create a community! Thanks for your support! You can click here to buy me coffee. WE CAN CONNECT ON : LINKEDIN  TWITTER  MEDIUM  SUBSTACK  Signing off, Chinmay


This AI newsletter is all you need #46

What happened this week in AI by Louie This week we were interested to see some of the latest advancements in NLP and LLMs being applied to biology and medicine. OpenAI was also in the news again with the gradual rollout of its Code Interpreter plugin (offering a personal data analyst assistant) and rumors suggesting an upcoming release of the 32k token context length GPT-4 variant. We also saw the flow of new Open Source models continue including 3B and 7B RedPajama-INCITE from Together and MPT-7B from MosaicML. Building upon parallels between NLP and single-cell biology, a new preprint introduced scGPT, a large language model tailored specifically for single-cell biology. Trained on a vast dataset of 10 million cells, scGPT demonstrates its ability to learn representations of cells and genes simultaneously. By conditioning existing cells, the model gradually learns to generate gene expressions based on varying conditions. scGPT incorporates a specialized attention masking procedure that determines the order of gene predictions using attention scores. This innovative approach captures gene interactions and establishes an order among genes, mimicking the auto-regressive nature of text generation. We also saw AI applied to vaccine development with scientists at the California division of Baidu Research introducing a novel software called LinearDesign. Drawing inspiration from computational linguistics, LinearDesign aims to design mRNA sequences with more intricate shapes and structures compared to current vaccines, enabling genetic material to persist for longer. We expect to see rapid application of the latest AI models to biology and medicine, as well as to AI assistants to doctors and health professionals. We see the huge positive potential for these products once optimized and approved for use, but as AI continues to advance, particularly in the fields of medicine and biology, the need for robust AI regulation becomes increasingly apparent. And on this note, we saw more movement on the regulatory front this week with a WhiteHouse AI meeting focussed on mitigating AI risks and prioritizing the safety of AI products. The government also released the Blueprint for an AI Bill of Rights and we expect to see more firm AI regulation over the coming months.


This AI newsletter is all you need #46

The HackAPrompt competition by Learn Prompting HackAPrompt, organized by Learn Prompting and advised by Towards AI, is aimed at enhancing AI safety. In this competition, participants will try to hack as many prompts as possible by injecting, leaking, and defeating the sandwich defense. The competition is designed to be beginner-friendly, welcoming even non-technical people to participate. The competition runs from May 5th, 6:00 pm to May 26th, 11:59 pm EST Participate here to win prizes worth over $35,000! Check the HackAPrompt submission tutorial here.


This AI newsletter is all you need #46

Hottest News Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs, by MosaicML MPT-7B is a transformer model trained from scratch on a dataset containing 1 trillion tokens of text and code. It is an open-source model that can be used for commercial purposes and delivers comparable quality to LLaMA-7B. The training of MPT-7B took place on the MosaicML platform, requiring only 9.5 days and costing approximately $200,000, without any human intervention. 2. Releasing 3B and 7B RedPajama-INCITE family of models including base, instruction-tuned & chat models RedPajama releases a model trained on its base dataset. It is a 3 billion and a 7B parameter base model that aims to replicate the LLaMA recipe as closely as possible. The 3B model is the strongest in its class, and the small size makes it extremely fast and accessible. 3. Google Plans to Make Search More 'Personal' with AI Chat and Video Clips Google is undergoing a shift in the way it displays search results by integrating conversations with artificial intelligence, as well as incorporating more short videos and social media posts. This marks a departure from its traditional format of displaying website results, which has been the foundation of its dominance as a search engine for several decades. 4. Microsoft's Bing Chat AI is now open to everyone, with plug-ins coming soon Microsoft is now offering its Bing GPT-4 chatbot to the public, eliminating the need for a waitlist. To access the open preview version powered by GPT-4, simply sign in to the new Bing or Edge using your Microsoft account. 5. The amazing AI super tutor for students and teachers, Khan Academy Sal Khan, the founder, and CEO of Khan Academy, believes that artificial intelligence has the potential to ignite the most significant positive transformation in education. He showcases some exciting new features of their educational chatbot, Khanmigo.


This AI newsletter is all you need #46

5-minute reads/videos to keep you learning Let's talk about ChatGPT with Code Interpreter & Microsoft Copilot With the advancing capabilities of GPT-4 and the imminent widespread availability of Microsoft Copilot, work dynamics are expected to undergo significant transformations within months, rather than years. 2. This Company Adopted AI. Here's What Happened To Its Human Workers The article explores the adoption of AI by a company and its impact on human workers. The implementation resulted in improved efficiency and the transformation of certain job roles, but it did not lead to job losses. Instead, employees were provided with upskilling opportunities to collaborate with AI in their work. 3. A list of open LLMs available for commercial use The list comprises open LLMs that are commercially available, featuring models like T5, Dolly, and specialized LLMs for Code such as Replit Code and Santa Coder. 4. Text-to-Video: The Task, Challenges and the Current State This blog post delves into the past, present, and future of text-to-video models. It begins by examining the distinctions between text-to-video and text-to-image tasks and explores the specific challenges associated with unconditional and text-conditioned video generation. 5. Google "We Have No Moat, And Neither Does OpenAI" This document recently leaked and shared by an anonymous individual on a public Discord server, originates from a researcher within Google. The anonymous individual has granted permission for its republication.


This AI newsletter is all you need #46

Papers & Repositories Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes The paper introduces Distilling step-by-step, a new mechanism that trains smaller models that outperform LLMs, and achieves so by leveraging less training data needed by finetuning or distillation. The method extracts LLM rationales as additional supervision for small models within a multi-task training framework. 2. mlc-ai/mlc-llm: Enable everyone to develop, optimize and deploy Large Language Models natively on everyone's devices MLC LLM is a universal solution that enables the deployment of language models on various hardware backends and native applications. It also provides a productive framework for optimizing model performance according to specific use cases. The system operates locally without the need for server support and utilizes local GPUs on mobile devices and laptops for accelerated processing. 3. Unlimiformer: Long-Range Transformers with Unlimited Length Input This work introduces Unlimiformer, a general approach that can enhance any existing pre-trained encoder-decoder transformer. It achieves this by efficiently distributing the attention computation across all layers using a single nearest-neighbor index. The index can be stored in either GPU or CPU memory and allows for sub-linear time queries. 4. Significant-Gravitas/Auto-GPT-Plugins: Plugins for Auto-GPT This repository contains Auto-GPT plugins, including both first-party (included by default) and third-party plugins (added individually). First-party plugins are recommended for general use and wide adoption, while third-party plugins are more suited for specific personal needs. 5. Are Emergent Abilities of Large Language Models a Mirage? The paper introduces an alternative explanation for emergent abilities, suggesting that when examining fixed model outputs for a specific task and model family, the choice of metric can influence the inference of emergent abilities. The analysis finds strong supporting evidence that emergent abilities may not be a fundamental property of scaling AI models. Enjoy these papers and news summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #46

Upcoming Community Events The Learn AI Together Discord community hosts weekly AI seminars to help the community learn from industry experts, ask questions, and get a deeper insight into the latest research in AI. Join us for free, interactive video sessions hosted live on Discord weekly by attending our upcoming events. ChatGPT and Google Maps Hands-on Workshop @george.balatinski is hosting a workshop to harness the power of ChatGPT to create real-world projects for the browser using JavaScript, CSS, and HTML. The interactive sessions are focused on building a portfolio, guiding you to create a compelling showcase of your talents, projects, and achievements. Engage in pair programming exercises, work side-by-side with fellow developers, and foster hands-on learning and knowledge sharing. In addition, we offer valuable networking opportunities with our and other communities in Web and AI. Discover how to seamlessly convert code to popular frameworks like Angular and React and tap into the limitless potential of AI-driven development. Don't miss this chance to elevate your web development expertise and stay ahead of the curve. Join us at our next meetup here and experience the future of coding with ChatGPT! You can get familiar with some of the additional content here. Date & Time: 2nd June, 12:00 pm EST Add our Google calendar to see all our free AI events!


This AI newsletter is all you need #46

Meme of the week! Meme shared by Recruiter6061#8864


This AI newsletter is all you need #46

Featured Community post from the Discord rmarquet#2048 has developed a Streamlit component that offers a text annotation tool. This tool enables users to highlight text, add annotations, and categorize annotations under various references. Take a look at it here and support a fellow community member! Feel free to share your feedback and questions in the discussion thread here.


This AI newsletter is all you need #46

AI poll of the week! Join the discussion on Discord.


This AI newsletter is all you need #46

Article of the week The Multilayer Perceptron: Built and Implemented from Scratch by David Cullen This article provides a comprehensive overview of the multilayer perceptron, addressing its key aspects through mathematical, visual, and programmatic explanations. The approach taken allows readers to grasp the intuition behind the multilayer perceptron through a step-by-step method presented in bite-sized portions.


This AI newsletter is all you need #46

Our must-read articles MLCoPilot: Empowering Large Language Models with Human Intelligence for ML Problem Solving by Sriram Parthasarathy Generative AI: A Game Changer by Patrick Meyer Beyond Accuracy: How to Enable Responsible AI Development using Amazon SageMaker by John Leung If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.


This AI newsletter is all you need #46

Job offers Senior Software Engineer (Python) @Reserv (Remote) Lead Software Engineer @Charger Logistics Inc (Remote) Intern - Software Engineer @Displayr (Sydney, Australia/Temporary) Software Engineering Manager @Lucidya (Remote) Data Processing Specialist @Hyperscience (Remote) Data Analyst intern @Dott (Remote) Interested in sharing a job opportunity here? Contact sponsors@towardsai.net. If you are preparing your next machine learning interview, don't hesitate to check out our leading interview preparation website, confetti!


How To Get Started With Computer Vision In 2023?

Motivation Computer vision (CompVis) is a field of artificial intelligence (AI) that involves training computers to interpret and understand images and videos. Practical applications of CompVis span from industrial manufacturing robots, self-driving cars, and video surveillance to medical imaging and augmented reality. In many cases, CompVis can automate tasks and saves time and effort for us Neanderthals, which makes it useful for practical applications. Additionally, in some cases, it also outperforms humans, making CompVis a vital tool for many industries. [1] In this article, I'll share a roadmap that you can use to get started with CompVis, either in industry or academia. First, I will share some free and publicly available learning resources. Then I will talk about platforms where you can apply the learned skills to build your portfolio. If you are new or have some experience, this guide can potentially make you even better in this very exciting and rapidly evolving field!


This article is organized as follows: Learning resourcesOnline competitionsIndustry and research collaborations Let's get started!


Learning resources In this section, I'll go over three resources that you should consider taking in order to get a good understanding of the theory as well as practice behind building CompVis systems. This is to increase your depth as a CompVis practitioner. The next two will be those which you should go over to get an idea of the various tasks and learning paradigms in CompVis. This is to increase your breadth. Deep Learning Specialization consists of a total of five courses that will teach you the foundations of deep learning applied to CompVis, natural language processing, etc. It covers both theoretical and practical concepts to build, train and test deep learning models. You'll get to build and train your own models via the course assignments. Take your time to finish all five courses sincerely! CS231n: Deep Learning for Computer Vision deep dives into the details of image classification architectures with a focus on learning end-to-end models. It consists of hands-on assignments which let you implement and train your own CompVis models on a real-world problem of your choice. It also provides details for practical engineering tips and tricks for training and fine-tuning deep learning models. Deep Learning in Computer Vision with PyTorch gives you a quick and easy walkthrough of training and testing image classification and semantic segmentation algorithms on your own datasets. Finally, it shows you how to build and run a simple web interface so that anyone can use your newly trained models. (Shameless self-publicity!) Deep Learning for Computer Vision, Justin Johnson covers implementing, training, and debugging neural networks and provides an in-depth understanding of cutting-edge research in CompVis. It covers CompVis tasks like object detection, semantic segmentation, 3D vision, and generative models, as well as reinforcement learning. Deep Learning in Computer Vision, Prof. Kosta Derpanis is a more recent course that covers a range of topics like action recognition, vision, and language, graph neural networks. It also covers learning paradigms like metric learning and self-supervised learning. Some other learning resources that could be useful to look at: Roboflow tutorials on using SOTA computer vision modelsHugging Face TasksHugging Face Transformers Tutorials There are a lot of code examples in the three links. Once you've done the courses I mentioned above, you'll already know what you need from them, so it will not be overwhelming. Pick your poison!


Online competitions Next, I'll enumerate some previous competitions/challenges you can do yourself and apply your learned skills from the courses mentioned above. This will also help you to get an idea as to how online competitions work (e.g., get data, train models, test and analyze, submit results, and iterate). Then, I'll mention names of competition platforms that also host challenges from popular CompVis conferences where you could possibly start your first online competition! Dogs vs. Cats: An image classification task where you will build a model to predict dogs and cats from images. Flower Classification with TPUs: Similar task as Dogs vs. Cats but many classes. This is known as multi-class image classification. Here you will build a model to classify over 100 types of flowers. Instead of using GPUs, you'll get familiar with using TPUs. Carvana Image Masking Challenge: A semantic segmentation task where the goal is to develop a model to remove the photo studio background from the car. This is similar to image classification but at a pixel level where each pixel is assigned a class label which leads to a final output mask of the desired object (i.e., car). Global Wheat Detection: An object detection problem where the goal is to build a model to localize (e.g., draw bounding boxes) on wheat heads from outdoor images of wheat plants. RSNA STR Pulmonary Embolism Detection: Previous classification tasks deal with 2D images; in this challenge, the goal is to detect and classify abnormalities from chest CT scans which are 3D images. This is 3D image classification. ML Competitions Platforms: The above competitions are hosted on Kaggle, which is the most popular competition platform. There exist other platforms where that host different competitions you could take part in. I'll go over a few: Grand Challenge: Mostly for biomedical imaging problems. Conferences workshops in MICCAI host competitions here.AIcrowd: Businesses, universities, government agencies or NGOs host various challenges. Competitions are also hosted by NeurIPS as workshops. You can also look at CodaLab and Eval.ai. To stay up-to-date with ongoing competitions, see mlcontests. GPU Issues? You've got Kaggle kernels and Google Colaboratory.


Industry and academic collaborations Now in this final section, I'll talk about ways that enable industry and academic collaborations. Once you do a few of the online competitions, they build your intuition on building CompVis systems, as they are mostly based on real-world data. From there, you can either go towards industry to work on business problems or academia to conduct research. Omdena AI: I asked perplexity.ai what Omdena is, and this is what it said: Basically, it is a platform where you get to work with companies on real-world problems. One caveat is that, in the beginning, the work you will do is unpaid. However, as you finish a couple of projects (each with a different company), you build your portfolio and can get into the Omdena Top Talent program, where you get paid to work on projects or even work full-time! As a starter, I think this is the closest you get can work with people in the industry, apart from getting an internship! This is an effective way for someone (even you!) could build experience on real-world problems and break into the industry. Your University: That's right, you're university! This seems very obvious, but I get this a lot. You can collaborate with your university professors, possibly as a research assistant, if you want to focus more on CompVis research and aim for good publications. This worked for me when I first started CompVis research. I'll leave that story for another piece! Here's what you can do. First, narrow down the professors in your university that you'd like to work with. Have a look at their research profile, what topics they work on, and see if you're actually interested in those. Then, email all of them saying you would like to work with them, it is nice to mention what topics. It's alright if you do not hear from most of them. This becomes a bit easy if you already know them in person and have taken their classes; just go to their offices! And that's how you get into academia!


Conclusion In this post, I talked about ways to get started with computer vision as a beginner, and break into the industry or in academia. I mentioned resources to learn the fundamentals of computer vision, as well as platforms to apply your new knowledge via online competitions and even get into industry/academic collaborations. I am currently writing this piece on a layover in Doha as I am traveling from Montreal, Canada to Dhaka, Bangladesh. To people who have asked me "how to get started with computer vision", this one is for you! Good luck.


About the author Aloha! I am a Ph.D. candidate at Concordia University in Montreal, Canada, working on computer vision problems. I also work part-time at Decathlon, where I help build data-driven tools to transform sports images and videos into actionable intelligence. If you're interested to learn more about me, please visit my webpage here.


References [1] Harl, Max., et al. "A Light in the Dark: Deep Learning Practices for Industrial Computer Vision". In arXiv, 2022.


Compare and Evaluate Object Detection Models From TorchVision

Introduction Object detection is one of the most popular applications of machine learning for computer vision. A detection model predicts both the class types and locations of each distinct object in an image. Object detection models have a wide range of applications, including manufacturing, surveillance, health care, and more. TorchVision is a Python package that extends the PyTorch framework for computer vision use cases. But how can you systematically find the best model for a particular use-case? I'm going to be using Comet to log my experiment data (full disclosure: I work at Comet), but feel free to use whatever tooling you prefer. Follow along with the full code in this Colab and check out the public project here!


What is Object Detection Object detection is a computer vision task that aims to identify instances of objects in images and assign them to specific classes. At a low level, object detection seeks to answer the question, "what objects are where?" Object detection algorithms are generally separated into two categories: single-stage (RetinaNet, SSD, FCOS, YOLO, etc.) and two-stage (Fast RCNN, Mask RCNN, FPN, etc.). In two-stage detectors, one model is used to extract generalized regions of objects, and a second model is used to classify and further refine the location of an object. Single-stage detectors do all of this in one step. Single-stage detectors tend to be faster and less computationally expensive than two-stage detectors, but they're also less accurate.


Finetuning Pre-trained Models The best object detection models are trained on tens, if not hundreds, of thousands of labeled images. What's more, image datasets themselves are inherently computationally expensive to process. To train an object detection model from scratch requires a lot of time and resources that aren't always available. To train several object detection models for comparison requires even more time and resources. Thankfully, we don't have to. Instead, we can use transfer learning or fine-tune pre-trained models. In essence, both these methods allow us to take advantage of the weights and biases learned from one task and repurpose them on a new task. By leveraging feature representations from a pre-trained model, we don't have to train a new model from scratch, saving us time and compute resources. What's more, these methods can contribute to rapid boosts in model performance for little overhead. Transfer learning and fine-tuning are similar processes but with one key difference. In transfer learning, all previously trained layers are frozen, and (optionally) additional layers are added for retraining. In fine-tuning, all previously trained layers are retrained, but at a very low learning rate. In both cases, models typically see boosted initial performance, steeper improvement slopes, and elevated final performance.


TorchVision's Pre-Trained Models In real-world applications, we often make choices to balance accuracy and speed. The performance of a model under a given set of circumstances might not be relevant if we aren't able to replicate those circumstances in production. So when looking for the "best" object detection model, it becomes essential to monitor a wide range of metrics pertaining your particular use case. TorchVision's detection module comes with several pre-trained models already built in. For this tutorial we will be comparing Fast-RCNN, Faster-RCNN, Mask-RCNN, RetinaNet, and FCOS, with either ResNet50 of MobileNet v2 backbones. Each of these models was previously trained on the COCO dataset. We will download the trained models, replace the classifier heads to reflect our target classes, and retrain the models on our own data.


Image Data Formats In computer vision, images are represented as matrices of pixel intensity values. Black and white (grayscale) images are usually two-dimensional, and color images are typically three-dimensional, with one "layer" each representing red, blue, and green pixels. Just as there are several ways to represent images, there are also several ways we can represent our labels and predictions. In the full code for this tutorial, we'll provide methods for logging bounding boxes, segmentation masks, and polygon annotations to an experiment tracking tool. But when comparing our torchvision models we will only use bounding boxes, as not all of our models are able to calculate the other types of predictions. To further complicate things, not all algorithms format bounding box annotations in the same way. Below we've listed a few of the most common bounding box formats you're likely to run into, but we'll be focusing on Pascal VOC and COCO formats in this tutorial. Pascal VOC: [xmin, ymin, xmax, ymax] -> [98, 345, 420, 462] Albumentations: normalized([x_min, y_min, x_max, y_max]) -> [0.153125, 0.71875, 0.65625, 0.9625] COCO: [xmin, ymin, width, height] -> [98, 345, 322, 117] YOLO: normalized([x_center, y_center, width, height]) -> [0.4046875, 0.8614583, 0.503125, 0.24375]


Evaluation Metrics for Object Detection A naive approach to evaluating object detection models might be binary classification ("match" or "no match", "1" or "0"), but this method leaves little room for nuance. We can do better!


Intersection Over Union (IoU) The standard evaluation metric for comparing individual bounding boxes is Intersection over Union, or IoU. IoU evaluates the degree of overlap between the ground truth bounding box and the predicted bounding box with a value between 0 and 1. IoU is determined for each set of bounding boxes in an image and then a threshold is applied. If an IoU meets the threshold, it's marked as a "true positive." All predictions not marked as "true positives" are marked as "false positives," and any items left in our "ground truth" annotations list are marked as "false negatives. The decision to mark a detection as TP, FP, or FN is completely contingent on the choice of IoU threshold. The IoU threshold is commonly set at 0.5, but you may want to experiment with this number. Once we've calculated our confusion matrix, we can compute precision and recall.


Mean Average Precision (mAP) and Mean Average Recall (mAR) Precision (also known as specificity) is the degree of exactness of the model in identifying only relevant objects. The equation for precision is: Recall (also known as sensitivity) measures the ability of the model to detect all ground truths. The equation for recall is: In a perfect world, our perfect model would have a precision and recall of 1, meaning it predicted zero false negatives and zero false positives. But in the real world this isn't generally achievable. A precision-recall curve plots the value of precision against recall for different confidence thresholds. The area under this curve is also referred to as the Average Precision (AP). Average recall (AR) describes double the value of the area under the recall-IoU curve. Mean Average Precision (mAP) and mean Average Recall (mAR) are calculated by taking the weighted mean of the AP or AR over all classes and/or over all IoU thresholds. They are two of the most common evaluation metrics for object detection and are used to evaluate submissions in popular computer vision competitions like the COCO and Pascal VOC challenges. We can derive many other metrics from mAP and mAR, including mAP across scales, at different IoU thresholds, and with a minimum number of detections per image.


Other metrics If we were building a model to detect very large objects (relative to the image's field of view), we might be willing to consider models with poor "AP_small" scores, as this metric would be less relevant to our use case. If we were planning on using our model to aid in medical diagnoses, we might place a higher emphasis on mAR values than mAP values, since it would likely be more important not to miss any positive samples than it would be to miss negative samples. For this tutorial we use a combination of the mAP and mAR values calculated by the COCO evaluator and torchmetrics.detection module. We'll also log all relevant values to a DataFrame to give us a fuller picture of how different models perform in different scenarios. Finally, we'll choose our "best" model accordingly.


Challenges of Comparing Object Detection Models Comparing object detection models can be challenging for a number of reasons. Different models have different requirements when it comes to input size and shape, annotation formats, and other dataset attributes. Hyperparameters vary from algorithm to algorithm and keeping track of which values produce which results can quickly become tedious and overwhelming. Most computer vision pipelines incorporate image augmentation in some form, so dataset versioning becomes essential for reproducibility and explainability. What's more, performance metrics only tell part of the story when it comes to object detection. Often, it's necessary to visualize prediction annotations to understand where things are going right and where they're going wrong. What's more, image datasets themselves are inherently computationally expensive to process. To train an object detection model from scratch requires a lot of time and resources that aren't always available. To train several object detection models for comparison requires even more time and resources. In real-world applications, we often make choices to balance accuracy and speed. The performance of a model under a given set of circumstances might not be relevant if we aren't able to replicate those circumstances in production. So when looking for the "best" object detection model, it becomes essential to monitor a wide range of metrics pertaining your particular use case.


Using Comet for Object Detection Clearly, comparing object detection models isn't nearly so simple as just minimizing a single loss function. We have a pretty wide range of metrics to calculate and log, some of which need to be visualized to completely understand, and each model has it's own graph definition, set of hyperparameters, code output, and other features. To help keep track of all of these moving pieces, we'll log our inputs, metrics, and outputs to Comet, a experiment tracking tool. By visualizing our data in an experiment tracking tool, we'll be able to get a much more complete understanding of how each of our models behaves, under which circumstances, and with which data. For this tutorial, we'll be using the Penn-Fudan dataset, which consists of 170 images labeled with 345 instances of pedestrians. Pedestrian detection has several applications, including surveillance, training self-driving cars, and other traffic safety applications. Since we're using PyTorch, we'll need to define a custom dataset class that inherits from the torch.utils.data.Dataset class. All of the models in TorchVision's detection module use Pascal VOC format, so we'll format our bounding boxes accordingly in our Dataset class. We'll then need to convert the model's prediction labels from Pascal VOC to COCO format for use with the COCO evaluator and Comet. If you're using this tutorial with your own model, check your specific model's annotation requirements to ensure proper formatting of all inputs.


Single Experiment View In order to get a good understanding of how each of our models is performing, and with which hyperparameters, we'll start by examining our results at an experiment-level.


Store Hyper-parameters Keeping track of our hyperparameters is essential for reproducibility and explainability. Model hyperparameters can affect model performance, computational choices and what information to retain for analysis. Hyperparameters vary from algorithm to algorithm, and some are more important than others, so this critical task can quickly become tedious and confusing. Here, we log important hyperparameters with just a single command. For our project we'll be monitoring the following hyperaparameters, which we can adjust by simply editing the relevant keys-value pairs: Sometimes metrics that work well for one model may not work at all for another. For example, the FCOS model tends to struggle with exploding gradients. When using it, we have to significantly decrease the learning rate to accommodate for this. If, however, we use the reduced learning rate on a model like Fast-RCNN, (typically one of our best-performing models), it performs unusually poorly because it fails to ever really "learn" the feature maps of our dataset. Since we are focusing on comparing different models in this tutorial, we will mostly be keeping the hyperparameters constant (except for learning rate). However, if we were looking to optimize the hyperparameters of a single model, we could also pass a list of values to each hyperparameter key and use an optimizer to iterate through them.


System Metrics Since object detection is such a resource-heavy task, we'll also want to monitor our system metrics, including CPU and GPU usage. This can also help diagnose bottlenecks in our pipeline, aid with reproducibility, and debug crashed experiments.


Evaluation Metrics Each of our PyTorch detection models comes with relevant evaluation metrics built-in. Comet is integrated with PyTorch, so each of these pre-defined metrics will be automatically logged to the experiment. This is very helpful when comparing multiple runs of the same model, or different object detection models with the same evaluation metrics, but the PyTorch models we've chosen don't all come with the same built-in metrics. We'll still use these auto-logged plots to get an initial impression of the performance of our models, but we'll want to log some of our own metrics for cross-experiment comparisons. We have the ability to manually log just about any metric, asset, artifact, or graphic we want. In this tutorial, we'll track: Mean Average Precision (mAP) of all validation images per epochMean Average Recall (mAR) of all validation images per epochTorchMetric's 12 metrics for characterizing the performance of an object detection (very similar to COCO's 12 metrics listed above) per imageRelevant code files from TorchVision (engine.py, transforms.py, etc.)Graph definitions of our various modelsEach image in the validation dataset, as well as our model's predicted bounding boxes, with their corresponding labels and confidence scores.


Graphics Tab Understanding where your model is going right and where it's going wrong can be especially difficult with image datasets. Loss metrics and other numerical values don't always tell the whole story and can be hard to visualize. So we'll also log each of our validation images, along with their predicted bounding boxes, per model per epoch. Flipping through a model's predictions can also be helpful to see how our models improve over time. To log images to Comet, we simply use the log_image method: Alternatively, we can also pass the annotations to the metadata parameter: In either case, the image annotations should be in JSON format and the bounding boxes should be in COCO format. Bounding boxes can either be passed as a dictionary (as shown below) or as a list of lists. Note that a new instance should be created for each bounding box and polygon points are passed in the format [x1, y1, x2, y2, ..., xn, yn].


Project Level View Since we're comparing object detection models in this tutorial, one of the most important ways we can use a tracking tool is to create a holistic project-level view. Comet automatically generates a basic model performance panel, but we also have the ability to customize our panels for our particular use case.


Image Panel The image panel allows us to visualize different models' prediction per experiment run, over time. Use the step slider to walk through each model's predictions, or click on an individual image for a closer look. From there, choose to smooth your image or render it in grayscale, select which class labels you want to examine, and set confidence thresholds with a sliding bar.


Data Panel Sometimes we really want a deep dive into the numbers. With Comet's Data Panel we can log any csv, DataFrame or table to our project and explore it interactively in the UI. We logged all twelve evaluation metrics from TorchMetric's mean_ap module, as shown below. If a given metric isn't relevant to a particular image, it's given a value of -1 (for example, if an image doesn't predict any "large" bounding boxes, then mAP_large for that image will be -1). We can reorder columns, sort them, and filter values. Below, we compare our most basic mAP and mAR measures and then sort them to see where precision is very different from recall. Alternatively, we could also check the epoch f1-score that we logged as an additional tool in our toolbox.


Multiple Dashboards Now that we've built all of these panels, we need a way to keep them organized! For this, we build and save multiple dashboards, each of which we'll use for a different purpose. We'll keep the auto-generated dashboard that Comet built for us, and we'll organize the rest of our panels into four more dashboards. We have a project overview dashboard that gives us a very basic overview of our project's stats (parameters used, number of experiments run, and some of the best metrics achieved). We'll put our image panel and data panel into a Debugging dashboard and we'll store our plots and charts in a Metrics dashboard. Now we can easily navigate through all of our panels to find exactly what we're looking for!


Accuracy-Speed Tradeoff At the beginning of this tutorial, we briefly explored machine learning's accuracy-speed tradeoff. Models with higher precision and accuracy tend to consume more compute resources, and fast models tend to be be less accurate. Depending on your use case, your definition of the "best" model may vary. Circling back to this thought, we'll compare four of our models in terms of their general speed and accuracy in order to understand which models work "best" for which scenarios. We create a final dashboard called "Accuracy-Speed Tradeoff" and plot some basic evaluation and system metrics for four different models: Mask RCNN, Fast RCNN, RetinaNet, and FCOS. Remember that both RCNN models are two-stage object detection models, which are generally more computational expensive. RetinaNet and FCOS are both single-stage models. Both of our two-stage object detection models (in green and light blue above) far out-perform the single-stage models in mean average precision, epoch f1-score, and loss. Shifting to the bottom row of charts, however, we can see that they are also much more computationally-expensive. It may come as no surprise that Mask RCNN is the slowest model of all, because it's actually based on Fast RCNN, but produces additional outputs (segmentation masks). For a general purpose object detection model, we might conclude that Fast RCNN performs the best with bounding box prediction. It has the highest mAP and f1, the lowest loss, and consumes far less memory than Mask RCNN. But the "best" model is subjective and entirely dependent on your use case! If we were looking to deploy our model to a mobile device, Fast RCNN's memory requirements might disqualify it from our consideration.


Conclusion Comparing and logging object detection models can be a tedious and overwhelming task, but when you have an experiment tracking tool like Comet, you can focus your attention where it really matters. Comet is a powerful tool for tracking your models, datasets, and metrics to keep your experiments organized, reproducible, and explainable. Try out the code in this tutorial in this Colab and apply it to a dataset of your own! You can view the public project here or, to get started with your own project, create an account here for free! If you liked this post, check out some of my other Medium articles below:


SAM: A Image Segmentation Foundation Model

Architecture The promptable segmentation requires a model that supports flexible prompting and computes masks in real-time to allow interactive use and must be ambiguity aware. Hence, SAM (Segment Anything Model) has proposed that contains 3 components, image encoder, prompt encoder, and mask decoder. Image Encoder:· It uses MAE pre-trained Vision Transformer (ViT). A heavyweight Image encoder on the server side computes image embeddings once per image that can be used any number of times with different prompts to avoid computational overheads. The image encoder has 632M parameters.Prompt Encoder: Sparse prompts like points, and boxes are embedded using positional encoding, and text uses CLIP embeddings. These learned embeddings are summed up. Dense prompts like masks are embedded using convolutions summed element-wise with image embeddings. The prompt encoder and mask decoder has 4M parameters.Mask Decoder: The mask decoder efficiently maps the image embedding, prompt embeddings, and an output token to a mask. It uses a modified Transformer decoder followed by a dynamic mask prediction head. If the given prompt is ambiguous, at output 3 masks are generated (whole, part, and subpart) for a single prompt with a corresponding confidence score to resolve ambiguity.With precomputed image embeddings lightweight mask decoder and prompt encoder take ~55 ms on a web browser without GPU for implementation.For training, a linear combination of focal loss and dice loss is used to supervise mask prediction.


Data Engine To train the SAM model, the segmentation masks are not abundant on the internet, hence data engine is created to collect a 1.1 Billion mask, SA-1B dataset. It has 3 stages: Assisted manual stage: The model is trained with public segmentation dataset. The output masks are corrected manually and after gathering sufficient corrected data model is re-trained. The model is trained 6 times.Semi-automatic stage: Additional unlabeled images are annotated in much more detail to improve diversity. The periodic retraining takes place 5 times.Fully automatic stage: As the model was able to predict valid masks in ambiguous cases, it is prompted with a 32 X 32 grid of points. For each point set of masks predicted, including valid objects. The confident and stable masks are selected and NMS is applied to remove duplicates. To improve the quality of small masks, overlapping zoomed-in image crops are processed.


Dataset (SA-1B) It contains 11 M diverse images (3300×4950 pixels resolution and downsampled to 1500 pixels for storage challenges) and 1.1 B masks (99.1% of masks are fully automatically generated).


Zero-shot Transfer Experiments By engineering appropriate prompts SAM can resolve downstream tasks. In addition to automatic labeling, SAM performs segmenting an object from a single foreground point, a low-level task of edge detection, a mid-level task of object proposal generation, a high-level task of instance segmentation, and a higher-level task of segmenting objects from the free-form text as a proof-of-concept. Let's check instance segmentation with SAM. For high-level tasks of instance segmentation, SAM can be used as a segmentation module. For this, we need to first run the object detector and the output of the object detector passes as a prompt to a SAM module. Here, yolov8 is taken as an object detector trained on the food dataset to detect classes: croissants, cookies, and cupcakes. 2. The output bounding boxes are fed as a prompt to a SAM module to do zero-shot segmentation. Refer code on the Github repo.


Summary A SAM can segment objects by clicking points to include and exclude the object. It also uses a bounding box as a prompt,To avoid ambiguity, multiple valid masks are generated when the prompt provided is ambiguous.It can automatically detect all objects from the image and mask them.As image embeddings are precomputed, SAM can generate masks for any prompt in real time.SAM can be prompted with gaze points detected by a wearable device, enabling new applications. It can become a powerful component in domains such as AR/VR, content creation, scientific domains, and more general AI systems.


You've Been Brainwashed: ChatGPT Is Stupider Than You Think!

Limitations of AI and the Myth of Scale: AI has grown tremendously in recent years, with some models, like ChatGPT, trained on tens of thousands of GPUs and a trillion words. These extreme-scale AI models seem to demonstrate sparks of AGI, but they often make small, silly mistakes. First, the extreme expense of training large-scale AI models leads to a concentration of power in a few tech companies, making independent research difficult. Second, the massive carbon footprint of these models has significant environmental impacts. Finally, AI without robust common sense will never reach the potential you think. Models such as ChatGPT have reached their peak. Let's take a look at the graph below. It demonstrates the size of these models, displaying how many parameters they have, and also plotting their accuracy, i.e., the model's performance. If we draw a curve of best fit through this, it will be even more apparent that there is a cap to the number of parameters that will bring about a massive increase in performance. It's not a linear relationship. So now we've established massive change isn't coming using language models such as ChatGPT; let's discuss ChatGPT itself.


What is Intelligence, Then? To understand why AI is not as intelligent as it may seem, we must first define intelligence. Intelligence requires explanation based on a causal understanding of the world and not just prediction. Human intelligence is not just predictive processing because it also excels at pattern recognition, explanation, evaluation, selective memory, and communication. Which ChatGPT doesn't - so let's look at why.


Why Does ChatGPT Lack Intelligence? ChatGPT, while an impressive language model, lacks the common sense and understanding of the context that characterizes human intelligence. This is primarily because it relies on pattern-matching from the vast amount of data it has been trained on, rather than understanding the underlying concepts. In short, how it works is that incredible amounts of text are passed into the AI model and it learns the relationship between these words. Let's look at an example. Here we can see the percentage probability of a corresponding word occurring after the previous, and of course, full stops and just stopping have corresponding probabilities as well. As a result, it often produces answers that sound plausible but are incorrect or nonsensical when analyzed closely. One reason for this shortcoming is the absence of a proper grounding in reality. While ChatGPT is trained on a massive amount of text data, it does not have direct experiences or interactions with the real world: it's just these probabilities. Consequently, it cannot form a coherent understanding of the world and is unable to reason about it effectively.


Can ChatGPT Be Paired with a Logic Machine, to Create Intelligence? So we know that ChatGPT lacks intelligence but is impressive with language, so why can't we add the intelligence on? And yes, it is possible to combine the capabilities of ChatGPT with a more logical system, like Wolfram Alpha, to create a more intelligent AI. Wolfram Alpha specializes in computational knowledge and can process complex mathematical and scientific queries. By combining the strengths of both systems, a more intelligent and versatile AI could be created. However, this would still not constitute AGI, as the system would still have limitations and lack the full range of human-like understanding and reasoning. For such a hybrid system to be effective, it would need to overcome several challenges. It would require seamless integration between the language model and the logical reasoning engine. This would necessitate a shared understanding of the context and meaning of user queries, as well as a mechanism for coordinating the capabilities of both systems. And honestly, this is where my doubts lie. Logic models, Wolfram Alpha, will have trouble dealing with such inconsistent LLMs, as it requires consistent inputs.The hybrid system would need to be capable of learning and adapting over time. This could involve incorporating feedback from user interactions, updating the knowledge base, and refining the reasoning and language generation capabilities. Ultimately, the goal would be to create a system that can truly understand and respond to complex queries in a way that demonstrates intelligence rather than simply regurgitating pre-existing knowledge.


The Future of Large Language Models (LLMs) if Not AGI: Even if LLMs don't achieve AGI, they still have significant potential for practical applications. They can be used in areas such as customer support, content creation, translation, and many other fields. By refining the technology and addressing its limitations, LLMs can become increasingly valuable tools for a wide range of industries. One potential avenue for improving LLMs is to focus on incorporating other types of data, such as visual or auditory information, into their training. This could help AI models develop a more comprehensive understanding of the world and improve their reasoning and problem-solving capabilities. Additionally, incorporating mechanisms for learning and adapting over time could help make LLMs more contextually aware and better able to engage in meaningful interactions. Another critical area of focus should be on ensuring that AI models are ethical, safe, and aligned with human values. This will involve developing better methods for curating and filtering training data, as well as designing algorithms that can recognize and correct biased or harmful outputs.


Conclusion Although I have slandered ChatGPT massively throughout this article, it's not all bad. As I've mentioned, it's text-generating abilities are unmatched and can blow people away, and this is where the danger lies. But hopefully, in this article, I've persuaded you otherwise and given you a glimpse of what is currently going on in the AI world behind all the flash media headlines.


The Art of Prompting: How Accurate Prompts Can Drive Positive Outcomes.

Principals used for promoting: Write clear and specific instructions: Output: Output: Output: Output: When giving Text 1 in step format, the model provides steps for preparing tea. In contrast, Text 2 without step format leads to the model outputting "No steps are provided for the given text." Providing an appropriate prompt enables the model to check conditions from the text. Output: 2. Give the model time to think. Output: Output: If you can read the question and the student's solution carefully, then you will notice that the student's solution was incorrect. You can fix this by instructing the model to work out its own solution first and then compare it with the student's solution. Below is the correct prompt that one should give: Output: Here, you can see model gives the correct output that is student's solution was incorrect. Conclusion: It is most important to provide accurate and precise prompts in order to ensure desired outcomes. The provision of correct prompts plays a pivotal role in achieving success and fulfilling predetermined objectives. Therefore, it is imperative to invest time and effort into crafting prompts that are tailored to meet the intended purpose and produce optimal results. Next, I'll cover the remaining topics including text summarizing, transformation, expanding, and chatbots. Stay tuned !!! If you found this article insightful, follow me on LinkedIn and Medium. Thank you !!! You can read my other blogs:


Most Data Annotation Bpos Are Missing the Point Amid the GPT LLMS Wave

Let me share a few examples Automation brings economies of scale and works opposite to the economics of outsourcing to developing countries while speed makes automation even more lucrative and outsourcing can't provide nearly the same level of quality. However, if both humans in the loop and automation are both married such that each is packaged together in such a way to leverage each other strengths then it can become the key competitive advantage What Humans in the loop in data annotation can do better now? Offer the expertise that automation still yet to learn or where it doesn't make economic sense to train AI while also considering the environmental impactHuman workers can bring a level of nuance and context to data labeling tasks that may be difficult for AI to replicate, particularly in situations where the data is complex or ambiguous.Human workers can adapt more easily to changing circumstances or unexpected situations, whereas AI may struggle when faced with novel or unexpected data What automation in data annotation can do better now? Annotate large types of annotations, especially classification, and content moderation as a first opinion.A significant percentage of data annotation manual work for NLP and computer vision and other types of AI is not going to compete with the costs offered by automation for the same level of decent quality enough to reduce the dependency on the human in the loop.Automation can help to standardize the annotation process, ensuring that all data is labeled consistently and reducing the potential for human error or bias. This can be particularly important in industries such as healthcare or finance, where accuracy and consistency are critical.Automation can be used to identify and flag potential issues or anomalies in the data, allowing human workers to focus their efforts on resolving these issues rather than spending time on routine tasks. This can help to improve the overall quality of the data and reduce the risk of errors or inconsistencies. What tool companies should do more? Companies focused on automation and its application should continue to invest deeper into this and increase the velocity of everyday new tech evolving and focus on innovating the User experience and enabling better consumerism of the tech and R&D possibly in the following ways:- Labelbox, Labellerr, CVAT, Scale AI, Appen, and others, are all leaders in the data annotation space and continue to innovate in areas such as quality control, model training, and data security.leveraging emerging technologies such as blockchain and federated learning to enhance data privacy and security, and exploring new applications of automation in fields such as autonomous vehicles, robotics, and smart cities.Partnering with other companies in the ecosystem, such as cloud providers(AWS, GCP, Azure, and on-prem) and tooling platforms, to create end-to-end solutions that can streamline the data annotation process and make it more accessible to a wider range of industries and use cases. What Data annotation BPOs should do more? Investing in talent can help companies to stay ahead of the curve when it comes to emerging technologies and best practices in data annotation. By hiring skilled workers who are up-to-date on the latest developments in the field, companies can ensure that their annotation processes are optimized for efficiency and accuracy.Talent can also play a crucial role in identifying and addressing potential biases or errors in the annotation process. By having a diverse team of experts with different perspectives and experiences, companies can ensure that their data is labeled in a fair and consistent manner and that any issues are quickly identified and resolved.In addition to investing in talent, companies should also focus on creating a culture of continuous learning and improvement. By encouraging employees to stay up-to-date on the latest developments in the field, and providing opportunities for training and professional development, companies can ensure that their annotation processes are constantly evolving and improving over time.


Finally, the most important question! What should a new-age state-of-the-art AI data preparation workflow look like? Data Management - The AI team connects their NLP and computer vision data from the cloud(say S3, API, GCS, Azure blob store) and manages the datasets via data management interfacesFlag bad data - Because the system will already have annotated datasets and previous knowledge and models, it will suggest what data is noisy, and invalid according to the AI use case in seconds and minutes. This if provided to the large army of outsource workers would require a huge amount of effort and time and still miss the context.Pre-labeling - Further, it will do pre-labeling according to mentioned prompts or use cases. Till now a significant part of the offering by BPOs was this part and now it is reduced to none in the majority of the use case such as content moderation, receipt annotation, NER tags, classifying documents, people counting, surveillance, etc <TBD>.Guided data QA - Then a system will generate analytics and guided workflow where it really requires human assistance or scenarios where it is not confident.Build benchmark AI models - This smart review capability is left to such a small volume that has an in-house team with better context and data residing in-house becomes much more viable than outsourcing the work while allowing access to outside folks and still less context. In the meantime, AI teams can start building models while they explore outsourcing the pending unlabeled or difficult data.Outsource only if required where it matters - Now finally there possibly 510% of data is left where the system doesn't know where it is wrong or scenarios where it is not confident and feedback loop from customers where it is going wrong. These are the cases that require a review and could lead to the creation of a human-in-the-loop requirement. Humans in the loop still play a crucial role in achieving the highest levels of accuracy and quality in data annotation. Tool companies need to focus on developing better technology that combines the strengths of both humans and automation to create a powerful hybrid approach. By leveraging the speed and efficiency of automation with the human touch, this model can deliver the best possible results. To keep up with the rapid changes in the industry, BPOs, and data engine platforms need to collaborate and offer AI as a service. Failure to adapt to these changes can result in being left behind in the progress of the industry. Investing in talent and staying ahead of the curve by incorporating the latest technologies is the key to success. With the right strategy and approach, companies can stay ahead in the data annotation game and achieve limitless possibilities. It's important to address the current and evolving limitations of technology while innovating and collaborating to create a better future for data annotation. My final thought on this! As a leader in AI data annotation, you must be aware that a game-changer like labelgpt is not just another tool in the market. It offers a lot more than just basic data annotation, making it an ideal choice for BPOs looking to increase their revenues by up to 10 times or retain their clients.


It makes little sense for BPOs to invest in building a tool from scratch when collaboration or strategic partnership can offer a more efficient and competitive way forward. In my upcoming article, "Why labelGPT is not just another data annotation tool," I will discuss the unique features and benefits that set it apart from other tools in the market. Let's connect over Linkedin as I write interesting and new aspects in computer vision data preparation, data ops, data pipelines, etc., and I am happy to chat on the same. Only technical deep dive!


MLCoPilot: Empowering Large Language Models with Human Intelligence for ML Problem Solving

Storing past ML insights to guide decision making ML insights from the past play a crucial role in guiding the large language models (LLMs) to answer future machine learning (ML) tasks. To facilitate this, vector databases like Pinecone come into the picture. These databases store all the necessary knowledge required by the LLMs, essentially serving as their memory. Vector databases are specialized databases designed to efficiently store and retrieve high-dimensional information, such as vectors. They are commonly employed in systems that provide recommendations, find similar images or text, and group similar items together. When given a query like "classify brain tumor," the vector database can search for documents or phrases that have similar meanings to the query. It achieves this by comparing the vector representation of the query with the vectors of the stored documents, which encompass past experiences and accumulated knowledge. By identifying the closest matches in semantic space, LLMs can retrieve relevant documents based on their semantic similarity, moving beyond the limitations of keyword matching alone. For instance, if a brain tumor ML task is presented, the LLM can potentially find a related experience or knowledge related to a lung tumor ML task from the knowledge base. By leveraging the similarities between these tasks, the LLM can access information about the algorithms used, the datasets employed, and other parameters and weights utilized in the previous ML task. This retrieved information then becomes invaluable input for the LLM as it endeavors to solve the new ML task at hand. By utilizing vector databases to store and retrieve ML insights from the past, LLMs gain access to a vast repository of knowledge. This empowers them to make informed decisions, draw upon established methodologies, and leverage successful solutions that have been previously applied to similar ML tasks. As a result, the LLM's capabilities are significantly enhanced, enabling it to provide more accurate and efficient solutions to a wide range of ML problems.


Leveraging past knowledge to solve ML tasks The primary objective of ML CoPilot is to harness both machine intelligence and human knowledge to propose solutions for new machine learning (ML) tasks. Achieving this goal requires ML CoPilot to learn from the past experiences of human practitioners. To accomplish this, the authors of ML CoPilot explored ML benchmarks to extract valuable information, which was then converted into text format for better comprehension by the large language models (LLMs). Text format is utilized because LLMs excel in understanding textual data rather than numerical values. Therefore, any information that the LLM needs to read and learn must be converted into a text-based format. This includes converting code into text summaries and transforming accuracy numbers into descriptive categories such as medium, good, and so on. These experiences are aggregated and transformed into knowledge, forming a repository of ML insights. When a new ML task is presented, ML CoPilot leverages the accumulated experiences and knowledge to suggest the final solution. By drawing upon past ML experiences, ML CoPilot enhances its ability to provide valuable recommendations and guidance for the new task at hand. This collaborative approach effectively combines the strengths of machine intelligence and human expertise, leading to more effective and efficient solutions for novel ML tasks. The next two sections explore the two steps below.


Step 1: Create an ML knowledge pool from historical ML tasks (from benchmark data) To facilitate the learning process from previous machine learning (ML) work, three ML benchmarks, namely HPO-B, PD1, and HyperFD, were employed. These benchmarks encompass a wide range of ML tasks and datasets, spanning various scenarios such as tabular data classification and regression, image classification, and object detection. The primary objective is to convert these benchmarks into a knowledge format that can be utilized by large language models (LLMs). HPO-B-v3, for instance, consists of 16 different sets of solutions for 101 datasets sourced from the OpenML platform. Each set focuses on a specific machine learning algorithm, such as random forest, SVM, or XGBoost. The main goal is to identify the optimal configuration for these algorithms when applied to a particular dataset. By exploring various configurations, the benchmarks aim to discover the best-performing settings for different ML algorithms and datasets. PD1 Neural Net Tuning Dataset, on the other hand, comprises diverse classification tasks that encompass areas like image classification, next token prediction, and translation. Each task is associated with a specific neural network architecture, such as CNN or transformer. The dataset concentrates on fine-tuning the performance of these neural networks by adjusting the parameters of an SGD optimizer with Nesterov momentum. Through this fine-tuning process, the benchmarks aim to optimize the neural networks for improved performance across different classification tasks. HyperFD represents a benchmark system designed to enhance the accuracy of a neural face detector on a novel dataset. This is achieved by appropriately adjusting various aspects, including data augmentation, neural architecture, loss function, and training recipe. By optimizing these elements, the benchmarks aim to enhance the performance of the neural network, specifically in the context of face detection tasks. The purpose of utilizing these ML benchmarks is to extract valuable knowledge from their outcomes and convert it into a format that LLMs can effectively utilize. By capturing the insights and experiences gained from these benchmarks, the LLMs can leverage this knowledge to provide valuable suggestions and solutions for new ML tasks. The benchmarks offer a diverse range of scenarios and datasets, enabling the LLMs to gain a comprehensive understanding of ML challenges and their corresponding solutions.


Step 2: Suggest solutions for ML tasks based on knowledge from the past When ML CoPilot receives a new task, it searches the experience pool for related tasks. For instance, if the task is to classify a brain tumor, ML CoPilot looks for ML tasks that are related to this domain. One example of a related task could be the classification of lung tumors. ML CoPilot also explores related knowledge recommendations associated with these tasks. The gathered information is then provided to the large language models (LLMs) and utilized to generate the final solution for the given problem. By leveraging the experiences and knowledge derived from similar ML tasks, ML CoPilot enhances its ability to provide accurate and effective solutions for the new task. The presence of ML experiences and knowledge is crucial for the proper functioning of this system. Without either of them, or in the absence of both, the reliability of end-to-end predictions diminishes. The availability and incorporation of ML experiences and knowledge serve as the foundation for ML CoPilot to provide accurate and dependable solutions to novel ML tasks.


Conclusion In conclusion, the collaborative approach of ML CoPilot, which combines the power of large language models (LLMs) with the accumulated knowledge from past machine learning (ML) experiences, offers immense potential in solving novel ML tasks. By leveraging vector databases to store and retrieve insights from ML benchmarks, LLMs can gain access to a wealth of information that guides their decision-making processes. ML CoPilot serves as a bridge between machine intelligence and human knowledge, incorporating the expertise of experienced practitioners by using established methodologies and successful approaches used in similar ML tasks. However, the success of this system heavily relies on the availability of ML experiences and knowledge. The absence of either one or both of these components undermines the reliability of end-to-end predictions. Therefore, the continuous accumulation and storage of ML insights become critical in enhancing the effectiveness and dependability of ML CoPilot. In conclusion, by leveraging past experiences, knowledge, and the expertise of the ML community, ML CoPilot opens up new possibilities and advances the field of machine learning towards more intelligent and effective solutions.


Chat With Your Data To Build ML-Driven Customer Segments Using a Chatbot Built With ChatGPT and LangChain

Data set used The Mall Customers data contains information about customers, including their unique CustomerID, Gender, Age, Annual Income, and Spending Score. These columns provide insights into the demographic profile of the customers, their age distribution, income levels, and spending patterns. The dataset can be analyzed to explore relationships between variables such as age and spending score, gender and spending behavior, or income and spending patterns. By understanding these aspects, businesses can gain valuable insights to tailor their marketing strategies and enhance the customer experience at the mall.


Questions we want to ask The following questions will help us explore the characteristics and behaviors of mall customers and provide insights into their spending patterns, demographics, and potential segmentation. From there, you can dive deeper into specific areas of interest or business objectives to uncover further insights from the dataset. Some of the questions are pure manipulation of the data and some of the questions needs to build predictive models behind the scenes. What is the gender distribution among mall customers?How does age vary among mall customers? Are there any dominant age groups?What is the range and distribution of annual incomes for the customers?Is there a correlation between age and annual income?How is the spending score distributed among the customers?Are there any differences in spending patterns between male and female customers?Which age group has the highest spending score on average?Is there a correlation between annual income and spending score?Can we identify any customer segments based on their gender, age, income, and spending score?Can we predict customer spending behavior based on demographic variables such as age, gender, and income?Are there distinct customer segments within the mall based on their shopping preferences and behaviors?Can we identify key factors that influence customer satisfaction or loyalty within the mall?Is there a correlation between customer demographics and their preferred shopping locations within the mall?Can we predict customer churn or attrition within the mall?What are the main drivers behind customer spending patterns and how can we leverage them for targeted marketing strategies?Can we recommend personalized product offerings or promotions based on customer segmentation and past purchase history?Can we identify potential high-value customers within the mall based on their spending habits and demographic characteristics?


Packages / Technologies used LangChain is a framework that allows you to build advanced applications powered by large language models (LLMs). It provides a standard interface to connect different components such as LLMs, data sources, and actions. LangChain makes it easier and faster to build powerful applications with features like chaining, data awareness, and agentic capabilities. At an high level, the agent functions by invoking the Python agent in the background, which in turn executes Python code generated by the LLM (Language Model). The user's question or prompt is interpreted and transformed into Python code by the LLM, and this generated code is then passed to a Python engine for execution. The Python engine has the capability to access and read the data needed for the process The web application is built with Streamlit. Streamlit is an open-source Python library to create interactive web applications and data dashboards. It simplifies the process of building user-friendly web apps with minimal effort, making it popular among developers, data scientists, and machine learning engineers. If you want to try out the code below, before you proceed below, you should have the following done to be able to try out this sample on your own. Install VS code (or some code editor)Install PythonInstall relevant extensions in your code editorSigned up with OpenAICreated an API KeyInstall openai libraryInstall Langchain (pip install langchain)Install streamit (pip install streamit)Install pypdf (pip install pypdf) All these steps are documented in length here with detailed instructions and examples.


Code setup to query your data using natural language Packages to install Install OpenAI, LangChain and StreamLit 2. Import the relevant packages Set the API keys 3. Load the data to query 4. Setup the LLM agent which will query your data The code below initializes an LLM (Language Learning Model) from the ChatOpenAI library, setting it up as a conversational agent. This agent is designed to interact with a pandas DataFrame, which contains data from the 'Mall_Customers.csv' file. The DataFrame holds information about mall customers, including their CustomerID, Gender, Age, Annual Income, and Spending Score. By creating a connection between the LLM model and the DataFrame using the agent, users can ask natural language queries or perform actions on the DataFrame. The agent handles the communication between the model and the DataFrame, enabling users to retrieve insights or manipulate the data without the need for SQL or code. The agent's functionality includes verbose logging to track its actions, and it allows a maximum of six conversational turns or iterations to limit the cost. Overall, this setup facilitates an intuitive and interactive way to query and interact with the mall customers dataset using natural language. In the next section, we will setup a basic web page that can be used to interact with this to ask question and get answers.


1. Build the Web interface We set up the streamlit app page by providing a title and a text box for the user to ask questions. When the user enters their question, it will be stored in the variable prompt so it can be used to query LLM. Now your basic app is ready


Use Natural language to get sights from this dataset Lets start asking questions. I have divided this in to 3 sections. In the first section called the basic section, I have few basic questions to ask. Then I will move on to some more advanced analytical questions to identify customers to target. And finally I will also build out machine learning models and use that to get insight by just asking questions in plain english.


1. Basic questions : Get to know your customers In the first section I will include not only the output but also the the thought process an agent goes through to get the answer. For other sections, I will suppress that to keep the article short.


1.1 Please display number of customers by gender. For the first question I will explain in detail how it works. Say I want to ask the question "Please display number of customers by gender". To do that I will say execute the following command Here is the description of what goes beyond the scenes. Here is from the Web UI Here is the thought process from LLM to answer the question working with Panda.:


1.2 Is there a correlation between age and annual income? Here is the thought process from LLM to answer the question working with Panda.:


1.3 Show the top 5 customers by spending Here is the thought process from LLM to answer the question working with Panda. Here is the output in the Web UI for that question. If you notice below, it won't show the actual data. To get the actual output of the customer IDs we have to specify that in the question we ask as below. Here is the thought process from LLM to answer the uestion working with Panda. See the actual customer ID is shown below.


1.4 How many customers spend more than 75? Here is the thought process from LLM to answer the uestion working with Panda.


1.5 Get a breakdown of customers by Age Question : Plot a bar chart for age using these 3 segments 030, 3060, above 60 Here is the thought process from LLM to answer the question working with Panda.: Inference: Lot more customers in the 3060 segment.


2. Advanced Analytical Questions: Find profitable customer segment to promote products To save space I will only print the final answer and not display the entire thread on the thinking process. I will ask a question in plain english and based on the answer I will ask further followup questions. The goal is to find specific customer segment to send personalized promotions.


2.1 Get a breakdown of customers by Spending Score Question : Plot a bar chart for Spending Score column using these 3 segments: 030, 3160, above 60 Inference: Lot more customers in the 3160 segment. We care about the people in the > 60 segment. Lets see the age breakdown of those customers whose spending > 60 next.


2.2 Get Age breakdown of customers whose spending > 60. Question : Plot a bar chart for age column using these 3 segments: 030, 3160, above 60 for customers whose Spending Score is greater than 60 Inference: We have a lot of young customers. Very interesting. Lets see their income breakdown next for those specific customers.


2.3 Get Age breakdown of customers whose spending > 60 and age < 30. Question : Plot a bar chart for Annual Income column using these 3 segments: 040, 4180, above 80 for customers whose Spending Score is greater than 60 and whose age is less than 30 Inference: Customers who are younger than 30 and who make less than 40k tend to spend more money. Next what is their gender breakdown so we can personalize our product recommendations.


2.4 Get Gender breakdown of customers whose spending > 60, age < 30 and Income < 40. Question : Plot a bar chart for gender column for customers whose Spending Score is greater than 60, age is less than 30 and Annual Income less than 40 By simply posing questions in plain English, we can effectively analyze the data. Initially, we examined the age and spending breakdown and inquired about the age distribution among high-spending customers. Upon reviewing this data, our focus shifted towards obtaining the income breakdown specifically for young customers with high spending habits. Ultimately, we acquired the gender breakdown to tailor product recommendations and personalize them for these customers.


3.1 Cluster your customers in to 3 groups Question : Create a kmeans cluster of size 3 using Income and Spending and plot a bar chart with those clusters Inference : That is pretty cool. I would like to see a average spending for each of these clusters.


3.2 Cluster your customers in to 3 groups and show average spending score in each segment Question : Create a kmeans cluster of size 3 using Income and Spending and plot a bar chart with those clusters on the x axis and average spending on the y axis Inference: I actually want to see the breakdown customers in each cluster by their spending. Lets do that next


3.3 Cluster your customers in to 3 groups and show stacked chart for each group broken down by spending score segments This is actualy a complex query behind the scenes. Let me show the thought process taken by the agent to create this query. As evident from the example, using plain English allowed us to employ k-means clustering and visualize the data. Initially, we visualized the data by the average spending within each cluster, and subsequently examined each cluster further by breaking it down into custom spending groups. This approach was successful not only with k-means clustering but also with other models. It demonstrates how non-technical users can provide clear instructions on data visualization without needing to write any code, thus expanding the possibilities in this domain.


Conclusion In conclusion, the post highlights the power of natural language processing (NLP) in querying data without the need for SQL or coding skills. By leveraging an LLM (Language Learning Model) and Langchain's pandas agent, users can create a conversational agent that interacts with the dataset containing information about mall customers. The agent functions by invoking the Python agent in the background, which in turn executes Python code generated by the LLM (Language Model) in response to the user question. This approach allows users to ask questions, perform data manipulations, and gain insights using plain language queries. With the ability to interact with data using natural language, users can unlock valuable insights and improve decision-making processes. The combination of NLP and DataFrame interaction offers a novel and user-friendly approach to data analysis, enhancing the overall experience of working with datasets.


This AI newsletter is all you need #45

What happened this week in AI by Louie This week witnessed several releases and developments in AI models, continuing the trend of open-source model alternatives. Among these, two popular consumer-facing LLM-based products, ChatGPT, and Github Co-pilot, faced new open-sourced competition with HuggingChat and Replit-Code, respectively. The focus on regulation and AI safety persisted as ChatGPT access was restored in Italy, and AI pioneer Geoffrey Hinton left Google partly due to his concerns about AI safety. Geoffrey Hinton has played a key role in many of the key breakthroughs in machine learning and the path to transformers and the recent success of LLMs, from backpropagation in 1986 to Alexnet in 2012. Hinton quit Google in part so he could talk more freely about the risks of AI, amid a recent adjustment of his views on the potential of AI and heightening competition between Google and Microsoft. "The idea that this stuff could actually get smarter than people - a few people believed that," "But most people thought it was way off. And I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that." Hinton's changing views and his increased perception of AI risk are a testament to the rapid progress in AI over the past year and illustrate just how little we really know about LLMs and what progress or roadblocks we might find even six months ahead. While we find many arguments over both AI's positive potential and risks (or lack thereof) are made with overconfidence given still huge uncertainty, Hinton's deep and earned respect in the community and flexible and measured views carry particular weight.


This AI newsletter is all you need #45

Towards AI and Learn Prompting are Announcing the HackAPrompt competition! We are thrilled to announce the first-ever prompt hacking competition, HackAPrompt, aimed at enhancing AI safety. In this competition, participants will try to hack as many prompts as possible by injecting, leaking, and defeating the sandwich defense. The competition is designed to be beginner-friendly, welcoming even non-technical people to participate. Register now to win prizes worth over $35,000! Launching on: May 5th at 6:00 PM EST


This AI newsletter is all you need #45

Hottest News Hugging Face release their own AI chatbot named HuggingChat Hugging Face has launched a new open-source chatbot model named HuggingChat, which is considered an alternative to OpenAI's ChatGPT. Users can test the model via a web interface, and it can also be integrated into existing apps and services through Hugging Face's API. 2. From DrakeGPT to Infinite Grimes, AI-generated music strikes a chord Last week, a song that used AI deepfakes of Drake and the Weeknd's voices gained popularity. In the meantime, Grimes has announced on Twitter that she will offer 50% royalties for any AI-generated song that features her voice. Musicians such as Holly Herndon and YACHT have adopted AI as a tool to push the boundaries of their creativity. 3. Amazon plan to improve Alexa using large language models During Amazon's first-quarter earnings call, CEO Andy Jassy announced that the company is developing a more advanced large language model (LLM) to power Alexa. Jassy stated that the upgraded LLM will be "more generalized and capable," and that it will help Amazon reach its goal of creating "the world's best personal assistant." 4. ChatGPT Will See You Now: Doctors Using AI to Answer Patient Questions UC San Diego Health, UW Health, and Stanford Health Care are collaborating to test a tool that leverages OpenAI's GPT to read patient messages and generate draft responses for their doctors. The pilot program's goal is to determine if AI can reduce the amount of time that medical staff spends responding to patients' online inquiries. 5. Replit introduced replit-code-v13b on ReplitDevDay During #ReplitDevDay, Replit unveiled replit-code-v13b, a new AI language model with 2.7 billion parameters. The model was trained on 525 billion tokens in just 10 days and demonstrates 40% better performance than similar models.


This AI newsletter is all you need #45

Five 5-minute reads/videos to keep you learning The Complete Beginners Guide To Autonomous Agents The article discusses the trend of "autonomous agents" among AI developers. It covers various topics such as the definition of autonomous agents, the opportunities they present, how they work, their potential future, and ways to build or use them. It also provides insights on how to connect with like-minded people interested in autonomous agents. 2. Why do we need RL for language models? The argument for RL is based on knowledge-seeking queries where we expect truthful answers, and the ability of the model to respond with "I don't know" or refuse to answer in situations where it is uncertain. For this type of interaction, RL training must be used as supervised training may teach the model to lie. 3. The future of generative AI is niche, not generalized ChatGPT has led to speculation about artificial general intelligence (AGI). However, the next phase of AI will likely be in specific domains and contexts. The true value of these systems lies not in their use as generalist chatbots, but rather as a class of tools that can be applied to niche domains, offering new ways to discover and explore highly specific information. 4. Navigating the High Cost of AI Compute This post presents a framework for analyzing the cost factors of an AI company. It covers the time and costs involved in building an AI system, provides a comparative analysis of CSPs and GPUs, and discusses how the costs are likely to evolve over time. 5. Introducing Hidet: A Deep Learning Compiler for Efficient Model Serving Hidet is a deep learning compiler that simplifies the process of implementing high-performing deep learning operators on modern accelerators like NVIDIA GPUs. It originated from a research project led by the EcoSystem lab at the University of Toronto (UofT) in collaboration with AWS.