The Importance of Data Visualization Prompt: Act like a teacher. Why is data visualization important in data science? ChatGPT : "Data visualization is a critical part of data science for several reasons:1. Understanding the Data: Visualizing the data helps in understanding the patterns, trends, and correlations that might not be apparent in tabular or raw data. For instance, it might be difficult to understand the trend in sales over a period directly from raw data, but a simple line plot can make it clear. 2. Communicating Insights Effectively: Graphs and charts are much more intuitive and engaging than spreadsheets or reports. They allow you to present complex data in a straightforward and easy-to-understand manner. This helps both technical and non-technical stakeholders to understand the insights easily.3. Speed: Humans are visually wired; we can process and understand visual information faster than text. Hence, visualizations help in quicker data analysis and decision making.4. Detecting Anomalies: Data visualization can help to detect outliers or anomalies in the data that might require further investigation.5. Supporting the Narrative: When combined with a narrative, visualizations can be a powerful tool to tell a story with data, helping to emphasize the main points and create a compelling case for action.6. Exploratory Data Analysis: Visualizations are a key component of exploratory data analysis (EDA), which is an essential step in data science. EDA involves looking at and describing the data set from different angles and under different visualizations to understand what's going on in the data.Remember, while data visualization is powerful, it is also essential to use it responsibly. Misrepresentation or distortion of data can lead to incorrect conclusions and decisions. Be mindful of the scales, axes, and dimensions being used and ensure they accurately represent the underlying data."


Supercharge Your Skills With ChatGPT To Get a Head 99% Of Data Scientist

Popular Tools for Data Visualization Prompt: Act like a teacher. Can you list some Python libraries used for data visualization and provide a brief description of each? ChatGPT : "Absolutely! Python has several libraries for data visualization, each with its own strengths and use cases. Here are some of the most popular ones:1. Matplotlib: This is the most widely used Python library for 2D graphics. It can create a wide range of static, animated, and interactive plots in a variety of hardcopy formats. Matplotlib is great for making simple bar graphs, line graphs, histograms, and many other types of plots.2. Seaborn: Built on top of Matplotlib, Seaborn is a high-level interface for drawing attractive and informative statistical graphics. It provides a more comfortable interface to Matplotlib and has some additional plot types. It works very well with Pandas dataframe objects and provides better default aesthetics and built-in plots for statistical exploration.3. Pandas Visualization: Pandas itself has built-in methods that call Matplotlib functions directly. This makes it easier to plot data directly from DataFrames and Series (Pandas data structures).4. Plotly: Plotly is a library that allows you to create interactive plots that you can use in dashboards or websites (you can save them as html files or static images). It's particularly good for interactive dashboards and presentations.5. Bokeh: Similar to Plotly, Bokeh is another library for creating interactive plots, dashboards, and data applications. It's geared toward creating visualizations for web browsers and hence ideal for web-based dashboards and applications.6. ggplot: Based on R's ggplot2, the Python version of ggplot is a plotting system that uses a declarative syntax. You declare the links between your data and the graphical representations you want to use, and ggplot takes care of the details.7. Altair: This is a declarative statistical visualization library which is built on Vega and Vega-Lite. Altair's API is simple, friendly and consistent, and it has a strong focus on effective visualization best practices.Remember, the choice of library often depends on the specific requirements of your task, including the complexity of the visualizations, the need for interactivity, and the volume of data. "


Tips for Effective Data Visualization Prompt: Act like a teacher. Can you provide some tips for creating effective and meaningful visualizations? ChatGPT : "Absolutely, creating effective and meaningful visualizations is an important aspect of data science. Here are some tips:1. Understand Your Audience: Know who will be using your visualizations and what information they need to gain from them. This will help you design your visualizations to be most effective for your specific audience.2. Choose the Right Chart Type: Depending on what you want to communicate, some chart types may be better than others. For example, bar graphs are excellent for comparing quantities across categories, line graphs are best for showing trends over time, and scatter plots are useful for observing relationships between variables.3. Simplicity is Key: Avoid making your visualizations too complex or cluttered. The primary aim is to communicate information clearly and efficiently. If a viewer has to spend too much time deciphering your chart, then it's not doing its job.4. Label Appropriately: Make sure your axes, legends, titles, and data points are properly labeled. This provides context and makes the visualization easier to understand.5. Use Consistent and Appropriate Scales: When comparing multiple visualizations, make sure you're using consistent scales. In a single visualization, ensure that your scales are appropriate for your data. Misleading scales can distort the interpretation of the data.6. Color Matters: Use color effectively to highlight important aspects of your visualization. However, be mindful of colorblind viewers and ensure that your visualization remains meaningful even without color distinctions. Avoid using too many colors as it can make the visualization confusing.7. Tell a Story: An effective visualization tells a story. This doesn't mean fabricating a narrative, but rather highlighting the insights and points of interest in your data.8. Keep an Eye on the Aspect Ratio: The ratio of width to height can dramatically affect the look of a visualization, and in some cases, even its interpretability. Try different ratios to see which one works best for your data.9. Check Your Work: Before you finalize any visualization, check it for accuracy and clarity. Are you presenting the data truthfully? Is the message clear? Could anything be misinterpreted?10. Iterate: Rarely will you get the perfect visualization on your first try. Experiment with different types and tweak your visualizations based on feedback from others and your own critical review.Remember, the goal of data visualization is to simplify complex datasets into understandable visuals. Always consider how you can make it easier for others to understand the insights from your data."


My Cheat Sheets and Source Codes. If you've made it this far, thank you! I continually update and add new Cheat Sheets and Source Codes for your benefit. Recently, I crafted a ChatGPT cheat sheet, and honestly, I can't recall a day when I haven't used ChatGPT since its release. Also, here is my E-Book, explains, how Machine Learning can be learned by using ChatGPT. Feel free to select one of the Cheat Sheets or projects for me to send you by completing the forms below Here is my NumPy cheat sheet. Here is the source code of the "How to be a Billionaire" data project. Here is the source code of the "Classification Task with 6 Different Algorithms using Python" data project. Here is the source code of the "Decision Tree in Energy Efficiency Analysis" data project. Here is the source code of the "DataDrivenInvestor 2022 Articles Analysis" data project.


This AI newsletter is all you need #49

What happened this week in AI by Louie New AI model releases and announcements continued at pace this week, though step-change progress in capabilities feels slower than earlier this year. AI remained prominent in the mainstream news including a focus on Nvidia and the increased pace of GPU rollouts in datacentres for LLM inference and training. AI risks and regulation were also in the news again with yet another letter, this time signed by most leading AI CEOs and researchers, including Sam Altman of OpenAI, Demis Hassabis of Deepmind, and Geoffery Hinton, warning on the need to manage AI risks. The statement released by the Center for AI Safety was simple and to the point; "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war". We agree again here. While other risks from AI such as inaccuracies (including hallucinations appearing in court in legal research!) still feel closer to hand, increasing the probabilities of all the positive outcomes from AI and reducing the risks of the negative should now be at the forefront of policymaker's minds. - Louie Peters - Towards AI Co-founder and CEO


This AI newsletter is all you need #49

Hottest News 1.Introducing speech-to-text, text-to-speech, and more for 1,100+ languages Meta's Massively Multilingual Speech project utilizes self-supervised learning with wav2vec 2.0 and a unique dataset to empower AI in comprehending and generating speech across more than 1,100 languages. This groundbreaking initiative outperforms existing models, boasting reduced character error rates and expanded language coverage. 2. Intel Announces Aurora genAI, Generative AI Model With 1 Trillion Parameters Intel has announced the Aurora Generative AI model, with 1 trillion parameters. This model will be specifically trained on scientific texts and structured scientific data, with a focus on cancer research, systems biology, cosmology, polymer chemistry, materials science, and climate science. The Aurora Gen AI model will be powered by the Aurora supercomputer. 3. Google is starting to let users into its AI search experiment Google's Search Labs is now open for experimentation, providing AI-generated summaries at the top of search results. These summaries are globally compatible with multiple languages, potentially altering the business model of the web and impacting SEO. 4. Sam Altman shares his optimistic view of our AI future Last week, OpenAI CEO Sam Altman engaged in discussions with heads of governments and startup communities in Europe regarding AI regulation and beyond. Altman highlighted the current significance of AI, emphasizing its proficiency in various domains and its proven ability to enhance productivity across diverse job sectors. 5. New superbug-killing antibiotic discovered using AI Researchers have made a groundbreaking discovery by utilizing AI to identify a compound that effectively eliminates Acinetobacter baumannii while displaying minimal signs of resistance. This achievement showcases the immense potential of AI in identifying prospective antibiotics and accelerating the pace of new treatment discoveries.


This AI newsletter is all you need #49

5-minute reads/videos to keep you learning 1.Making LLMs even more accessible with bits and bytes, 4-bit quantization and QLoRA HuggingFace introduced QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLoRA backpropagates gradients through a frozen, 4-bit quantized pre-trained language model into Low Rank Adapters~(LoRA). 2. How To Finetune GPT Like Large Language Models on a Custom Dataset Lit-Parrot is a nanoGPT-based tool developed by Lightning AI that offers clean and optimized LLM implementations for fine-tuning on custom data. It includes prompt adaptation, LLM approximation, and LLM cascade for cost reduction. Lightning AI offers step-by-step guidance on installation, dataset preparation, and model fine-tuning. Check out the article to learn more. 3. The hard stuff no one talks about building on LLMs Language models are fantastic new tools with high potential. However, they suffer from a set of challenging issues that makes them hard to deploy in production. This post discusses prompt reliability, monitoring, and more. It gives a glimpse into systems you'd need to build to deploy language models in your applications. 4. Transform Any Image with a Single Movement of Your Mouse: DragGan Explained The new DragGan approach allows users to edit images by simply dragging points from A to B, revolutionizing the way we interact with image editing. The AI realistically adapts the entire image, modifying the object's position, pose, shape, expressions, and other frame elements. This video tutorial explains how it works. 5. How to use ChatGPT with your Google Drive in 30 lines of Python This tutorial explains how to build a Python app powered by GPT and GDrive. This combination is made possible by LangChain's Google Drive document loader and serves as a stellar foundation for an infinite number of great apps.


This AI newsletter is all you need #49

Papers & Repositories 1.LIMA: Less Is More for Alignment LIMA is a language model that can improve alignment without reinforcement learning or human preference modeling, trained on 1,000 curated prompts and responses. It outperformed GPT-4 in 43% of cases without human feedback and highlights the importance of pretraining over large-scale instruction tuning and reinforcement learning. 2. Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training Sophia, Second-order Clipped Stochastic Optimization, a simple scalable second-order optimizer that uses a lightweight estimate of the diagonal Hessian as the pre-conditioner. It achieves the same validation pre-training loss with 50% fewer steps than previous optimizers and can be easily integrated into existing training pipelines. 3. The False Promise of Imitating Proprietary LLMs Google researchers have discovered that utilizing imitation models to enhance a weaker language model may demonstrate initial improvements. However, these improvements are restricted to tasks for which the training data provides significant support and cannot effectively bridge the gap for unsupported tasks. The approach outlined in this paper involves fine-tuning a sequence of language models that imitate ChatGPT, followed by an evaluation of these models using crowd raters and canonical NLP benchmarks. 4. LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond A recent paper delves into the accuracy of large language models (LLMs) in detecting factual inconsistencies. While certain LLMs exhibited strong performance, the majority faced difficulties when dealing with complex formulations, indicating issues with current benchmarks. To address this, researchers developed the new SUMM EDITS benchmark, which evaluates the LLMs' capability to identify factual inconsistencies. 5. Reasoning with Language Model is Planning with World Model The paper introduces a novel framework called Reasoning via Planning (RAP), which reimagines the LLM as both a world model and a reasoning agent. It incorporates a well-structured planning algorithm (based on Monte Carlo Tree Search) for strategic exploration within the expansive reasoning space. Enjoy these papers and news summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #49

Weekly AI Podcast In this week's episode of the "What's AI" podcast, Louis Bouchard is in conversation with Yotam Azriel, the co-founder of TensorLeap, where he shares his fascinating journey, insights, and vision for the future of Explainable AI. In this episode, Yotam will take you on a journey through his remarkable experiences, mind-boggling insights, and vision that will leave you amazed. Discover how the power of passion, curiosity, and focus can accomplish wonders! Tune in on YouTube, Spotify, or Apple Podcasts and expand your knowledge of Explainable AI.


This AI newsletter is all you need #49

Meme of the week! Meme shared by NEON#8052


This AI newsletter is all you need #49

Featured Community post from the Discord Elvitronic#5445 presents an innovative approach to developing AI models that mimic human-like cognitive abilities. The Roadmapping Cognitive Development in AI through Human Experience Data Analysis project aims to investigate and model the cognitive development process in artificial intelligence, guided by the fundamental principles of human cognitive development. They propose to construct tests that trigger the least possible areas of the brain, thereby isolating these basic skills, to train AI models using transformer agents to master each skill individually. If you wish to collaborate, reach out in the thread here and support a fellow community member by sharing your feedback!


This AI newsletter is all you need #49

AI poll of the week! Join the discussion on Discord.


This AI newsletter is all you need #49

Article of the week Holy Cow! Introducing DragGAN by Dr. Mandar Karhade, MD. PhD. Synthesizing visual content that meets users' needs often requires flexible and precise controllability. Existing approaches gain controllability of GANs via manually annotated training data or a prior 3D model, which often lack flexibility, precision, and generality. In this work, you will study a powerful way of controlling GANs, that is, to "drag" any points of the image to precisely reach target points in a user-interactive manner


This AI newsletter is all you need #49

Our must-read articles From Pixels to Artificial Perception by Ali Moezzi Policy Gradient Algorithm's Mathematics Explained with PyTorch Implementation by Ebrahim Pichka FineTuning Local Large Language Models on Your Data Using LangChain by Serop Baghdadlian If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.


This AI newsletter is all you need #49

Job offers Senior Manager, Instrument Software @Deepcell (Menlo Park, CA, USA/Hybrid) Senior Data Engineer @Jupiter (Bangalore, India) Software Engineer (Infrastructure) @X1 (Remote) Full Stack Software Engineer - CBM+ @Redhorse (Remote) Senior Software Engineer @H1 (Remote) Engineering Manager, Machine Learning - NLP @BenchSci (Remote) Interested in sharing a job opportunity here? Contact sponsors@towardsai.net. If you are preparing your next machine learning interview, don't hesitate to check out our leading interview preparation website, confetti!


LangChain DeepDive - Memory  The Key to Intelligent Conversations

Different Types of Memory LangChain provides memory components in two forms: helper utilities for managing and manipulating previous chat messages and easy ways to incorporate these utilities into chains. The memory types provided by LangChain include: ConversationBufferMemoryConversationBufferWindowMemoryEntity MemoryConversation Knowledge Graph MemoryConversationSummaryMemoryConversationSummaryBufferMemoryConversationTokenBufferMemoryVectorStore-Backed Memory The different types of memory in LangChain are not mutually exclusive; instead, they complement each other, providing a comprehensive memory management system. For instance, ConversationBufferMemory and ConversationBufferWindowMemory work together to manage the flow of conversation, while Entity Memory and Conversation Knowledge Graph Memory handle the storage and retrieval of entity-related information. let's look at each one of them in detail. Buffer Memory: The Buffer Memory in Langchain is a simple memory buffer that stores the history of the conversation. It has a buffer property that returns the list of messages in the chat memory. The load_memory_variables function returns the history buffer. This type of memory is useful for storing and retrieving the immediate history of a conversation.Buffer Window Memory: Buffer Window Memory is a variant of Buffer Memory. It also stores the conversation history but with a twist. It has a property k which determines the number of previous interactions to be stored. The buffer property returns the last k*2 messages from the chat memory. This type of memory is useful when you want to limit the history to a certain number of previous interactions.Entity Memory: The Entity Memory in Langchain is a more complex type of memory. It not only stores the conversation history but also extracts and summarizes entities from the conversation. It uses the Langchain Language Model (LLM) to predict and extract entities from the conversation. The extracted entities are then stored in an entity store which can be either in-memory or Redis-backed. This type of memory is useful when you want to extract and store specific information from the conversation. Each of these memory types has its own use cases and trade-offs. Buffer Memory and Buffer Window Memory are simpler and faster but they only store the conversation history. Entity Memory, on the other hand, is more complex and slower but it provides more functionality by extracting and summarizing entities from the conversation. As for the data structures and algorithms used, it seems that Langchain primarily uses lists and dictionaries to store the memory. The algorithms are mostly related to text processing and entity extraction, which involve the use of the Langchain Language Model.


Comparison with Indexes Module In LangChain, memory and indexes serve different but complementary roles in managing and accessing data. Here's a comparison of the two:


Memory Memory in LangChain refers to the various types of memory modules that store and retrieve information during a conversation. Memory is crucial for maintaining context over a conversation, answering follow-up questions accurately, and providing a more human-like interaction. It also enables the agent to learn from past interactions and improve its responses over time.


Indexes Indexes in LangChain refer to ways to structure documents so that Language Models (LLMs) can best interact with them. Indexes are used in a "retrieval" step, which involves taking a user's query and returning the most relevant documents. The primary index types supported by LangChain are centred around vector databases. These indexes are used for indexing and retrieving unstructured data (like text documents). For interacting with structured data (SQL tables, etc) or APIs, LangChain provides relevant functionality. Indexes are crucial for the efficient retrieval of relevant documents based on a user's query. They enable the agent to sift through large amounts of data and return the most relevant information.


Comparison While both memory and indexes are used for storing and retrieving data, they serve different purposes. Memory is used for storing conversation history and context, which is crucial for maintaining a coherent and context-aware conversation. On the other hand, indexes are used for structuring and retrieving documents based on their relevance to a user's query. In terms of their implementation, memory types in LangChain can be backed by various databases like Postgres, DynamoDB, Redis, MongoDB, and Cassandra. Indexes, on the other hand, are primarily centred around vector databases. In summary, memory and indexes in LangChain work together to provide a comprehensive data management and retrieval system that enhances the capabilities of AI agents.


pre-requisites: GOOGLE_API_KEY: get it from https://console.cloud.google.com/apis/credentialsGOOGLE_CSE_ID: https://cse.google.com/cse/create/newOPENAI_API_KEY: https://platform.openai.com/account/api-keys ans set them as environemnt varibales Imports GoogleSearchWrapper setup Prompt Template setup The agent with Memory Setup the next question will refer to chat before answering


The agent without Memory Setup In this example, the agent is able to remember the context of the conversation due to its memory. When asked a follow-up question that relies on information from the previous exchange, the agent can correctly answer because it remembers that the previous question was about Wayne Rooney's penalty success ratio not the generic definition of it. references:


Generating Professional Introduction Emails with LlamaIndex

The Idea Have the user input their LinkedIn login information and the links to the LinkedIn pages of soon-to-be-connected individuals;Scrape, clean, and format LinkedIn data so it is ready to be fed to an LLM;Feed LinkedIn data to LLM using Llama index and query for introduction email.


Install modules/packages The packages you need are llama_index, langchain, selenium, and linkedin-scraper, all of which can be installed with pip; you also need chromedriver, which you can download using this link. You'll also need an OpenAI key for this project.


Importing packages Before importing, run this code (as required by linkedin-scraper) through your Mac's Terminal. Don't forget to replace your-path-to-chromedriver: Let's import our packages! But before that, let me briefly explain what each of our packages is used for. LlamaIndex is a tool that helps us build LLM apps: with it, we can easily feed our own data to LLMs. LangChain is similar to LlamaIndex in this way, but we are only using it to borrow its OpenAI model. LinkedIn Scraper scrapes LinkedIn data - simple enough. And last but not least, Selenium automates browsers, which is crucial for our web scraper.


Getting and Cleaning Linkedin Data From Linkedin Scraper's GitHub page, we see what fields a Person object has: Let's keep it simple and only use name and experiences from this very long list. We first define a function get_experiences(links, driver) where, given some LinkedIn links and a web driver, returns a dictionary where the keys are names of the individuals in question and the values are given by function get_person_exp(person). We also define function clean_name(person), since our scraper, when returning a person's name, sometimes includes strings like "\n1st degree connection" which we would like to get rid of. Next up, we create another function: get_person_exp(person). It takes a Person object as argument and returns that person's experiences in dictionary form, {institution_name: description}. The complicated-looking structure, with dictionaries and sets, is my attempt at getting rid of duplicate information given by the LinkedIn Scraper - I won't bore you with more details. Now we can see that the dictionary given by get_experiences(links, driver) is in the form { name: { institution_name: description } }. Almost there! Let's put it all together and get the data we want. Using Terminal, we ask the user to input their LinkedIn login information, as well as the links the LinkedIn pages of the individuals we want to know about. Then, using ChromeDriver as our driver, we log in to LinkedIn. Data acquired! Although it is looking a little like gibberish. To optimize the performance of our LLM, let's make it a little more readable before saving it. Before anything else, let's tell the LLM what the data is about by saying, "Here is a dictionary of this person's work experiences, in the format {institution_name: job descriptions}". Then, we list each institution_name followed by their job descriptions. Let's run it and take a look at some of the output. Not too bad!


Setting up LlamaIndex Now that we have our LinkedIn data ready, we need to feed it to LlamaIndex. Here is some code I'm recycling from another project of mine. The make_query_engine() function takes the files from our "data" directory, uses vector embeddings to make the data more readable to an LLM, then returns a query engine. The query(question, query_engine) function takes that query engine and a question you would like to ask the LLM as input, and outputs the LLM's response. All done! Last step, let's formulate a question for the query engine and print its response. Note that we try loading the existing index from our storage folder (which should be in the same directory as the email generator python file). My purpose for this was to save API calls when running this file multiple times on the same LinkedIn links, but you don't have to structure the code this way. I arrived at this final version after some trials and errors in prompt engineering, but feel free to try different things out. Let's see how it performs! Sometimes it takes a couple of tries, but the generated email template doesn't look too bad at all. With a bit of tweaking, it should be ready to send - and that's it! The code could definitely use improvement, ex. better cleaning and formatting of the LinkedIn data, more customization for the query engine, etc. If you would like to play around with the code, you can find it on my GitHub repo.


Docker in MLOps For Starters

Motivation I am writing this article to provide valuable information and guidance to individuals who are new to the field of MLOps and are looking to understand the concepts and practices related to Docker containerization in machine learning projects. In an era dominated by GPTs (Generative Pre-trained Transformers), the article aims to simplify and consolidate the necessary knowledge on this topic, making it accessible to beginners like me.


Docker in MLOps For Starters

What's Docker Docker is an open-source platform that helps developers to automate the deployment and manage applications inside isolated containers. It is very lightweight compared to traditional Virtual Machines. It has end-to-end packaging by building a self-sufficient environment that includes all the libraries, packages, and tools that removes the dependency from the source side and facilitates seamless deployment of ML models across different platforms or cloud environments.


Docker in MLOps For Starters

Why it's important in MLOps A Machine Learning life cycle is a long process where the same process is run multiple times to generate results and small changes can make a significant difference in the Model Performance. Using docker in this life cycle can improve the following things: Collaboration: It immensely helps MLEs to work as a team. It reduces dependency by providing the packaging with all the necessary tools and libraries. For example, Developer A has built an environment and a model to detect fraudulent payments. Once the first version was built, Developer B wanted to go through the process and add some inputs. If the code is dockerized by A then B can easily use that and run the model without A's intervention and build a better model by fine-tuning the hyperparameters or by changing the algorithm. Reproducibility: It reduces the confusion between the development and production teams by providing a consistent and easily reproducible environment. This also diminishes the discrepancies in software versioning, hardware configuration, etc., which ultimately helps to improve the model performance. Scalability: Not only it helps to automate deployment, but it also simplifies the scaling for ML applications by providing a consistent deployment model. The containers can be easily replicated and distributed across cloud instances, allowing efficient utilization of resources and scaling of ML applications.


Docker in MLOps For Starters

Docker Installation in WSL without Desktop Docker To install Docker in Windows Subsystem for Linux (WSL) without the desktop Docker, you can follow these steps: 1. Enable WSL: Make sure you have WSL enabled on your Windows machine. You can enable it by following the instructions provided by Microsoft for your Windows version. 2. Install a Linux distribution: Choose and install a Linux distribution from the Microsoft Store. Ubuntu is a popular choice, but you can select any distribution that suits your requirements. 3. Set up WSL: Once the Linux distribution is installed, launch it from the Start menu or use the wsl command in the command prompt. Follow the initial setup instructions to create a username and password for your Linux environment. 4. Update the Linux distribution: Run the following commands in the Linux terminal to update the package manager and installed packages: 5. Install Docker: Execute the following commands in the Linux terminal to install Docker: Here, the "-y" option is used to grant permission to install required packages automatically 6. Verify Docker installation: To verify that Docker is installed correctly, run the following command: You should see the Docker version information if the installation was successful. 7. (Optional) Allowing non-root access (recommended): By default, Docker commands require root privileges. If you want to run Docker commands without using sudo, you can add your user to the Docker group. Execute the following command: Remember to log out and log back in or restart your system for the group changes to take effect. With Docker successfully installed in your WSL environment, you can now start using Docker to build, run, and manage containers from within your Linux distribution. Note that since you are not using the desktop Docker application, you'll interact with Docker exclusively through the command line in your WSL environment.


Docker in MLOps For Starters

How to Start Docker Engine Open a WSL terminal and run the docker engine using this command If Docker failed to start because it found a PID (Process ID) file, which suggests that Docker might already be running or there was an issue with a previous Docker instance. Debugging If Docker is running, you should see an output indicating its status. Stop Docker service: If Docker is running, you need to stop it before starting it again. Execute the following command: To resolve this issue, you can follow these steps: Start Docker: After removing the PID file, you can start Docker again by executing the following command: Docker should now start successfully without any errors.


Docker in MLOps For Starters

Basic Commands in Docker Open another terminal to run and learn the basic commands of Docker. 1. docker ps: List running containers. 2. docker images: List of all Images 3. docker run <image_name>:tag : Run the docker image with Ubuntu in the layer Image 4. docker pull<image_name>:tag : Pull the docker image from the registry 5. docker stop <container_id>: Stop a running container 6. docker rm <container_id>: Remove a container 7. docker rmi <image_name>:tag: Remove an image 8. docker build -t <image_name> .: Build an image based on current directory 9. docker-compose up: Start containers defined in a Compose file. 10. docker exec -it <container_id> bash: Run a command inside a running container.


Docker in MLOps For Starters

Simple Package code I have built a simple codebase to do basic arithmetic operations. I have one folder or package called arithmetic inside it. I have written a script to do all the operations. Inside the root directory, I have the main, which uses these functions to do all the calculations.


Docker in MLOps For Starters

Docker File Setup After finishing the code, we need to set up a docker file which will be used to build the image. Here I am attaching the demo docker file for better understanding: Now the final directory structure will look like this:


Docker in MLOps For Starters

Docker Build Build the Docker image using the following command: Run the newly built docker image by running this command:


Docker in MLOps For Starters

Docker Account Setup Go to the Dockerhub Website and create your own account to publish your local Docker Images. The process to set up the repository:


Docker in MLOps For Starters

Docker push Push the tagged image to Docker Hub using the docker push command: This will upload the image to your Docker Hub repository. This is the URL that can be shared among the team: https://hub.docker.com/r/sawon17/sandocker_2023/tags The format of the link is: https://hub.docker.com/r//>


Docker in MLOps For Starters

Testing Open a terminal or command prompt. Use the docker pull command to pull the Docker image from Docker Hub: To run a container using the pulled image, we will use the docker run command:


Docker in MLOps For Starters

Summary If you have made it this far, congratulations. We have built our first Docker Container. Let's summarize the journey to understand the key points: Docker proves to be an invaluable tool for streamlining the development and deployment process in machine learning projects. Through the article, we explored the significance of Docker in MLOps, also provided a step-by-step guide on Docker installation in WSL without the desktop Docker, demonstrated basic Docker commands, and showcased a code structure for arithmetic operations that can be containerized using Docker. Embracing Docker containerization in machine learning projects offers numerous advantages, including enhanced productivity, simplified deployment, and accelerated innovation. By harnessing the power of Docker, ML practitioners can focus more on developing cutting-edge models and less on the intricacies of environment setup and deployment.


Docker in MLOps For Starters

Resources 1. https://youtu.be/8vmKtS8W7IQ 2. https://linuxhint.com/run-docker-in-wsl-without-docker-desktop/ 3. https://www.docker.com/ 4. https://medium.com/analytics-vidhya/mlops-dockers-and-kubernetes-essentials-for-a-data-scientist-8bc7b5866957 5. https://towardsdatascience.com/why-using-docker-for-machine-learning-74c927ceb6c4


Interactive Product Documentation: Creating an Engaging Conversational Q&A Bot with OpenAI and LangChain

1. Interactive Product Manuals Can we transform the thousands of pages in a static product manual into integrated guides? Users would simply ask questions to find the information they need. For instance, let's consider a user who has just purchased a new DSLR camera and wants to know how to change the aperture. Instead of sifting through a bulky manual, they could ask an LLM-powered chatbot, 'How do I change the aperture on my new camera?' The chatbot would search the product manual, comprehend the context, and provide a straightforward, step-by-step guide.


2. Instant Troubleshooting Assistance When a product or device encounters an error, users tend to aggressively search for troubleshooting information. The time it takes to find this information and resolve the issue directly impacts customer satisfaction. This is where Conversational UI and LLMs can provide instant assistance to users, making the process much smoother. For instance, let's consider a user whose printer is not working and displays an error message. They could ask a chatbot, 'My printer is showing Error Code B200. What does it mean, and how can I fix it?' The chatbot, utilizing the power of LLM, would interpret the error code from the product documentation and guide the user on resolving the issue.


3. Product Feature Explanation Gone are the days when users would attempt to read the entire getting started guide. Nowadays, they only seek information about a particular topic when the need arises. For instance, a user might come across a feature in a product they already own and want to learn more about it. This is where Conversational UI can bridge the gap by providing on-demand explanations of specific product features. For example, let's imagine a user who is curious about the features of their new smartwatch. They could ask, 'What can I do with the heart rate monitor on my smartwatch?' The chatbot would then provide a comprehensive list of capabilities offered by the heart rate monitor, along with explanations of how to use each one


4. Personalized User Onboarding Conversational UI and LLMs can provide a personalized onboarding experience for new users, helping them understand how to use the product effectively. For example, let's say a user has just signed up for project management software. Instead of overwhelming them with all the features at once, a chatbot could guide them step-by-step. The user can ask questions like, "How do I create a new project?" or "How can I assign tasks to my team?" The chatbot would then provide tailored responses based on the product documentation.


5. Comparative Product Analysis Everyone compares multiple options before buying a product. The way they do that is to pour over the product documentation to find the relevant details and compare and contrast. A chatbot equipped with a Large Language Model can help in this regard. For example, a customer is debating between two models of a smartphone. They could ask the chatbot, "What's the difference between the camera capabilities of Phone Model A and Phone Model B?" The chatbot could then analyze the product documentation for both models and provide a comparative analysis.


B. Quick prototype to showcase querying product documentation To showcase how to query product documentation, I used the product documentation that comes with a camera. This is an example document you could use. Or you can download whatever documentation you would like to try this sample with. This will be a web-based chatbot where users can ask specific questions about the features of their camera. In this case, I will ask questions from Chapter 2. I would like to know what setting to enable for cloudy conditions? I would like to know what setting to enable for shooting night scenes. In the next section, I will walk through how this was built.


Initial setup Install OpenAI, LangChain, and StreamLit Import the relevant packages Set the API keys Define the LLM to use


Ingesting product documentation We used Langchain to ingest product documentation data. LangChain supports multiple formats, including text, images, PDFs, Word documents, and even data from URLs. In the current example, the pdf loader was utilized to ingest the product documentation, but if you wish to work with a different format, you simply need to refer to the corresponding loader specifically tailored for that format. This document I used has a lot of pages. So I saved this into a vector database. (default is chroma). You can use other vector datastores like Pinecone. The advantage of this is, your content is split into smaller chunks. So when a question is asked, instead of sending all the pages to LLM, we first query the vector database to retrieve the chunks that are relevant to the question and only send those chunks to the LLM to answer the question. Within the LangChain framework, the VectorstoreIndexCreator class serves as a utility for creating a vector store index. This index stores vector representations of the documents (in chromadb), enabling various text operations, such as finding similar documents based on a specific question. Now we are ready to query these documents.


Setting up the web application The application is presented in the browser using Streamlit, providing a user-friendly interface. When a user enters a question, this question is sent to the vector database to query relevant chunks, and this, along with the question is sent to LLM via index.query() method. The response is displayed on the screen. Here is an example of asking questions about night shooting using this chatbot, where the precise answers are from the product documentation. As you can see, it's a lot easier to query using a conversational UI to get the exact answer you are looking for. And as you saw from the above simple prototype, it's not difficult to build either. If you have tons of documentation in your company, it's a great opportunity to wrap up chatbot UI on top of it to query your documents.


Conclusion In conclusion, the integration of Conversational UI and Large Language Models brings a host of benefits to the realm of product documentation. By enabling interactive manuals, instant troubleshooting, feature explanations, personalized user onboarding, and comparative analysis, these technologies can significantly enhance user experience and engagement. The rise of conversational UIs raises a series of intriguing questions. Are we on the verge of seeing traditional, static product documentation being completely replaced by conversational ones? Will information-heavy product documentation be substituted with interactive, conversational interfaces that offer users personalized information in direct response to their inquiries? While it's too early to confirm these possibilities, the signs certainly suggest a shift in this direction. As we continue to make strides in AI, these LLM applications are only the beginning, promising an exciting future for user support and product understanding.


Towards Generating Ultra-High Resolution Talking-Face Videos with Lip-Synchronization

Vector Quantization & VQ-GAN Given a dataset, Vector Quantization involves learning a latent representation of the data using an autoencoder, but with unique features. The latent space consists of discrete vectors, rather than continuous ones. This may not sound like a huge difference, but it is. Using discrete tokens allows for a huge level of compression, and also enables token-based models such as the transformer to operate on the latent space. Vector Quantization is achieved by jointly learning encoder and decoder networks, as well as a codebook consisting of learnable and continuous latent vectors. This may seem counterintuitive, as we want to learn a discrete latent space. However, the codebook actually allows us to do this. We can represent the latent space by the index of the latent vector in the codebook. As an example, the codebook may be: {0=(0, 0.2), 1=(0.5, 0.6), 2=(0.8, 0.4)} then a latent vector (0.8, 0.4) could be written using the index 2. In practice, many latent vectors are used to represent each image, as shown in the diagram below. There are several subtypes of Vector Quantization models. The one used for this paper is a VQGAN [3], based on the Generative Adversarial Network. Here, in addition to a reconstruction loss, an adversarial loss is also applied to improve the reconstruction quality. Two VQGAN networks are trained, one for the full face images, and one for the same images, but with the lower half of the face masked.


Reframing the Problem Given these VQGANs, the problem can now be reframed. Rather than trying to inpaint the bottom half of the face image, containing over a quarter of a million pixels, the problem is now to convert the quantised masked face image to the quantised full face image. This is a much easier problem. The main section of the paper now largely follows the Wav2Lip style approach. If you're not familiar with it, I would suggest reading my previous post on it. But, in essence, to generate a frame with the lips changed you do the following: Mask out the lower half of the face.Take an audio window corresponding to the new lips, using MEL spectrograms, centred around the target frame.Take a random reference frame from the same video.Encode each separately, combine them by concatenation and then decode. In this paper, it is much the same, except now the masked frame and reference frame are first converted in the quantized space. In a similar fashion, the lip-sync expert discriminator, the SyncNet, is also trained in the quantised space. The SyncNet is now given both an audio segment and a sequence of quantised frames and is asked to predict if they are in or out of sync. This network can then be used as a loss for the lip-sync generation. It is worth noting that, as this model is trained in the lower dimensional quantised space, it can be trained on longer sequences. For this work, a full second is used, which helps further improve the sync.


Further Improving the Quality Despite the improved visual quality obtained by using VQGAN, there is still some noticeable lack of quality in the final output, particularly in the lip region. To address this, the authors propose an additional post-processing step. They train a face restoration GAN [4], which is able to improve the sharpness of the lips and use this as a post-processing step.


Verdict This paper makes impressive strides in improving the visual quality of person-generic models. However, the results are not publicly available anywhere, as far as I can tell, making it hard to gauge how good the model is. What's more, the results that can be seen in the paper seem to suggest that this model still struggles with the other problem in person generic models: capturing person-specific idiosyncracies. I suspect this is largely due to idiosyncrasies being overwritten by the VQGAN where it is unable to capture them. As an example, look at the lips in the following figure: For the man in the bottom row, the lips are far more pink than one would expect, suggesting that the model has not learned the idiosyncrasies correctly. It would be interesting to see what happens if more than one reference frame is given per person. Despite these drawbacks though, the quantitative and quantitative evaluations show that the model is very high quality and I look forward to seeing the videos in the future!


References [1] Towards Generating Ultra-High Resolution Talking-Face Videos with Lip-Synchronization. Gupta et. al. WACV 2023 [2] A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild. Prajwal et. al. ACM Multimedia 2020 [3] Taming Transformers for High-Resolution Image Synthesis. Esser et. al. CVPR 2021 [4] GAN Prior Embedded Network for Blind Face Restoration in the Wild. Yang et. al. CVPR 2021


Tokens and Models: Understanding LangChain  Part:3

Tokens : Tokens can be thought of as parts of words. The API breaks down the input into tokens before processing the prompts. Tokens are not cut exactly where words start or end, and can include trailing spaces and even sub-words. In NLP, we have a term known as tokenization, where we cut all paragraphs into sentences or words. Here we are doing exactly the same and cutting sentences and para into small chunks of words. The above fig shows how texts are divided into tokens. the different colors are representing different tokens. A general rule of thumb is that one token is roughly equivalent to 4 characters of text in common English text. This means that 100 tokens are approximately equal to 75 words. If you want to check any particular text for a number of tokens then you can directly check on OpenAI's Tokenizer. Another method to count tokens is to use tiktoken library. And at last, apply the above function


Output Likewise, you can count the number of tokens.


How does the number of tokens impact the selection of the model? First, let's see which different models are available in OpenAI. In this blog, I am explicitly focusing on OpenAI models. we can use hugging faces and cohere AI models but I will write about it in the next blog. Let's understand basic models first.


Models GPT is so powerful because it is trained on a massive dataset. However, more power comes at a cost, so OpenAI provides multiple models to choose from. The available models, also known as engines, are named. Davinci is the largest and most capable engine. It can perform any task that any other engine can perform. Babbage is the next most capable engine. It can do anything that Curie or Ada can do. Ada is the least capable engine, but it is the best-performing and lowest-cost engine. there are various generations of these models as GPT is getting advanced. there are approximately 50 + models present in the GPT family. So there are different models for different purposes like generating and editing images, working on audio, and coding. We want models which will help us to do text processing accurately and the model which will work on natural language. In the above image, there are 3 models which we can work with. In all above, we cant use GPT4 directly because GPT-4 is currently in a limited beta and only accessible to those who have been granted access. We have to join a waitlist and wait for our term. we can't do that. So we now have 2 options left. GPT3 and GPT 3.5.


If you have found this article insightful It is a proven fact that "Generosity makes you a happier person"; therefore, Give claps to the article if you liked it. If you found this article insightful, follow me on Linkedin and medium. You can also subscribe to get notified when I publish articles. Let's create a community! Thanks for your support! Also, medium doesn't give me anything for writing, if you want to support me then you can click here to buy me coffee. You can read my blogs related to, Signing off, Chinmay


Time Series Regression Using Transformer Models: A Plain English Introduction

What is time series data? The obvious question, with a simple answer. Time series data is simply a series of data points indexed in time order. An easy example to imagine is measuring the temperature outside your window once every minute, for a day. You would end up with 24x60 = 1440 measurements of temperature, in time order. Thinking about it, a lot of the data we collect in the world is time series data - think about financial data (stock prices), weather data, or any sensor data in wearable tech that one might wear. The thing to note with time series data is that we can analyse it in two ways - we can either ignore the fact there is time information and treat it as a collection of measurements (e.g., if you want to determine whether the average temperature is higher in your car or your kitchen in the same time period), or we can use the time information (e.g., if you want to determine what the temperature trend in your kitchen is over the course of a few days). What can we do with time series data? Assuming now that we want to use this time information, what can we actually do? Well, all tasks will fall into one of three categories. These are shown in the figure below, and they are forecasting, classification and regression. Forecasting is the most common of the three: given a set of time-indexed data points, the task is to predict what this data might do in the future. An example could be - given the temperature measurements from the last week, can you forecast what the temperature might be tomorrow? Classification is the next most common: given a set of time-indexed data points, the task is to say in which (pre-defined) category the data falls. Imagine we've collected walking data from humans, dogs, and cats (for example, with the accelerometer on an Apple Watch). We want to be able to determine which of these created the data when we do classification. Finally, regression tries to give a numerical value to a set of time-indexed data. For example, using data accelerometer data collected from a human, we want to be able to say at what speed the person is walking. In the past, using machine learning algorithms for these tasks has proven to be quite challenging. It turns out that traditional ML techniques are bad at modelling long-range dependencies (so a model will just look at data that is close together when making a prediction and not data that is far apart), for various reasons outlined in this great article. This is where the transformer comes in, introduced in 2017 [2].


What is a transformer? Please note in the following discussion, I might refer to the use of sentences as input to models. Transformers were originally created for language processing tasks, and sometimes are easier to explain using examples from language rather than time series data. A sentence can be thought of as a piece of time series data - the order that the words are in matters. I will try to explain this very simply, but for those wanting a thorough introduction, there are some brilliant articles on Medium, such as this one. Transformers originally came about for language modeling tasks. Previous models, such as the RNN, would take an input sequence (such as a sentence, or time-series data) and process it sequentially, meaning one by one in the order that the data came in. This caused the issue of vanishing gradients and made them unsuitable for modeling long-range dependencies. Transformer models, however, attempt to capture relationships between all elements in the input sequence simultaneously. They do this using self-attention. For a mathematical introduction to self-attention, please see here. Below is a simple intuition behind what it does. Imagine we are trying to translate the following sentence from one language to another: If I was translating this to French, I would want to make sure that I know the words for fox and dog in French. I would also want to be able to get the order of them correct (i.e., the fox jumps over the dog, not the other way around). I would maybe pay less attention to the word the, as it's not the subject or object of the sentence. Attention tries to do just this - it tries to look at an input and see what bits of the input are more important to the task at hand. Imagine we had the same sentence, but the task at hand was to extract colours from the sentence. Attention would weigh heavily on the word brown and weigh less on the other words in the sentence. In the example in the gif above, we are translating from French to English. If hidden state is more highlighted it means that attention weights it more heavily. You'll see that when generating the English I, only the French Je is paid attention to. However, for the English a, both suis and etudiant are paid attention to. This is because in English, we need to know what comes after the connective to know whether to use a or an in our translation. If the final word had been pomme, the translation would have been I am an apple. This is why attention is useful - it lets the model understand what is important for the task at hand. Transformers for time series data Now we have some kind of intuition behind the attention mechanism. Let's dive into what a transformer actually is. Below is an image of a very simple transformer. At this level, all that it is is an encoder and decoder network. We can think of an encoder as something that takes input and puts it in a format that is useful for the model to have for the task at hand (e.g., imagine extracting just the background color from a series of images because you want to know what color the sky was in the images). A decoder does the opposite - it takes the machine-readable format (the encoding) and puts it back into a format that a human can understand (e.g., from before, taking the encoding and outputting the word blue). In the transformer, the encoder performs self-attention, as described above (to weigh the various parts of the input more or less), and then passes it through a feed-forward layer - see here for a good introduction. The decoder does the opposite. In the original transformer proposed by [2], the authors actually made the encoder run 6 times (i.e., the output was passed into another identical encoder which in turn had self-attention and feed-forward layers a further 5 times), and similar for the decoder. Adapting the transformer for classification/regression The decoder is needed in the case of machine translation because we expect the output to be a sequence of words, and not just a single number. In the time series data case, this would be similar to forecasting, where the output is a series of data points. See here and here for examples of this. However, in our case, we actually just care about outputting a single number (in the case of regression), or a single vector (in the case of classification). So we can actually just get rid of the decoder part of the transformer entirely and focus only on the encoder. The encoder will work in exactly the same way, providing a useful representation of the data for the model. Once we have the encodings, we then simply add a linear layer on top that flattens the output of the encoder and projects it to one number. Adapting the transformer for multivariate data Great - we now have a transformer-based model that can take a series of time-indexed datapoints and output a classification or regression. Up until now, we have considered univariate time series. This means that for each point in time, we just have one measurement (for example, temperature). Sadly, most data is not so simple - even data from a single accelerometer actually has three points for each time-step (one for each axis, x, y, and z). In my case - using transformers for disease diagnostics using motion sensor data, I have 17 sensors around the body (totalling 17x3=51 datapoints per time-step). Will our transformer work for this data? Well... kind of. It won't break, and it will still give some intelligible outputs, but we can make it better. Using this formulation of attention, attention will treat all of the datapoints at one time step the same, and weigh them the same. Whilst this is ok, it means that our model will only really model variations in time, and not in space. In my use-case, (I'm drastically oversimplifying here) imagine that a telltale sign of the disease is that when you move one arm in a certain way there is a twitch in your foot at exactly the same time. The model would not be able to pick up on that, losing out on that important information. What we actually want is depicted on the right of the above image. Whilst on the left, each variable at a given time-step has the same amount of attention given to it, on the right we have attention that can vary both in the space and time dimensions. The simplest way to achieve this is described by [3] - we linearise our input data. This means that rather than at each time-step having a vector to describe our data (one value for each variable), we simply repeat the time index multiple times to cover our data. For example, if we have T time-steps and N variables, our input data would originally have a shape of TxN. In this modified version, we'd instead have an input of NT x 1. A note on complexity: the computational complexity of a transformer scales with the width of the vector (originally T, now NT) as O(T²). Going from a vector of width T to NT will now give us a computational complexity of O(N²T²). In my case, with 51 dimensions, this would mean a complexity that is 2600 times greater than in the original case. This is far from ideal, and the paper [3] discusses various ways to mitigate this (one for another post!).


Test case - human activity recognition (HAR) To test if my model works, I am going to benchmark it on a multivariate timeseries classification problem, which is human activity recognition. I have taken the dataset from this resource, which is a great database of time series classification problems. The original dataset is presented in [4]. In the study a total of 24 participants in a range of gender, age, weight, and height performed 6 activities in 15 trials in the same environment and conditions: walking downstairs, walking upstairs, walking, jogging, sitting, and standing. The task is: given a 2.5s snippet of data taken from an iPhone 6s accelerometer, can we detect which of these activities is being performed? The dataset is therefore a time-series with 12 dimensions (3 dimensions for each attitude, gravity, user acceleration, and rotation rate). In the original paper, using a variety of (non-transformer) techniques, the best-performing models were able to obtain an accuracy of ~92%. When I tried this with the vanilla transformer above, training for less than 10 minutes and doing no hyperparameter tuning, I was able to obtain an accuracy of 90%. This shows great promise that with more training time and better hyperparameters, my transformer should be able to beat this baseline. The improved transformer should be able to do even better! Given the purpose of this test was to just ensure it was working as expected, I did not decide to push the performance any further.


Appendix: implementation in PyTorch As usual, my implementation is available on my GitHub. Please ensure you download the data from here. My implementation has a Transformer class, as well as a Trainer and DataHandler for ease of training in PyTorch. The transformer has been taken and adapted from this great repo (and paper!).


References [1] Wen, Q., Zhou, T., Zhang, C., Chen, W., Ma, Z., Yan, J. and Sun, L., 2022. Transformers in time series: A survey. arXiv preprint arXiv:2202.07125. [2] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł. and Polosukhin, I., 2017. Attention is all you need. Advances in neural information processing systems, 30. [3] Grigsby, J., Wang, Z. and Qi, Y., 2021. Long-range transformers for dynamic spatiotemporal forecasting. arXiv preprint arXiv:2109.12218. [4] Malekzadeh, M., Clegg, R.G., Cavallaro, A. and Haddadi, H., 2019, April. Mobile sensor data anonymization. In Proceedings of the international conference on internet of things design and implementation (pp. 4958).


Test-Driven Application Development with Large Language Models

TLDR; Developing applications with Large Language Models (LLMs) using Test Driven Development (TDD) presents several challenges and insights. Testing generative models like LLMs is difficult given their complexity and the 'creative' nature of output - but it is crucial for automation and safety.The nature of testing is shifting; it's easier to discriminate than generate. Therefore using another LLM to test the outputs of the original LLM can be beneficial.Not all LLMs are created equally. Thus, the selection is crucial and should be done according to the use case and by following relevant benchmarks as well as privacy, security, and cost. considerations. Robustness is another important consideration and can be done using perturbation testing, ensuring that similar inputs give similar outputs.Duck typing in Python is powerful but can cause integration headaches like runtime errors, incorrectness, and difficulties in generating code and documentation. Tools like MyPy and Pydantic are a must for type-checking and output parsing.Execution testing involves checking the output of LLMs. Two ways to accomplish this are compile-time property testing (tuning instructions or prompts) and run-time output testing (using another LLM to auto-generate test cases).Bug discovery, planning, and iteration require an interactive approach with the LLM, prioritizing recall over precision. Using an LLM to enumerate and iterate on use cases and test cases is suggested, with the added advantage of LLMs being excellent at summarizing and categorizing information.


Types of testing - by Development Stage First things first, testing generative models is tough due to the 'creative' nature, but it is fundamentally essential, especially with instruction-tuned and safety aligned models. One of the frequent failure cases is the model refusing to produce an output and answering with "As a language model .." . While developing the LLM application in a TDD way, I have found it helpful to think of the various types of tests needed by the 'stage' of development.


Stage One: LLM Selection The starting point of my process was realizing that not all LLMs are identical. The demands of particular use-cases may necessitate specialized LLMs, taking into account quality and privacy requirements. Here's what I focused on during selection:


1. Benchmarks GPT-4 has released performance on several benchmarks . Likewise, the Open LLM Leaderboard aims to objectively track, rank, and evaluate the proliferation of large language models (LLMs) and chatbots, sifting through the hype to showcase genuine progress in the field. The models are evaluated on four main benchmarks from Eleuther AI-Language Model Evaluation Harness, ensuring comprehensive assessment across diverse tasks. The leaderboard also enables community members to submit their Transformer models for automated evaluation on the platform's GPU cluster. Even models with delta-weights for non-commercial licenses are eligible. The selected benchmarks - AI2 Reasoning Challenge, HellaSwag, MMLU, and TruthfulQA - cover a broad spectrum of reasoning and general knowledge, tested in 0-shot and few-shot scenarios.


2. Perturbation testing: The general approach is as follows: Introduce Perturbations: Utilize another LLM to rephrase the original prompt while retaining its semantic essence. Then, supply the perturbed and original prompts to the LLM under assessment.Analyze Generated Outputs: Assess the generated responses for either accuracy (if a standard response is available) or consistency (judged by the similarity of generated outputs if no standard response exists). - The essence of this articleIterate: Any errors should lead insights towards better prompts, different models, better instruction tuning, etc. Read the excellent article and tool by FidderAI on Perturbation testing.


Stage Two: Type checking and Integration Testing Python, the lingua franca of this domain, provided the flexibility of duck-typing, crucial for quick iterations. However, to develop robust software engineering applications, I found it indispensable to ensure thorough syntactic correctness. I would advise always using type-hints, but handle or fail gracefully instead of enforcing breaking errors during run time. Here are some tools I found useful: MyPy: An efficient static type checker for Python.Pydantic: It has become my go-to tool for output parsing. It's high extensibility and excellent integration with Langchain are bonus points.Langchain: The output parsers in Langchain can be employed to create repeated instructions for output as well as automated tests. Beyond this, integration testing is not much different for LLM applications than for other software applications, so I won't go into much detail.


Stage Three: Runtime Output Testing Testing the outputs of a generative model can be tricky, a.k.a. The 'test oracle' problem. Nevertheless, using the principle that discrimination is less complex than generation can be helpful here. Property-based software testing, such as metamorphic testing, is a useful approach for addressing the test oracle problem as well as for test case generation. In essence, this is done by testing on a known or derivable property. For example, when testing a response to a query, how many distinct cities are there in a particular state; how do we determine if the results of this are correct and complete? This is a test Oracle problem. Based on a metamorphic relation, we can ask the LLM to tell us how many cities are there in the state that begin with the letters A through M. This should return a subset of the previous results. A violation of this expectation would similarly reveal a failure of the system. In this case, several tests can be determined either during development or live during runtime, as explained below. Property-based testing and MT in general were originally proposed as a software verification technique, the concepts cover verification, validation, software quality assessment.


Pre-composed Property testing: This is particularly useful for instruction tuning or prompt engineering. For instance, if the LLM is supposed to summarize a webpage while eliminating all hyperlinks and emojis, I would start by writing straightforward procedural test cases or LLM prompts for these tasks. This approach works well if the types of output expected are known in advance. In these scenarios, the testing isn't much different than what is possible by using testing frameworks like Robot, Pytest, Unittest, etc. Using semantic similarity with a threshold for fuzziness is useful. For instance: Application to extract and summarize the 'main' article in a webpage while ignoring extra links, comments, etc. Then a battery of tests can be designed using existing known web pages. Positive examples: Semantics matches the main page. Negative examples: Unrelated topicsApplication to remove negative sentiments, emojis, etc., and summarize text to less than 3 sentences. Test using procedural or LLM models to test those cases specifically. This is an important topic, so let's dive further. It is counter-intuitive to think of using another model to test the output of a model. If the second model can 'grade' the output, why not use the second model to generate the output in the first place? For instance, In Generative Adversarial Networks (GANs), two components, the Generator and the Discriminator, interact in a game-theoretic way. The Generator creates new data instances while the Discriminator assesses these instances for authenticity. This structure exemplifies the idea that "discrimination is easier than generation." The Generator's task is to generate new data, such as images, that convincingly mimic real-world examples. This requires learning complex patterns and subtleties, making it a difficult and intricate job. In contrast, the Discriminator's role is to classify whether given data instances are real or generated. This task is relatively simpler as it involves identifying distinguishing patterns and features without the need to understand how to create the data instances. Take the following example. The first image was created using Stable Diffusion (by the author) for the prompt: The generative model needs to not only understand what is 'cat' is well enough to create it, but also needs to understand what a bottle is, what big Ben is, and what it means to be a rainy day, and compose it all together (it does give the cat a human hand, but oh well!). However, the discriminative model only needs to understand each of these concepts in isolation, as in the figure below. Thus, even if you use a more powerful model (like gpt-4) to generate a response, it can be tested for correctness - execution test or output test - even by using the same or a lesser model (gpt 3 for instance)


Run time output testing: These types of tests are useful if the nature of the application or task is not known in advance. For instance, any application that accepts a 'prompt' from a user and does work dynamically. - summarize this, convert this to SQL etc. In these cases, not all is lost. As we can easily use another LLM to design quick correctness tests on the fly. (see below)


Stage Four: Bug Discovery, Planning, and Iteration A noteworthy lesson I've learned is, "You can test for the presence of a bug but never for the absence of one." At this stage, it is paramount to interact with the LLM in a more inquisitive and explorative manner. Here, I've found it more beneficial to prioritize recall over precision. Here's how I apply this: Use an LLM to generate and test use cases: For each LLM application, I employed an LLM to both produce and test use cases. While these might already be defined in some instances, the creative nature of LLMs can prove advantageous.Iterate on test cases: I've discovered that LLMs are exceptional partners in summarizing and categorizing information and ideas. This is extremely useful when iterating on the test cases for each use case.Repeat or Auto-repeat with a low temperature: Drill down and consistently repeat or automate this process, adjusting with a low-temperature setting for more reliable outputs.


Conclusion In conclusion, it's still early phases, and things are in flux in the ever-evolving landscape of generative AI and LLMs, but one thing is for sure, the testing process seems to be as vital as development. I hope that these insights from my personal journey are helpful to you in some way. This article is by far not complete as I haven't talked about other forms of testing, especially for security around prompt injection, jail breaks, etc. However, that is a different topic. If you like the article, follow me here and on linkedin and youtube for more. Comment your thoughts.


References Open LLM Benchmark (https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)Langchain Output ParsersPydanticGuidance - enables you to control LLM by interleaving generation, prompting, and logical control.Metamorphic Testing - property-based software testing frameworkAdaptive Testing and Debugging Language ModelsAuditor by FidderAIMetamorphic Testing


5 Use Cases to Leverage Large Language Models in Customer Service Interactions

What is a Large Language Model Imagine a large language model (LLM) as a highly intelligent AI assistant that specializes in understanding and processing human language. To become such an adept language processor, the LLM underwent extensive training on massive volumes of text data. During this training process, it learned intricate patterns, grammar rules, and the meaning behind words and sentences. Consequently, it gained the remarkable ability to generate responses that are not only coherent but also contextually appropriate. LLMs excel at a variety of language-related tasks. For instance, they can comprehend and interpret natural language, generate text that resembles human writing, translate languages, summarize lengthy texts, and perform many other language-centric activities. A prime example of a powerful LLM is OpenAI's GPT-3, which has been meticulously trained on extensive datasets. As a result, it possesses the remarkable capacity to comprehend and generate text that is strikingly similar to what a human would produce. It can effortlessly tackle a wide range of subjects and adapt to various contexts. In the next few sections, we will discuss how large language models can be helpful in Optimizing Customer Service interactions.


1. Contextual Response Generation Lets take an example of a customer who has previously contacted your support team regarding a specific product issue. When they reach out again, wouldn't it be fantastic if your support team could instantly recall all prior interactions and respond keeping in to account all the past conversation context? By analyzing a customer's previous interactions, LLMs can generate personalized responses to the customer's request. For instance, if a discount was promised to the customer in a prior email, this context is read and incorporated in the response back to the customer


2. Triage and Categorization A vital part of customer service is managing and sorting incoming customer emails and forward it to the right department to follow through. LLMs can help analyze these emails, automatically categorizing them based on content, which might range from "Technical Support" to "Billing Inquiries". If a customer emails about a "billing error", the LLM identifies the email, categorizes the email under "Billing Inquiry", and even assigns a priority level based on predefined criteria for the right department to act upon. This helps streamline the support process, enabling agents to address critical or time-sensitive inquiries promptly. This article talks about how to build out domain-specific models for your use cases.


3. Quick automated self-help responses Let's say a customer sends an email requesting guidance on account setup. Say this is a well understood problem and can be handled automatically. Here, LLMs can generate comprehensive instructions tailored to the specific situation. They can include step-by-step guidelines and relevant links and even anticipate common follow-up questions or concerns, ensuring the customer receives a thorough response to their inquiry. While the LLM can generate a draft response, its important to realize that human review, and customization remain critical. Support agents can add a personal touch, ensuring the response aligns perfectly with the customer's specific needs and circumstances.


4. Knowledge Base Integration Most companies have a knowledge base of questions and answers that has been answered before which is a good source to tap in to. But what if your customer service could go a step further and directly provide customers with the most relevant articles or resources from your knowledge base? When a customer emails about a specific product issue, the LLM can analyze the customer's query, search the company's knowledge base for relevant articles, and provide a summary or excerpt of the article directly in the response. Not only does this empower customers to resolve their issues independently, but it also promotes self-service options and ensures that the most up-to-date information is readily available. This article walks through how to integrate your private documents with LLMs.


5. Sentiment Detection and Emotion Recognition The ability to understand a customer's sentiment or emotion can significantly enhance the way businesses respond to inquiries. LLMs can analyze customer emails for sentiment detection, sarcasm detection, and emotion recognition. This means that whether a customer's tone is positive, negative, or neutral, or if it contains elements of sarcasm, the LLM can detect these nuances. It can even recognize and interpret emotions such as anger, frustration, or happiness, allowing businesses to prioritize responses, escalate urgent cases, and provide empathetic support tailored to the customer's emotional state. Empathizing with the customer and correctly responding to them based on customer's state of mind will go a long way to help nurture customer relationship.


Conclusion In today's subscription driven environment, customer is the king. Meeting customer expectations is very important to thrive in a competitive landscape. Companies are looking for ways to augment traditional approaches and help forge meaningful and personalized connections with their customers. LLMs has the potential to enable businesses to offer personalized experiences by generating contextually relevant responses. They can take in to account the past conversations as well as customers sentiment and provide tailored and empathetic response, providing a sense of personalized attention. For instance, imagine a customer who contacts a company's support team expressing disappointment with a recent service disruption. Through sentiment analysis, an LLM can detect the negative sentiment, lookup any past conversations and generate a response that acknowledges and empathizes with the customer's frustration. This personalized and empathetic approach can help restore trust and loyalty in the customer, strengthening the brand-customer relationship. Note that a human review of the response generated is critical. In conclusion, incorporating LLMs into customer service operations allows businesses to augment conventional approaches, delivering personalized, efficient, and empathetic experiences. Companies have the opportunity to revolutionize their customer service operations, elevate the overall customer experience, and cultivate satisfied customers who become invaluable brand ambassadors.


This AI newsletter is all you need #48

What happened this week in AI by Louie This week in AI has witnessed significant announcements from OpenAI as well as increasing pressure on the company, including Sam Altman's testimony in the Senate, the release of the ChatGPT iOS app, and plugins for ChatGPT+ users. During a Senate Committee hearing, OpenAI CEO Sam Altman, along with other industry leaders, largely agreed on the need to regulate the rapidly advancing AI technology developed by companies such as OpenAI, Google, and Microsoft. While we agree on the need for AI regulation, it has to be very carefully managed and executed and one key risk with more regulation is its potential to entrench the market leaders with high regulatory barriers to competition. For this reason some question the motives of large tech companies in calling for AI regulation. As a counter to this, in its "Governance of Superintelligence" OpenAI is advocating for the most burdensome regulation to only kick in over "a significant capability threshold". OpenAI was also busy with new releases including the "Continue generating" button in ChatGPT, aiming to enhance user experience. This function eliminates the need to manually input "continue" for longer responses and inputs, streamlining the process. In addition, OpenAI has recently launched the official ChatGPT iOS app and plans to release an Android version soon. The iOS app is free to use and incorporates a speech-to-text feature, enabling users to interact with ChatGPT through speech. Furthermore, OpenAI has introduced plugins exclusively for ChatGPT Plus subscribers, granting them access to web browsing and all the beta plugins. These beta plugins offer a diverse array of functionalities, including text-to-audio conversions and real-time data updates on stocks and cryptocurrencies. As open-source AI models gain traction, and debate continues on the pros and cons of open source vs closed API models for AI going forward, OpenAI also announced the upcoming release of a new open-source large language model. Perhaps this comes due to increased pressure from the open-source LLM world. The specific details of OpenAI's forthcoming open-source model have not been disclosed yet. The momentum of open-source AI also continued at Stability AI this week, which has made its DreamStudio platform open-source. The Stable Diffusion model that powers it has been open source from the start, but it is an interesting progression to see the commercial interface become open source also and a trend we will watch for more broadly. - Louie Peters - Towards AI Co-founder and CEO


This AI newsletter is all you need #48

Hottest News Stability AI releases StableStudio in the latest push for open-source AI Stability AI is unveiling an open-source version of DreamStudio, a commercial interface designed for their AI image generator model called Stable Diffusion. This release empowers community developers to enhance and experiment with the interface, potentially leading to valuable improvements for the company. 2. Introducing the ChatGPT app for iOS ChatGPT has launched an iOS app that includes open-source speech recognition for its subscribers. The app, available for free, allows users to sync their chat history across multiple devices. ChatGPT Plus subscribers get exclusive access to GPT-4's capabilities, early access to features, and faster response times, all on iOS. 3. Meta bets big on AI with custom chips and a supercomputer Meta has revealed its internal endeavors to develop AI workload infrastructure, specifically focusing on generative AI. They have introduced a new chip called the Meta Training and Inference Accelerator (MTIA). The MTIA is an ASIC that incorporates various circuits onto a single board, allowing it to be programmed to carry out one or many tasks in parallel. 4. Apple reveals new accessibility features, like custom text-to-speech voices Apple previewed a suite of new features aimed at enhancing cognitive, vision, and speech accessibility. These tools are scheduled to be introduced on iPhone, iPad, and Mac devices later this year. Specifically designed for individuals with cognitive disabilities, Assistive Access is set to roll out across a range of Apple products in the coming year. 5. Zoom will soon integrate Anthropic's chatbot across its platform Zoom has announced a partnership with Anthropic to integrate the latter's AI assistant into its productivity platform. The initial implementation of this integration will involve incorporating Claude into Zoom's Contact Center product. The aim is to enhance customer support services by leveraging Claude's capabilities to develop improved self-service features.


This AI newsletter is all you need #48

5-minute reads/videos to keep you learning Auto-Evaluation of Anthropic 100k Context Window Anthropic's 100k-token Claude model has ignited a debate regarding the necessity of document retrieval in question-answering (Q&A) systems. The retriever-less architecture performs well but has higher latency than retriever-based approaches, and is most suitable for small corpora and non-critical latency apps. This discussion raises important considerations regarding the trade-off between retrieval-based and retriever-less approaches in Q&A systems. 2. The "Three-Hills" Model for evaluating GPT startups The "Three-Hills" Model provides an evaluation framework for assessing the success of GPT startups based on three key factors: "Productivity Enhancements," "Non-zero-sum-game Value," and "Moat = Value from Context." These factors are assessed across three distinct levels. Additionally, the model identifies three categories of moats that can help prevent generic GPT displacement. 3. ChatGPT vs. Bard: A realistic comparison The article conducts an unscientific field experiment to compare the performance of ChatGPT and Bard. The author subjects both models to various challenges, including summarizing lengthy content, composing a Python function, devising social media marketing tactics, and more. 4. A Decade In Data Engineering - What Has Changed? Over ten years, numerous significant breakthroughs and tools emerged in the field of data engineering. It played a pivotal role in transforming the way data engineers operate. This article provides an overview of the history of data engineering, highlighting the key milestones that have shaped the field and brought us to the present state. 5. What is AI Art and How is it Created? This article provides a comprehensive exploration of the concept of AI art, examining its meaning, the historical background of AI-generated art, the various types of AI employed in art generation, the ways artists utilize AI, the challenges involved in creating AI art, the ethical concerns associated with AI-generated art, and more.


This AI newsletter is all you need #48

Papers & Repositories ray-project/llm-numbers At Google, there was a document put together by Jeff Dean, the legendary engineer, called Numbers Every Engineer should know. Having a similar set of numbers for LLM (Language Model) developers is highly beneficial for quick estimations. This repository aims to share specific numbers utilized by Anyscale, explaining their significance and demonstrating how they can be leveraged to one's advantage. 2. CodeT5+: Open Code for Large Language Models Introducing CodeT5+: a collection of encoder-decoder LLMs designed for code, allowing flexible combinations of component modules to cater to diverse downstream code tasks. The analysis highlights the models' state-of-the-art (SoTA) performance on various code-related tasks, including code generation and completion, math programming, and text-to-code retrieval. 3. Tree of Thoughts: Deliberate Problem Solving The paper presents an innovative framework for language model inference called Tree of Thoughts (ToT). ToT builds upon the Chain of Thought approach commonly used to prompt language models but with the added capability of exploring coherent units of text referred to as "thoughts." These thoughts serve as intermediate steps toward problem-solving. 4. mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences mLongT5 enhances the architecture of LongT5 by utilizing the multilingual datasets employed in pretraining mT5 and the pretraining tasks of UL2. The model underwent evaluation across several multilingual summarization and question-answering tasks, demonstrating superior performance compared to existing multilingual models like mBART or M-BERT. 5. Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback This research delves into the possibility of utilizing different AI models, taking on the roles of buyer, seller, and critic in a negotiation game, to learn and enhance their deal-making abilities without human intervention. The study discovered that specific models were capable of learning from the game and improving their negotiation strategies. Enjoy these papers and news summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #48

Weekly AI Podcast In this week's episode of the "What's AI" podcast, Louis Bouchard interviews David Mertz, a senior developer, data scientist, and author. They delve into the fascinating world of generative AI and its applications in coding, including AI coding assistants like Copilot and ChatGPT. The discussion also addresses copyright concerns associated with generative AI and delves into the intricate issue of moral rights and attribution. David highlights the strengths and weaknesses of these assistants, elucidating how they can both assist and impede developers in their coding endeavors. You can find the podcast on YouTube, Spotify, or Apple Podcasts.


This AI newsletter is all you need #48

Meme of the week! Meme shared by NEON#8052


This AI newsletter is all you need #48

Featured Community post from the Discord Chris M#0847 recently introduced DataLab, an open-source Python implementation that is easy to use for all types of data, including images, text, tabular data, and audio. In Software 2.0, data has become the new code, models are the new compiler, and manual data validation is the new unit test. Datalab combines any ML model with innovative data quality algorithms to provide a linter for the Software 2.0 stack. It automatically analyzes datasets for "bugs" by adaptively checking for issues commonly found in real-world ML datasets, without requiring you to specify their potential form. Datalab's checks consider all the relevant information learned by your trained ML model. Find it on GitHub or check the Jupyter tutorial here and support a fellow community member. Share your thoughts in the thread here!


This AI newsletter is all you need #48

AI poll of the week! Join the discussion on Discord.


This AI newsletter is all you need #48

Article of the week Compare ChatGPT to Machine Learning Techniques for Sentiment Analysis in 2023 by Courtlin Holt-Nguyen Sentiment analysis, powered by NLP (Natural Language Processing), enables AI systems to comprehend the opinions and emotions expressed in text. The advanced NLP capabilities of ChatGPT have garnered significant attention in recent months. However, the question remains: Is ChatGPT superior to existing machine learning NLP techniques? This article aims to demonstrate ChatGPT's notably enhanced accuracy compared to pre-ChatGPT NLP techniques.


This AI newsletter is all you need #48

Our must-read articles Understanding LangChain : Part:2 by Chinmay Bhalerao Building a Q&A Bot over Private Documents with OpenAI and LangChain by Sriram Parthasarathy If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.


This AI newsletter is all you need #48

Job offers Data Processing Specialist @Hyperscience (Remote) Senior Software Engineer - Backend @Harness (Mountain View, USA/ Hybrid) Machine Learning Engineer @ConverseNow.ai ( Buenos Aires, Argentina/Hybrid) Senior AI Data Trainer @Cohere (Remote) Staff Machine Learning Engineer @Clari (Remote) Senior Software Engineer (Python) @Reserv (Remote) Software Engineering Manager @Lucidya (Remote) Interested in sharing a job opportunity here? Contact sponsors@towardsai.net. If you are preparing your next machine learning interview, don't hesitate to check out our leading interview preparation website, confetti!


How to Fine-Tune Meta SAM

Setup First, we need to install all the necessary dependencies as specified in the Meta SAM readme as follows. To understand better, here will provides three implementations of how to fine-tune SAM: Fine-tuning SAM for new images based on the foundation model checkpointFine-tuning SAM for new images with masks to improve domain-specific accuracyFine-tuning SAM according to SAM preferred optimizer and dataset formats


How to Fine-Tune Meta SAM

First Fine-Tuning Example Based on the Checkpoint Training SAM requires 256 A100 GPUs for 3~5 days. It is very expensive to train the entire model from scratch each time. Training or fine-tuning from a pre-trained foundation model checkpoint is strongly recommended, which is also the reference baseline for transfer learning. SAM provides sam_model_registry method to load its model checkpoint. Below illustrates how to fine-tune a foundation model like SAM based on the checkpoint. In the above, "<model_type>" and "<path/to/checkpoint>" need to be replaced with the appropriate model type and path to the checkpoint file. Also, '<path_to_images>' is the path to the image dataset. Here, we are fine-tuning the entire model. In some cases, we may want to freeze the weights of some layers (typically the earlier layers) and only fine-tune the later layers. This can be done by setting the requires_grad the attribute of the parameters you want to freeze to False. Also, it is intended to illustrate and simply use CrossEntropyLoss, which may not be the best choice for a task. Depending on the specific use case, we may want to use a different loss function. For instance, for a segmentation task, we might want to use a loss function that is more suitable for comparing images, such as the Dice loss or the Jaccard/Intersection over Union loss. In the beginning, we fine-tune the model with unlabeled data (i.e., in an unsupervised manner). Next, we have labels for images by modifying the CustomDataset class to load the labels and adjust the loss computation accordingly.


How to Fine-Tune Meta SAM

Domain-Specific Fine-Tuning with Masks For a segmentation task with labeled data, we need to make a few adjustments: The dataset should return both images and their corresponding segmentation masks.The loss function should be suitable for segmentation tasks, such as the Dice loss or the Jaccard/Intersection over Union loss. For simplicity, we'll use the BCEWithLogitsLoss, a combination of a Sigmoid layer and the BCELoss (Binary Cross Entropy Loss) in one single class. Below is the adjusted Python code for this requirement: Again, "<model_type>" and "<path/to/checkpoint>" need to be replaced with the appropriate model type and path to the checkpoint file. And '<path_to_images>' and '<path_to_marks>' are the paths to the image dataset and mask dataset, respectively. Here, assume the masks are in the same format as the images and are named identically to their corresponding images. If the masks are in a different format or are named differently, we need to adjust the __getitem__ method of the CustomDataset class accordingly. It also assumes the masks are single-channel (grayscale) images where the pixel value indicates the class of that pixel. If the masks are in a different format, we need to adjust the code accordingly. The BCEWithLogitsLoss is suitable for binary segmentation tasks (i.e., tasks where each pixel can belong to one of two classes). If we have a multi-class segmentation task (i.e., each pixel can belong to one of more than two classes), we need to use a different loss function, such as the CrossEntropyLoss. In this case, we also need to ensure the masks are in the correct format (a single-channel image where the pixel value indicates the class of that pixel).


How to Fine-Tune Meta SAM

SAM Specific Fine-Tuning With the above practices, we can try to fine-tune Meta SAM with prompt encoding for domain-specific images. This is closer to what SAM supports. In the above, input_size and original_image_size are the dimensions of the input images and original images, respectively. 'custom_dataset' is the actual dataset loader that yields images, bounding boxes (or other prompts), and ground truth masks. It is a JSON object like: After the training loop, the code saves the fine-tuned model weights to a file named 'fine_tuned_sam.pth'. We can load these weights later for inference on similar data or further fine-tuning.