This AI newsletter is all you need #25

Hottest News Pytorch just introduced PyTorch 2.0!Along with a lot of cool features, PyTorch 2.0 introduces torch.compile, a compiled mode that accelerates your model without needing to change your model code. On 163 open-source models ranging across vision, NLP, and others, using 2.0 speeds up training by 3876%.A ChatGPT Chrome extensionIntegrate ChatGPT into every text box on the internet, such as a What's app message box or Twitter post you are writing, thanks to work by Gabe Ragland. Just directly ask ChatGPT to help you out.The AI Film FestivalRunway is introducing the first annual AI Film Festival. A celebration of the art and artists making the impossible at the forefront of AI filmmaking! All films will be reviewed by a panel of globally renowned filmmakers, creators, and AI innovators.


This AI newsletter is all you need #25

Three 5-minute reads/videos to keep you learning Overview of Computer Vision Tasks & ApplicationsIn this article, Youssef Hosni explores the main computer vision tasks and the most popular application for each task. Learn more about this field and how AI actually helps the vision industry!ChatGPT is taking over the internet. But do you know how it actually works? It's so clever...Here's an explanation using simple words (in a great Twitter thread by my friend Lior)...Teaching ChatGPT about the birds and the beesIn this article, my friend Jeremy Pinto shares a great use case of ChatGPT using it to help him code a machine learning model. He shares his experience, the model's weaknesses and strengths, and more. Really interesting and applied article for anyone interested in using ChatGPT and practicing coding. Enjoy these papers and summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #25

Featured Community post from the Discord by Louis As head of community, I will be conducting an interview with someone in charge of the hiring process for a big AI company. I would love to know if you would have any questions you'd like to ask about: The interview processThe interviews themselvesThe ideal resumeWhat they are looking for in your profile, etc. Please send your questions to me on Discord or by email! The interview will be shared on my YouTube channel What's AI and on the Discord server around the winter holidays.


This AI newsletter is all you need #25

AI poll of the week! Join the discussion on Discord.


This AI newsletter is all you need #25

Towards AI Article of the week Paper Review Monolith: Towards Better Recommendation Systems by Building Blocks The author discusses a recently published paper: Real-Time Recommendation System With Collisionless Embedding Table. Recommendation systems are the most prevalent application of machine learning. We see them on every social media platform, streaming website, online marketplace, and many others. Despite their prevalence, there's plenty of room for improvement in how recommendation systems are designed.


This AI newsletter is all you need #25

Other must-read articles Window Functions in SQL and PySpark ( Notebook) by Muttineni Sai Rohith DeepMind's DeepNash Plays The Game "Stratego" by Dr. Mandar Karhade, MD. PhD. If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.


This AI newsletter is all you need #25

Featured jobs this week Product Lead, AI/ML @ Inworld.AI (USA, Remote) Data Engineer @ Sword (USA, Remote) Machine Learning Engineer @ Covariant (Berkeley, CA) AI Delivery Manager (Healthcare) @ ClosedLoop (Remote) Senior/Staff Machine Learning Engineer, Infrastructure at Earnin (USA, Remote) Interested in sharing a job opportunity here? Contact sponsors@towardsai.net or post the opportunity in our #hiring channel on discord!


Popular Data Science Python Packages You Need to Know

1. Dask Dask is a Python package for parallelization computing built on top popular packages such as Pandas, Numpy, and others. There are two main features of Dask: Task Scheduling for automating tasks and scheduling the activity,Big Data Collection for parallel data processing data. In summary, Dask simplifies the Pandas Python package, such as a data frame object. Still, you could schedule the activity and have a faster execution time by parallelizing the process. Let's try Dask's simple functions. But first, we must install the package (Anaconda already installed Dask by default). As a starter, we could initiate the Dask dashboard to monitor our activity with Dask. If we click on the dashboard link, we will end up with the following image. You can play around with the dashboard and see the function, but let's try out the Dask data processing function. We could create a dask data frame from the Pandas object. The Dask data frame is similar to the Pandas data frame, but it is lazy, which means the result would not be shown on the Jupyter Notebook. To display the data frame object, we need to run it with the computemethod. The data selection method in the Dask data frame is also similar to Pandas. Dask is about fast computing with parallelization. Dask is good to use when we have big data and need high computational performance. However, smaller data is preferable to be processed with Pandas. If interested in Dask, refer to the documentation to learn further.


2. Faker Faker is a Python package for generating synthetic data with easy steps. It's one of the basic packages which become the base of many advanced data synthetic packages. Let's try Faker to see how it works. First, we need to install the package. The first step to using the Faker package is to initiate the Faker class. After the class was initiated, we would try various methods to generate synthetic data. Using the method name, address, and text respectively we end up with different synthetic data. Each time we ran the method, we would end up with something new. Faker is not limited to the above example as there are still many more variables with Faker we could generate; for example, bank, credit score, and many more. If you want to explore more, please refer to the documentation.


3. Dulwich Dulwich is a Python package for implementing Git without relying on the Git file and protocol. The package purely accesses the Git capability with Python without any hassle. Let's try the package to know better. First, we need to install Dulwich. Using Dulwich would follow a similar procedure on Git. So, let's try to initiate the repository. The above code would initiate a repository called 'myrepo'. Next, let's try to stage a file that we have created. In the example above, we create a CSV file and stage it to our repository. Then, we would commit the file with the following code. The commit would provide us with the ID we can use later for versioning. If we want to see the commit message, we can use the following code. Finally, if we want to access the Git log, we could do that with the code below. You can access the documentation here if you want to know the full capability.


4. Pendulum Pendulum is a Python package to make datetime data processing easy. It is intended to replace the standard datetime class function with a more intuitive function. Let's start by installing the package. In the example below, we could initiate our timezone and access the time to shift it with the following code. With the Pendulum, it's also easy to do date iteration to acquire the data you want. There is so much more that you could do with the Pendulum. Please refer to the documentation to learn more.


5. Selenium Selenium is a Python package to automate browser activity. It automates any web browsing activity to do what you like-from opening the browser, clicking the button, logging in, collecting data, and others. You could automate the activity similar to the activity you do in a web browser. Let's start by installing the package. To open a particular browser, Selenium required the driver to be downloaded. You could refer to the documentation for all the browser detail, but I would use chromedrive_autoinstaller package to automate the download of Chrome driver. For example, I want to open the Teepublic website and click the login button. To do that, I use the following code. There are still many things you could do with Selenium. Learn from the documentation to learn more. Also, refer to this article for an example of Selenium implementation.


Conclusion There are many popular data science Python packages that have been developed, and here are some of the packages that would help your data workflow: DaskFakerDulwichPendulumSelenium I hope it helps!


Twitter Sentiment Analysis: FIFA World Cup 2022

What is Sentiment Analysis? Sentiment analysis is a type of NLP that aims to label data according to its sentiments, such as positive, negative, and neutral. This analysis helps companies understand how their customers feel about their products or services or identify trends in public opinion about a particular topic. For example, a company like Audi can learn whether people like the colors of its new car by examining Twitter shares like the image below.


What is Social Media Scraping? With the developing technology, it is now much easier to express all kinds of emotions, feelings, and thoughts through social networking sites. Social media scraping is the process of extracting data from social media platforms. One of the most used social media platforms is Twitter. When we look at the statistics, Twitter has around 450 million monthly active users as of 2022 and 6,000 tweets are posted every second. Twitter is one of the most important data sources for data scientists. Data scientists can extract data from Twitter with web scraping tools. There are several libraries to do this, such as Tweepy and Snscrape. Let's take a look at the pros and cons of these libraries.


Tweepy Tweepy is an easy-to-use Python library that allows you to access the Twitter API. You can connect the Twitter API with this library and scrape the tweets you want. However, this library has some drawbacks. The standard version of the Twitter API only allows you to extract 3200 tweets in a timeline. There are limits to how many tweets you can collect from a user's account. You can find more information about these limits here.


Snscrape Another library you can use to collect data from Twitter is Snscrape. Snscrape is a data scraper for social networking services that helps you extract data such as user profiles, tweet contents, hashtags, or searches. It also does not require the use of a Twitter API. With Snscrape, you can also scrape data from other services like Facebook, Instagram, Reddit, and Telegram.


How to scrape with Snscrape? In this section, we'll look at a brief overview of Snscrape to orient ourselves on its structure/philosophy before diving into Twitter sentiment analysis. First, we're going to install this library with the following command: Next, let's retrieve 1000 tweets sent on the first day of the tournament from #WorldCup2022 and then convert this data into Pandas DataFrame. Note that you can use this notebook to follow this analysis. Voilà! We easily scraped tweets from Twitter with Snscrape using just a few parameters. Snscrape tweet objects have many parameters. You can see these parameters in the image below. It's time to perform a Twitter sentiment analysis to find out what football lovers think of FIFA World Cup 2022.


Twitter Sentiment Analysis with Hugging Face So far, we've covered what sentiment analysis and social media scraping are and then how to collect tweets from Twitter. In this section, In this, we'll walk you through how to perform a Twitter sentiment analysis. Before I show you this analysis, let me explain a few NLP concepts you need to know.


Transfer Learning To carry out the sentiment analysis, you can build a model from scratch. When analyzing big data, this approach takes a lot of time and money. When working with big data, I recommend using the transfer learning technique. Transfer learning is a machine learning approach where a model trained on one task is reused on another related task. Although transfer learning has become the standard approach in computer vision, this method was not suitable for NLP for many years as it required large amounts of labeled data. After 2017, with the development of models such as BERT and GPT3 based on transformers, the transfer learning technique has also been used for NLP. However, some problems have arisen in the use of these models, for example: Using different frameworks, such as PyTorch or TensorFlow, when coding the model architecture.Loading the pretrained models from a server.Fine-tuning the pretrained models. To tackle these problems painlessly, the Hugging Face framework was developed.


What is Hugging Face? Hugging Face is a data science platform and large open-source community that provides tools that help users build, train, and deploy machine learning models. It offers a standardized interface to a wide range of transformer models for adapting them to new usage tasks. You can also use this framework to work with three major deep-learning libraries (PyTorch, TensorFlow, and JAX).


What is Hugging Face Hub? Hugging Face Hub is an open-source platform where people can easily collaborate and create machine-learning models. For example, you can find over 762 publicly available sentiment analysis models in this hub. The model we'll use for this analysis is the cardiffnl/twitter-roberta-base-sentiment-latest model, which is a roBERTa-base. This model was trained on ~124 million tweets from January 2018 to December 2021. The easiest way to load this model is to use a pipeline. Pipelines automatically load the model you want and its preprocessing class. Let's load our model with the pipeline method: Awesome! Our model is ready to make predicts. If you want, you can also use different models for this analysis.


Running Twitter Sentiment Analysis Let's put together what we've talked about so far and run a Twitter sentiment analysis. To do this, let's first scrape tweets from the first day of the tournament in the hashtag #WorldCup2022 and then find the sentiment in each tweet with the for a loop. We now have the sentiments of the tweets. To find out what football lovers think of FIFA World Cup 2022, let's first convert the results into a DataFrame and then take a look at the first five rows of this DataFrame. As you can see, we have a Twitter dataset with some information like date, likes, source of tweets, tweets, and sentiments. You can find this dataset here. Let's go ahead and visualize all these labels with a pie chart.


Data Visualization After predicting the labels, it's a good idea to see the distribution of these labels. First, let's group the data according to the sentiment column and take a look at the sentiments: Next, let's draw a pie plot according to these groups. As you can see, the percentage of positive and neutral tweets is close to each other. The percentage of negative tweets is the lowest. It turned out that football fans think more positively about the 2022 FIFA World Cup.


Generating Word Cloud Now let's create a word cloud and see which words stand out for each sentiment with the worldcloud library. First, let's look at the word cloud of positive tweets. Some of the words associated with positive tweets contain FIFA, Canada, Black, Brazil, start, etc. Next, let's take a look at the word cloud of negative tweets. Some of the words associated with negative tweets: are rights, Qatar, BBC, opening, human, ceremony, etc.


Final Thoughts Social media are platforms where people freely share their thoughts. Twitter is one of the most used social media platforms. The data on this platform is gold for companies. Companies can make more accurate decisions using this data. In this blog post, I first showed you how to scrape data from Twitter with Snscrape. Next, I explained how to perform sentiment analysis with a pre-trained model in Hugging Face. We explored the thoughts of football fans about the FIFA World Cup 2022. The model we built predicted more positive labels. It turned out that football fans think more positively about the 2022 FIFA World Cup. That's it. Thanks for reading. I hope you enjoy it. You can find the notebook I used on this blog here. Don't forget to follow us on YouTube  Twitter  Instagram  TikTok


Resources Getting Started with Sentiment Analysis using PythonTwitter Sentiment AnalysisWeb Scraping with PythonNatural Language Processing with Transformers


Will ChatGPT Settle Chatbot War?

ChatGPT Journey ChatGPT is a chatbot developed by OpenAI. It uses GPT-3, a language model with 175 billion parameters, making it one of the largest in existence. GPT-3 was trained on vast text data and could generate human-like responses to natural language inputs. GPT-3 quickly gained popularity and was used in various applications, such as language translation, summarization, and content generation. However, the researchers at OpenAI wanted to push the boundaries of GPT-3 even further. They decided to create a variant of the model that could generate novel images from text descriptions. OpenAI named this new model DALL-E and the new version DALL-E 2, and it was trained on image-text data using a combination of convolutional and transformer layers. DALL-E 2 can generate unique and creative images that amaze people all over the world. Inspired by the success of GPT-3 and DALL-E 2, the researchers at OpenAI decided to create a chatbot that could generate responses to natural language inputs in a conversational context. They named this chatbot ChatGPT and trained it on conversation data using GPT-3 as its underlying language model. ChatGPT quickly became a hit and was used in various chatbot applications, such as customer support, virtual assistants, and social media bots. The AI journey from GPT-3 to DALL-E 2 and ChatGPT was a huge success, and the researchers at OpenAI continued to push the boundaries of AI even further.


Will ChatGPT Settle Chatbot War?

Chatbot War ChatGPT is not alone but a latecomer. There are many competitive chatbots. Dialogflow: Developed by Google, Dialogflow is a chatbot platform that uses AI and natural language processing to create chatbots for various applications. It offers a range of tools and capabilities, such as integrations with popular messaging platforms and support for multiple languages.Microsoft Bot Framework: Developed by Microsoft, the Microsoft Bot Framework is a set of tools and services that can be used to create and deploy chatbots. It offers natural language understanding, conversation management, and integration with various messaging platforms.IBM Watson Assistant: Developed by IBM, Watson Assistant is a chatbot platform that uses AI and natural language processing to understand and respond to user inquiries. It offers personalized responses, context-aware interactions, and integration with other AI technologies.Amazon Lex: Developed by Amazon, Amazon Lex is a chatbot platform that uses AI and natural language understanding to create chatbots for various applications. It offers automatic speech recognition, text-to-speech, and integrations with popular messaging platforms.DialoGPT: Developed by Microsoft, DialoGPT is a chatbot that uses GPT-3 as its underlying language model. It is trained on conversation data and can generate human-like responses to natural language inputs.Hugging Face: Developed by Hugging Face, Hugging Face is a library of natural language processing models that includes chatbots. It offers a range of chatbot models that can be customized and integrated with various applications and platforms.Other GPT-3 based ChatBots: Developed by other startups or enterprises, they are chatGPT-based like GialoGPT. These platforms are similar to ChatGPT in that they all provide chatbot capabilities and use natural language processing and machine learning to generate intelligent responses. However, ChatGPT has several unique features, such as its use of GPT-3 as its underlying language model and its focus on personalized and conversational responses, that differentiate it from its competitors.


Will ChatGPT Settle Chatbot War?

Chatbot Extensive Use Cases Extensive use cases demand the rise of chatbots. They can be used in a variety of applications, including: Customer support: Chatbots can provide 24/7 customer support to businesses and organizations. It can handle many customer inquiries and provide timely and relevant responses.Virtual assistants: Chatbots can be used to power virtual assistants, such as Siri and Alexa. It can understand and respond to natural language inputs and provide users with information, advice, and recommendations.Social media bots: Chatbots can create social media bots that can interact with users on platforms such as Twitter and Facebook. The bots can provide information, answer questions, and engage with users in real-time.E-commerce: Chatbots can be used in e-commerce applications to provide personalized product recommendations and answer customer questions about products, prices, and availability.Education: Chatbots can be used in educational applications to provide personalized learning experiences and answer student questions in real-time. It can also be used to evaluate student progress and provide feedback. There are many more. This can open up opportunities for different chatbots.


Will ChatGPT Settle Chatbot War?

How to Differentiate a Chatbot? To measure the performance of a chatbot model, the following metrics can be used: Perplexity: This measures the ability of the model to predict the next word in a conversation based on the previous words. A lower perplexity score indicates a better-performing model.BLEU score: This measures the model's ability to generate human-like responses by comparing the generated responses to a reference set of human-generated answers. A higher BLEU score indicates a better-performing model.Accuracy: This metric measures how accurately the model predicts the next word or sentence in a conversation. It can be calculated by comparing the predicted words or sentences with the actual ones in the conversation.Fluency: This metric evaluates how natural and smooth the generated responses are. It can be measured by having human evaluators rate the responses on a scale of 15, with 5 being the most fluent.Coherence: This metric measures how well the responses are connected and make sense within the conversation context. It can be measured by having human evaluators rate the responses on a scale of 15, with 5 being the most coherent.Consistency: This metric evaluates how consistent the responses are regarding tone, style, and content. It can be measured by comparing the responses to a set of pre-defined standards and scoring them accordingly.User satisfaction: This metric measures how well a chatbot meets the expectations and needs of the users. It can be measured through user surveys and feedback.


Will ChatGPT Settle Chatbot War?

What are Potential ChatGPT Business Models? ChatGPT is a newcomer, but we can expect some potential business models like others as follows: SaaS (Software as a Service): ChatGPT can be offered as a cloud-based service that businesses and organizations can use to power their chatbot applications. Customers can pay a monthly or annual subscription fee to access chatGPT's capabilities.Consulting: ChatGPT can provide consulting services to businesses and organizations. For example, ChatGPT can analyze customer conversations and provide insights and recommendations to improve the customer experience.Licensing: ChatGPT can be licensed to businesses and organizations that want to use its capabilities in their own chatbot applications. The licensing model can be based on the number of users, messages, or transactions processed by ChatGPT.Data monetization: ChatGPT generates a large amount of data, including conversation logs and customer feedback. This data can be monetized by selling it to third parties or using it to improve chatGPT's capabilities.Advertising: ChatGPT can display ads to users based on their interests and preferences. The revenue from the ads can be shared with the businesses and organizations that use ChatGPT in their chatbot applications.


Will ChatGPT Settle Chatbot War?

What's Next about ChatGPT? The future of ChatGPT is likely to involve continued improvements in its capabilities, such as better natural language understanding, more coherent and fluent responses, and increased personalization. It may also involve integrating other AI technologies, such as computer vision and natural language generation, to enable ChatGPT to handle a broader range of tasks and applications. Additionally, ChatGPT may be used in more specialized domains, such as healthcare, finance, and law, to provide expert-level advice and assistance in these fields. It may also be used in new applications, such as virtual events, telemedicine, and remote work, to support and enhance these emerging technologies. ChatGPT will likely continue to evolve and become an increasingly important and valuable tool for businesses and organizations.


Will ChatGPT Settle Chatbot War?

Can ChatGPT Unify All Chatbots? Based on the above analysis, it is unlikely that ChatGPT will settle the chatbot war, as there are many chatbots and chatbot platforms on the market, and each has its own strengths and weaknesses. ChatGPT is a highly advanced chatbot that uses GPT-3 as its underlying language model, but other chatbots have such capabilities. Other chatbots, such as DialoGPT and Hugging Face, also use GPT-3 and offer similar capabilities. Additionally, the chatbot war is not just about the capabilities of the chatbots themselves but also about the platforms and tools that support them. Different chatbot platforms, such as IBM Watson Assistant and Google Dialogflow, offer other features and capabilities, and various businesses and organizations use them. Therefore, it is unlikely that chatGPT will settle the chatbot war and unify all chatbots. Instead, it will likely continue to evolve and compete with other chatbots and chatbot platforms on the market.


How to access Scientific Knowledge with Galactica

What is Meta AI's Galactica? Galactica is a 120B parameter large language model trained on a curated scientific corpus. The training data consists not only of massive volumes of scientific literature but also datasets for downstream scientific NLP tasks and special tokens representing scientific phenomena. Specialized tokenization is an integral part of Galactica as it enables the model to predict citations or process modalities such as protein sequences or SMILES formulas: Citations: citations are wrapped with reference tokens [START_REF] and [END_REF].Reasoning: <work> token enables step-by-step reasoning by mimicking an internal working memory (not covered in this tutorial).SMILES Formula: SMILES formula sequences are wrapped with tokens [START_SMILES] and [END_SMILES]. For isomeric SMILES, tokens [START_I_SMILES] and [END_I_SMILES] are used.Amino Acid Sequences: amino acid sequences are wrapped with tokens [START_AMINO] and [END_AMINO].DNA Sequences: DNA sequences are wrapped with tokens [START_DNA] and [END_DNA]. Meta AI reports that Galactica's generative approach for citation prediction outperforms retrieval approaches, which demonstrates the potential for language models to replace search engines. Also, Galactica beats existing methods on reasoning task benchmarks (e.g., MMLU and MATH) and sets a new state-of-the-art on several downstream scientific NLP tasks (e.g., PubMedQA and MedMCOA). Despite its powerful capabilities, similar to most language models, Galactica is prone to hallucination; that is, the model outputs nonsensical results in some cases. Therefore, researchers who use Galactica should always fact-check the generated outputs.


How to use Galactica Galactica is accesible via the galai Python library. You can download the model with load_model. The name arguement is for the name of the model version to use. There are five versions of the model available ('mini', 'base', 'standard', 'large', 'huge'), each with varying parameter size (125M, 1.3B, 6.7B, 30B, 120B, respectively).The num_gpus arguement is for the number of GPUs to use. I was able to load the 'standard' version on two NVIDIA RTX 3090 GPUs; the model took up about 19GB memory for each device.


Text Generation Similar to most large language models, Galactica frames every NLP task as text generation. You can use generate to generate text. The input_text is the input context for the model to use for its generation. Galactica's other advanced capabilities can be accessed via prompt engineering of the input context.The max_length modifies the maximum token length of the generated text. Default value is 60 tokens, so max_length should be set to a higher value for longer generations. The maximum context length of the model is 2048 tokens.If new_doc is set to True, a padding token is automatically appended to the front of the input text so that the model would treat it as the beginning of a new document. For free-form text generation, new_doc it should be set to False.The top_p argument is for nucleus sampling. If the generated results seems too repetitive, set the value to a float between 1 and 0, such as 0.9. Otherwise top_p defaults to None and greedy decoding is used.


Papers and Surveys You can generate various types of academic literature with Galactica via prompt engineering. If a prompt is designed to resemble a certain kind of document, so will its completion. For paper documents, use Title:. This functionality is particularly useful when you need a comprehensive survey on a particular topic. Simply design the prompt as Title: TOPIC, A Survey and Galactica will automatically generate one for you.


Lecture Notes and Wikipedia Articles For Wikipedia-style articles or lecture notes, begin the prompt with #.


Citation Prediction Galactica is trained on a large scientific corpus comprising more than 360 million in-context citations and over 50 million unique references normalized across a diverse set of sources. This enables the model to suggest citations and help discover related papers. Citations are represented as [START_REF] TITLE, AUTHOR [END_REF].


Search To search for a paper that discusses a certain topic, make use of the following prompt - PAPER TOPIC [START_REF]. Since the [START_REF] token is appended to the end of the input context, Galactica would treat it as the beginning of a citation and complete the rest.


Prediction Instead of explicitly searching for a citation, you can also prompt Galactica to suggest a relevant paper within a document. The model has been trained on numerous academic texts that include representations of implicit citation graphs. Therefore, given a prompt such as TEXT [START_REF], Galactica can automatically suggest citations relevant to the TEXT.


Is generation Better than Retrieval? One might have doubts about relying on Galactica for citation prediction since large language models are known to produce contents that are far from factual. However, Meta AI reports that Galactica's generative approach outperforms tuned sparse and dense retrieval approaches at citation prediction tasks. Galactica is far from perfect, but the experiments indicate that the model yields better results than traditional search engines.


Downstream NLP Tasks You can use Galactica for conventional downstream NLP tasks, such as summarization, entity extraction, and question-answering. While general-purpose language models might struggle with scientific terminology or medical jargon, this is not the case with Galactica. For example, Meta AI reports that the model achieves a new state-of-the-art result on the PubMedQA and MedMCOA question-answering benchmarks; both are tasks that require a rigorous understanding of high-level biomedical concepts.


Summarization To yield a summary, simply append TLDR: at the end of the document.


Entity Extraction You can extract entities from documents in a question-answering format. Design the prompt as follows - TEXT\n\nQ:What scientific entities are mentioned in the abstract above?\n\nA:. Depending on the document's topic, you can replace scientific entities with a more domain-specific term, such as biomedical entities.


Question Answering In the paper, the authors prefix questions with Q: or Question:. A typical format is Question: QUERY\n\nAnswer:.


Multi-modal Tasks As mentioned above, Galactica can process modalities other than unstructured text, such as LaTeX equations, code, SMILES formula, DNA sequences, and amino acid sequences. The model uses task-specific tokens to support various forms of scientific knowledge. This design enables users to tackle multi-modal tasks that involve interactions between natural language and representations of scientific phenomena.


Math Mathematical equations are represented as LaTeX. Galactica wraps LaTeX equations with brackets \[ EQUATION \], so to generate mathematical descriptions, make sure to end your prompt with a \[. Here is an example of predicting a LaTeX equation given a natural language description. If converting natural language to math is possible, so is vice versa. Galactica can fluently translate between modalities, including math-to-natural language. This functionality would especially come in handy when encountering an indecipherable mathematical formula in a research paper. Interestingly, Galactica has somehow learned to convert mathematical equations into code.


Code Here is a demonstration of translating a code snippet into a natural language description and a mathematical description.


Chemical Understanding Galactica can aid the organization of chemical information by providing an interface to SMILES data. SMILES formulas represent chemical structure as a sequence of characters and appear alongside natural language descriptions or IUPAC names (method of naming organic compounds) within the Galactica training corpus. This indicates that the model might have learned to predict IUPAC names given a SMILES formula input. For IUPAC Name prediction, design a prompt similar to a PubChem document - [START_I_SMILES] SMILES_FORMULA [END_I_SMILES]\n\n## Chemical and Physical Properties\n\nThe following are chemical properties for.


Biological Understanding Galactica's capability to process biological modalities (i.e., DNA and amino acid sequences) could potentially play a role in aiding the organization of biomedical information. For instance, the model can annotate protein sequences with functional keywords. The authors claim that the model learned to match sequences with similar ones it has seen in training, and can use this to predict the functional keywords. Prompts for protein annotation can be designed as follows - [START_AMINO] AMINO_ACID_SEQUENCE [END_AMINO]\n\n## Keywords.


Conclusion This tutorial provides an overview of how Meta AI's Galactica enables users to access scientific knowledge and leverage this knowledge to tackle a variety of scientific NLP tasks. These tasks range from generating academic literature to processing multi-modal data, and all of these tasks could potentially play a role in scientific discovery. Although the model is far from perfect, the experiments show that the model's predictions can often outperform traditional methods. The model's impressive performance and ease of use are worthy of further study and exploration. In the future, I hope we can continue exploring how large language models like these can aid scientific research and the process of science itself.


The Art Of Negotiation: CICERO AI

HOW is CICERO built? The language model supporting CICERO has been trained on 2.7B parameters. The data used for this model is the text from the internet on 40,000 games played on webDiplomacy.net. The dialogue generated using this model is called "controllable dialogue". The plan for actions to interpret and plan the next negotiation by CICERO happens in 4 distinct steps: Forming initial predictionrefining the prediction and intentComing up with candidate messages as repliesFilter replies to identify the one which has the most "Value" in the game OpenAI Five-like AI models were generated using self-play reinforcement learning (RL), where the game chooses random steps to propagate, ending with outcomes (either positive or negative). At each outcome, the model learns to correct itself to get the positive outcome over the negative, making it a highly iterative process where the game keeps learning better ways than itself to achieve the most positive outcome.


The Art Of Negotiation: CICERO AI

piKL model The approach that CICERO uses is a hybrid approach between using purely supervised learning and unsupervised learning. The algorithm that Meta has developed is called piKL. At each step, CICERO predicts everyone's policy based on the dialogue it has shared with other players. Then CICERO chooses a new policy that has the highest chance of winning the intent during that step, given predicted original policies. This is similar to a Baysian approach however, the dependency in the equation is predicted by the AI itself. First, by predicting the current policy of opposing players based on the current message to the opposing players and then utilizing those policy predictions to inform its own policies, CICERO avoids being tricked by very straightforward "lies" that result in a loss.


The Art Of Negotiation: CICERO AI

Generating the dialogue The first task is generating a dialogue with other players in a way that can elicit a response to inform policy predictions is a critical task. By using a controlled vocabulary, CICERO is able to communicate clearly and strategize with other players.


The Art Of Negotiation: CICERO AI

What does it mean for human-AI interaction? Meta mentions that "The emergence of goal-oriented dialogue systems in a game that involves both cooperation and competition raises important social and technical challenges in aligning AI with human intentions and objectives." Meta believes that they have made significant headway in this work to align language models with intentions. A part of this headway is to identify and prioritize intentions. Meta would like to keep this model open-sourced to allow other researchers to continue taking advantage of what they have achieved.


The Art Of Negotiation: CICERO AI

Closing thoughts: Trust is earned over time Like many other articles that I have written so far, the progress in AI is impressive. Being able to generate text that can successfully have a semi-natural dialogue to achieve what the AI is programmed to do is an impressive technical achievement. However and unfortunately, I had to use the headline "Trust is earned over time" for this section of closing thoughts. Currently, we are living in the time of the largest tech gap between generations. Boomers and Gen X did not grow up in a world that involved tech complexity. Expecting them to be able to identify and outsmart an AI/fake persona specifically made to negotiate to defeat you is unrealistic. Even Millennials and the generations born with tech gen-Z have suffered from unregulated and profit-driven social media proliferation. Meta hopes that other researchers can build on its code "in a responsible manner" but can we really expect it? We need to invest in abilities to be able to identify and assess the social impact. This technology will likely be used to manipulate humans. I strongly believe that without appropriate guardrails, this technology will be misused.


A Guide to NeurIPS 202210 Topics and 50 Papers You Shouldn't Miss

1. Language Models and Prompting Arguably the most sought-after research area in AI - especially since the introduction of GPT-3 in 2020 - gets massive amounts of attention at the conference. With many blockbuster publications from the likes of Google, DeepMind, OpenAI, Meta, Stanford... all the big compute guys. Expect to find a lot of work on "simple prompting" techniques like Chain of Thought techniques (or tricks?) that squeeze more performance out of plain pretrained autoregressive LMs. Multimodality also gets a spotlight this year, DeepMind's Flamingo (vision + language) being the most popular of the bunch, Google's Minerva shows how good LMs can be at math when pretrained with the right data, and InstructGPT showcases how human feedback and Reinforcement Learning can be used to fine-tune large LMs. 1⃣ Chain of Thought Prompting Elicits Reasoning in Large Language Models  OpenReview   Virtual Poster   More like this paper  Simply prompting LMs to output reasoning steps instead of direct answers drastically improves performance. See also follow-up works like STaR. 2⃣ Flamingo: a Visual Language Model for Few-Shot Learning  OpenReview   Virtual Poster   More like this paper  DeepMind introduces a "simple" single model pretrained on vision + language, sets state of the art on various multimodal tasks 3⃣ Solving Quantitative Reasoning Problems with Language Models (Minerva)  OpenReview   Virtual Poster   More like this paper  A large LM trained on mathematical data can achieve strong performance on quantitative reasoning tasks, including state-of-the-art performance on the MATH dataset. 4⃣ Data Distributional Properties Drive Emergent In-Context Learning in Transformers  OpenReview   Virtual Poster   More like this paper  What aspects of large-scale pre-training drive in-context learning? Training data distribution needs burstiness and a large number of rarely occurring cases. 5⃣ Training language models to follow instructions with human feedback (InstructGPT)  OpenReview   Virtual Poster   More like this paper  OpenAI uses Reinforcement Learning from Humans in the Loop (RLHF) to fine-tune GPT-3 using data collected from human labelers. The resulting model, called InstructGPT, outperforms GPT-3 on a range of NLP tasks.


2. Diffusion Models Granted, if anything deserves the coolest-kid-in-town crown in 2022, it has to be text-to-image generation models, most of them powered by Diffusion Models: OpenAI's DALL·E 2, Google's Imagen, or Stable diffusion. As an early sign of maturity - and barely 2 years since they became popular! - the modeling technique has now spread beyond the realm of 2D still image generation and is being applied to 3D scene synthesis, video generation, and molecular docking, among others. As we saw with Transformers in 2017, the time it takes for a research idea to become mainstream just keeps shortening. 1⃣ Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (Imagen)  OpenReview   Virtual Poster   More papers like this  Imagen, a simple approach to text-to-image synthesis using diffusion models. 2⃣ Object Scene Representation Transformer (OSRT)  OpenReview   Virtual Poster   More papers like this  A highly efficient 3D-centric model in which individual object representations naturally emerge through novel view synthesis. 3⃣ Denoising Diffusion Restoration Models (DDRM)  OpenReview   Virtual Poster   More papers like this  Super-resolution, deblurring, inpainting, and colorization using pre-trained Denoising Diffusion Probabilistic Models (DDPMs) without problem-specific supervised training. 4⃣ Flexible Diffusion Modeling of Long Videos  OpenReview   Virtual Poster   More papers like this  DDPMs applied to the video domain. To capture long-range dependencies between frames, they present an architecture that can be flexibly conditioned on any subset of video frames. 5⃣ EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equations  OpenReview   Virtual Poster   More papers like this  Energy-guided stochastic differential equations (EGSDE) that employ an energy function pretrained on both the source and target domains to guide the inference process of pretrained SDE for realistic and faithful unpaired Image-to-image (I2I).


3. Self-Supervised Learning Self-Supervised Learning (SSL) has become such an essential ingredient of modern ML that it's now pretty much baked into most research in one way or another. For the first time in Deep Learning, NLP led the way with BERT in 2018, and Computer Vision later joined the SSL bandwagon with successful techniques like SimCLR. This region of our semantic map has a heavy representation of novel SSL techniques for Computer Vision: learning from unsupervised videos, the effects of data augmentation on images... But arguably, the cherry on top is DeepMind's Chinchilla: a study on how much of a Language Model's pretraining budget should be spent on model parameters and how much in a larger training corpus (finding that most large LMs are too big or undertrained), resulting in Chinchilla, a 70B parameter LM that outperforms its bigger counterparts by training for longer. Finally, we also couldn't miss out on wild new (partly) SSL techniques for Information Retrieval, such as the Differentiable Search Index. While SSL is now so commonplace it's often relegated to a disinterested footnote. All this research is proof that there are many stones unturned in this area with new insights to discover. 1⃣ An empirical analysis of compute-optimal large language model training (Chinchilla)  OpenReview   Virtual Poster   More papers like this  It's better to train a smaller Language Model on more tokens. DeepMind showcases this with their 70B Chinchilla model outperforming bigger models such as Gopher (280B), GPT-3 (175B) or Megatron-Turing NLG (530B). 2⃣ VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training  OpenReview   Virtual Poster   More papers like this  Pretraining video representations on video done exquisitely. 3 takeaways: high masking ratio is best, technique works well even on small datasets, and quality > quantity when it comes to Self-Supervised Video Petraining. 3⃣ Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP  OpenReview   Virtual Poster   More papers like this  A systematic study of the interactions between the pretraining data sources for CLIP. Surprisingly (?) mixing multiple data sources does not necessarily yield better models, which is corroborated by our theoretical analysis of toy models. 4⃣ A Data-Augmentation Is Worth A Thousand Samples: Analytical Moments And Sampling-Free Training  OpenReview   Virtual Poster   More papers like this  An analytical study of data augmentation (DA) and how it impacts the parameters of a model. E.g., given a loss at hand, common DAs require tens of thousands of samples for the loss to be correctly estimated and for the model training to converge. 5⃣ Transformer Memory as a Differentiable Search Index  OpenReview   Virtual Poster   More papers like this  A single Transformer is trained to directly output document identifiers autoregressive given a query as a prompt. Followup work is also presented at NeurIPS, such as the A Neural Corpus Indexer for Document Retrieval.


4. Graph Neural Networks Equivariances, 3D molecule generation, Partial Differential Equations... Graph Neural Networks (GNNs) have been around for a while now, and while they haven't achieved celebrity-level popularity like Transformers or Diffusion Models, they've steadily grown their pie in the last couple of years, extending into applications such as drug design, differential equation solving or reasoning. This heterogeneous landscape makes sense because, as outlined in the Geometric Deep Learning blueprint, in a way, GNNs is a new abstraction on NNs to unlock thinking from first principles about how to cast arbitrary problems into the right architectures, escaping the curse of dimensionality by leveraging symmetries and invariances. For example, this is key for finding the right representations to computationally solve Partial Differential Equations or predicting the shapes of organic molecules to design new drugs more efficiently. 1⃣ Zero-Shot 3D Drug Design by Sketching and Generating (DESERT)  OpenReview   Virtual Poster   More papers like this  A zero-shot drug design method powered by pre-training techniques. Existing deep-learning-based methods for drug design often rely on scarce experimental data or slow docking simulation. DESERT splits the design process into sketching and generation phases, speeding up generation while preserving high accuracy. 2⃣ Torsional Diffusion for Molecular Conformer Generation  OpenReview   Virtual Poster   More papers like this  Drug-like Molecule Conformer Generation via diffusion process on torsion angles the Fast and accurate conformer generation via diffusion modeling on the hypertorus and an extrinsic-to-intrinsic score model, while being orders of magnitude faster than previous diffusion-based approaches. 3⃣ MAgNet: Mesh Agnostic Neural PDE Solver  OpenReview   Virtual Poster   More papers like this  A novel mesh-agnostic architecture that predicts solutions to PDE at any spatially continuous point of the PDE domain and generalizes across different meshes and resolutions. 4⃣ MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields  OpenReview   Virtual Poster   More papers like this  Message Passing NNs (MPNNs) is a powerful way to model interatomic potentials, but they're inefficient. MACE introduces higher-order message passing in a highly parallel manner achieving SOTA in various benchmarks. 5⃣ Few-shot Relational Reasoning via Connection Subgraph Pretraining (CSR)  OpenReview   Virtual Poster   More papers like this  CSR can make predictions for the target few-shot task directly by self-supervised pre-training over knowledge graphs.


5. Reinforcement Learning Making agents more efficient learners is a key question RL researchers are still grappling with, and this year's NeurIPS contains many proposals on how to achieve it. For instance, using offline learning and imitation learning at scale to overcome the initial inefficient exploration phase, improving credit assignment techniques to better navigate sparse rewards landscapes, or using pretrained Language Models to bootstrap policies with human priors. Other points of interest often revolve around robustness and reproducibility, which are closely related to efficiency in challenging open-ended settings. Finally, RL also sees successful application in areas like chip design with quite a few papers on the topic (seriously, check out the "More like this!"). 1⃣ Using natural language and program abstractions to instill human inductive biases in machines  OpenReview   Virtual Poster   More papers like this  Meta-learning agents can learn human inductive biases through co-training with representations from language descriptions and program induction. 2⃣ MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge  OpenReview   Virtual Poster   More papers like this  Leveraging large pretrained models to automatically label videos with actions to create large-scale datasets for offline learning, just with video data from Minecraft. 3⃣ MaskPlace: Fast Chip Placement via Reinforced Visual Representation Learning  OpenReview   Virtual Poster   More papers like this  RL agents that learn to distribute components on a silicon chip design better than humans. 4⃣ Spending Thinking Time Wisely: Accelerating MCTS with Virtual Expansions  OpenReview   Virtual Poster   More papers like this  MCTS made more efficient by allocating a higher compute budget on harder states. 5⃣ Trajectory balance: Improved credit assignment in GFlowNets  OpenReview   Virtual Poster   More papers like this  A new training objective for generative flow networks that tackles the problem of credit assignment (what action within a trajectory is most responsible for a final reward?) leads to faster convergence and better fitting to a target distribution.


6. Brain-Inspired Let's not forget NeurIPS is also home for neuroscience adjacent literature; after all, our brains are the OG neural information processing system that sparked much of the modern artificial NNs. This diverse area involves a lot of learning from brain imaging techniques such as fMRI, alternatives to backprop for learning that bode better with what we know about neurons, spiking Neural Networks, and more! 1⃣ Learning on Arbitrary Graph Topologies via Predictive Coding  OpenReview   Virtual Poster   More papers like this Backprop does not allow training on networks with cyclic or backward connections, which are hypothesized to be essential in brain-like computation. They show how predictive coding (PC), a theory of information processing in the cortex, can be used to perform inference and learning on arbitrary graph topologies. 2⃣ Theoretically Provable Spiking Neural Networks  OpenReview   Virtual Poster   More papers like this  A theoretical investigation on the approximation power and computational efficiency of spiking neural networks with self-connections. 3⃣ Self-Supervised Learning of Brain Dynamics from Broad Neuroimaging Data  OpenReview   Virtual Poster   More papers like this  Novel self-supervised learning techniques for neuroimaging data inspired by prominent learning frameworks in natural language processing, using one of the broadest neuroimaging datasets used for pre-training to date. 4⃣ On the Stability and Scalability of Node Perturbation Learning  OpenReview   Virtual Poster   More papers like this  Node perturbation is scalable against overparameterization but unstable in the presence of a model mismatch. 5⃣ An Analytical Theory of Curriculum Learning in Teacher-Student Networks  OpenReview   Virtual Poster   More papers like this  A solvable model of curriculum learning and comment on the implications for the ML and the experimental psychology literature.


7. Out-of-Domain Generalization Out-of-Domain generalization and closely related, causality, are topics of high interest in academic research. While standalone OOD research hasn't fully broken into the mainstream, there's an undeniable trend broadly in Machine Learning benchmarking culture towards caring more and more about robustly generalizing in challenging conditions such as zero/few-shot or under heavy data distribution shifts because static in-domain evaluations have been cracked one after the other at a record-breaking pace. The area - as most in the early stages - still suffers from a lack of standardization, which is why we're highlighting a couple of publications on the topic (2, 3). In addition, we're highlighting a simple ensemble technique to domain generalization (1), tabular embeddings (4), and sparsely connected factor graphs at scale for causal discovery (5), which represents progress in a research direction we've highlighted before: the use of sparsity + communication bottlenecks to constrain the model into learning models of the world that capture its robust causal structure. 1⃣ Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization  OpenReview   Virtual Poster   More papers like this  A simple hyper-parameter-free strategy of using the simple moving average of model parameters during training and ensembling achieves SOTA on domain generalization benchmarks and can be explained using the Bias-Variance trade-off. 2⃣ Assaying Out-Of-Distribution Generalization in Transfer Learning  OpenReview   Virtual Poster   More papers like this  A large-scale empirical study of out-of-distribution generalization. 3⃣ Is a Modular Architecture Enough?  OpenReview   Virtual Poster   More papers like this  Metrics to study a broad range of mixture-of-experts styled modular systems. Such systems suffer from problems of collapse and specialization and might require additional inductive biases to overcome this sub-optimality. 4⃣ On Embeddings for Numerical Features in Tabular Deep Learning  OpenReview   Virtual Poster   More papers like this  Representing numerical features with vectors instead of scalar values can significantly boost DL models for tabular data. 5⃣ Large-Scale Differentiable Causal Discovery of Factor Graphs  OpenReview   Virtual Poster   More papers like this  Using factor graphs for large-scale causal discovery learning with interventional data.


8. Learning Theory Okay, let's start by admitting that many of us kinda find the heavy-duty mathy stuff off-putting. Still, there's so much to learn from it at NeurIPS. For instance, large batch sizes have proven to be essential to successfully learn representations with contrastive learning. This makes sense intuitively, but is there a more grounded statistical explanation? (1) Or what guarantees can we expect when using autodiff blindly over functions that aren't smooth enough? (3) Or under which conditions is performing well in Out-of-Distribution even possible? (2). Check out the papers below if you're intrigued by these questions. 1⃣ Why do We Need Large Batch sizes in Contrastive Learning? A Gradient-Bias Perspective  OpenReview   Virtual Poster   More papers like this  A Bayesian data augmentation method to disentangle negative samples for gradient bias mitigation in contrastive learning. 2⃣ Is Out-of-Distribution Detection Learnable?  OpenReview   Virtual Poster   More papers like this  Through the lens of Probably Approximately Correct (PAC) learning theory, this work studies the generalization of OOD detection: classifying whether a sample falls within the training distribution or not. They find this is impossible under some conditions and prove formal theorems around it, but these conditions are mostly not a concern in real-world problems. 3⃣ Automatic differentiation of nonsmooth iterative algorithms  OpenReview   Virtual Poster   More papers like this  What happens when you apply autodiff to functions that aren't smooth enough? It's mostly fine. They converge to classical derivatives anyway. 4⃣ Efficient and Modular Implicit Differentiation  OpenReview   Virtual Poster   More papers like this  Autodiff but in the implicit form (i.e. when you can't isolate f(x) in the left side of the equal sign). Implemented in JAX. 5⃣ Gradient Descent: The Ultimate Optimizer  OpenReview   Virtual Poster   More papers like this  Using gradient descent to tune not only hyperparameters but also hyper-hyperparameters, and so on... This work shows how to automatically compute hypergradients with a simple and elegant modification to backpropagation.


9. Adversarial Robustness, Federated Learning, Compression Adversarial Robustness in ML has been a thing for longer than I can remember, and this year it's no different (3, 5). This is understandable as this is a concern of high importance when it comes to using models in safety-critical applications such as self-driving. On the other hand, Federated Learning, sometimes married with Differential Privacy is another prolific area of research that still needs to find its use in the mainstream real-world applications (2). A camp that decidedly already has widespread usage in the real-world, compression, pruning, and other speed/efficiency enhancing techniques (4). Finally, a new twist on pruning that looks at pruning data, not weights, to train NNs faster than scaling laws would allow for! (1) 1⃣ Beyond neural scaling laws: beating power law scaling via data pruning  OpenReview   Virtual Poster   More papers like this  In theory and practice, power-law scaling of error with respect to dataset size can be improved via intelligent data pruning. 2⃣ Self-Aware Personalized Federated Learning  OpenReview   Virtual Poster   More papers like this  We propose a new adaptive federated learning algorithm for personalization. 3⃣ Increasing Confidence in Adversarial Robustness Evaluations  OpenReview   Virtual Poster   More papers like this  A test that enables researchers to find flawed adversarial robustness evaluations. Passing this test produces compelling evidence that the attacks used have sufficient power to evaluate the model's robustness. 4⃣ On-Device Training Under 256KB Memory  OpenReview   Virtual Poster   More papers like this  A framework for on-device training on tiny IoT devices, even under a limited memory budget of 256KB. 5⃣ Pre-trained Adversarial Perturbations  OpenReview   Virtual Poster   More papers like this  A novel algorithm to generate adversarial samples using pre-trained models which can fool the corresponding fine-tuned ones and thus reveal the safety problem of fine-tuning pre-trained models to do downstream tasks.


10. Datasets & Benchmarks Last but not least, the unsung heroes of ML. As the pace of progress keeps accelerating and even modern datasets saturate faster than most predicted [link to palm and instruction fine-tuning], new datasets that measure interesting phenomena must emerge to substitute them. Like last year, NeurIPS had a special track for datasets and benchmark papers (see the full list of them here). Many of the proposed benchmarks are incredibly interesting and useful, so the list below is guaranteed to miss some of those. Still, here it goes 1⃣ LAION-5B: An open large-scale dataset for training next generation image-text models  OpenReview   Virtual Poster   More papers like this  An open, publically available dataset of 5.8B image-text pairs and validate it by reproducing results of training state-of-the-art CLIP models of different scales. 2⃣ DC-BENCH: Dataset Condensation Benchmark  OpenReview   Virtual Poster   More papers like this  Dataset Condensation aims to learn a tiny dataset that captures the rich information encoded in the original dataset. Comparing condensation remains a challenge, which is why this benchmark is proposed. 3⃣ NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning  OpenReview   Virtual Poster   More papers like this  The Near real-world offline RL benchmark (NeoRL) is a benchmark with emphasis on the complete pipeline for deploying offline RL in real-world applications that aims to bridge the performance gap between offline evaluation and online deployed performance. 4⃣ A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks  OpenReview   Virtual Poster   More papers like this  Injecting a backdoor during the training phase can be a powerful way for adversaries to control NLP systems such as Language Models. This work presents an open-source toolkit OpenBackdoor that enables rigorous evaluation of how vulnerable models are to these kinds of attacks. 5⃣ PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding  OpenReview   Virtual Poster   More papers like this  A comprehensive and multi-task benchmark for protein sequence understanding, which studies both single-task and multi-task learning.


Working on a Computer Vision Project? These Code Chunks Will Help You !!!

1. Know your dataset's instances for the object detection or segmentation project, we annotate our dataset with the help of external annotation tools like makesense.ai, VGG annotator, LableIMG, etc. We know the exact number of images, but it's hard to know how many instances of each class we have. Knowing instances of the class will tell you if your dataset is imbalanced or not. It will have a deep impact on the learning model if your instances are not balanced. So after downloading annotated dataset and its annotation file, you can use the following chunk of code to see the class balance status. You can see, at last, the counter gives instance values for each class, and then on your model criteria, you can decide if the dataset needs further balancing or not.


2. Preprocessing of images In our image dataset, other than class instances, we have many other objects/things. If we take it for the learning purpose of the model, then these other items can be classified as noises. There are many use cases that claim that removing these noises and then sending them to the model for training improves the performance of the model. So how do preprocess images? See the below code. If you run the above code, then you will have your training image in front of you, and your mouse will act as a mask maker. After clicking and hovering the mouse on an unnecessary object will direct create a mask on that object. I took white color for use case purposes, but you can take any according to your problem. You can train a separate object detection model for noise, and below that, you can attach this code. At first, the model will detect noise, and then this code will mask that bounding box with your desired color. There are many things you can do for image preprocessing, like cropping, making blur/contrast, etc. you can read my blog for more image preprocessing techniques.


3. Data Augmentation In every computer vision project, you want to augment the dataset to make it bigger to make the model's work easier. There is much open-source software that does Augmentations for you, like Roboflow. But many times, there can be a problem with data security and confidentiality. So you can do your own dataset augmentation on your python editor. There is a library by TensorFlow known as "ImageDataGenerator" which helps you to do this. See the below code. The first chunk of code is for the folder of images. you can do mass augmentation from that. The second chunk is for single images. You can use any of the above according to your use case. The last "i" is the number of synthetic images you want to create. Choose an appropriate number and Augment it.


4. Dataset creation Many times, you require images from a webcam. but it's hard to click it and save it in the labeled folder for classification or object detection. It also involves a lot of manual tasks. below code is to click images for particular labels, and it directly will store them at the proper location. You will get the below result after running the code chunk. and images will get stored at the specified location.


5. Extracting areas from the image This is the most useful thing not just to detect or segment objects but to extract their areas. We use many techniques like pixel measurement and others. But the thing is, you have to do calibration before extracting areas to match the original dimensions and their representations in the image and their ratios. So for calibration, people use inbuilt ratios and reference object schemes, but I tried a new way of calculating the calibration factor. You have to draw just a line to do the calibration. I mentioned how to do that in the below blog.


If you have found this article insightful If you found this article insightful, follow me on Linkedin and medium. you can also subscribe to get notified when I publish articles. Let's create a community! Thanks for your support!


If you want to support me : As Your following and clapping is the most important thing, but you can also support me by buying coffee. COFFEE.


You can read my other blogs related to : Signing off, Chinmay Bhalerao


I used AI to Generate Nietzschean Aphorisms

God is Dead - Nietzsche, the Herald of Nihilism According to the 19th-century German philosopher Friedrich Nietzsche, best known for heralding the death of God, modern civilization is plagued by nihilism; the loss of all value structure and meaning in life. When Nietzsche proclaimed "God is dead", he wasn't merely observing the decline of religion but was also pointing out that the universal values which have been the cornerstone of Western civilization for centuries no longer hold any meaning, thus each individual is left to fend for themselves, trying to make sense of a world that seems increasingly chaotic and full of suffering. Although the age of Enlightenment and the scientific revolution played a significant part in ushering the loss of faith, Nietzsche mainly blames the predominant worldview of the time, i.e., Platonic idealism, Christianity, and Utopianism, for the spiritual weakening of modern man. For Nietzsche, these philosophies are a form of escapism, as they are based on the belief in a perfect, idealistic world (e.g., Platonic Forms, Paradise, communist utopia, etc.) which is separate from and superior to the material world, in which we often feel our lives are cruel, harsh and painfully short. Their function is to provide grand narratives of the "true world" and give hope that there is something more to life than our earthly existence, but such a view is ultimately decadent, as it provides false hope and encourages people to withdraw from life and the real world. When these idealistic visions of reality are shown to be false (as they inevitably are), then the entire value structure that they support collapses, leading to a spiritual crisis and a loss of meaning in life. This is what Nietzsche called nihilism, and he saw it as the greatest threat to Western civilization. Nietzsche's warnings about the dangers of nihilism have proven to be prophetic, as the 20th century was marked by two world wars and the horrific genocide of millions of people, which were primarily fueled by fascism, totalitarianism, and communism; ideologies that aim to bring an end to history by creating a utopia, be it racial, national or class-based. According to philosopher Julian Young, although most of them rejected metaphysical ideas, they are nevertheless "true world" philosophies in disguise, in which the old distinction between physical and metaphysical is reinterpreted as a distinction between present and future. The modern man, after the death of God and consequent disasters, is left with a feeling of absurdity and pointlessness, and the need for a new grand narrative, or better yet, overcoming the dependence on such narratives altogether, arises.


Revaluation of All Values - Generating Aphorisms with a LLM How can man accomplish the task of overcoming nihilism? Nietzsche proposes that we must first carefully examine the ethical norms we take for granted, a process he calls the revaluation of all values. This process exposes how these "true world" grand narratives promote decadence, so that man can create values of his own without relying on such life-denying philosophies. For Nietzsche, the method of revaluation was to write aphorisms - brief, pithy, highly condensed statements that identify the genealogy and psychological motivations behind systems of values and their futile attempts to cover up the existential uncertainties of the human condition. Recently, I've been reading a lot of Nietzsche - I was fascinated by his style of writing in aphorisms and how he used this literary form to uncover truths that modernity would rather keep hidden. I thought it was a shame that, ever since Nietzsche suffered a severe mental breakdown at the age of forty-four, he couldn't produce remarkable works again. This had me thinking: what if we could summon Nietzsche from the abyss of madness and resurrect him via AI technology? Specifically, what if we could use LLMs to generate previously unheard-of Nietzschean aphorisms? NLP enthusiasts have shown that it is possible to get OpenAI's GPT-3 to generate works of poetry in the style of a specific poet by providing only a handful of examples. This approach is known as few-shot learning, where a LLM is able to learn to perform a certain task from a small amount of data (a.k.a. few-shot samples) during inference time. These few-shot samples are included in the input text (a.k.a. prompt), usually in the form of a numbered list. The LLM, a model which is primarily trained for natural language completion tasks, will then generate text that conforms to the format and style of the provided samples. I thought there would be no reason why the same approach couldn't be used to generate Nietzschean aphorisms. The main idea for Nietzschean aphorism generation is as follows: I take a bunch of Nietzsche's aphorisms, format them in a specific way, and then feed them into a LLM. The model will then "learn" the style and content of Nietzsche's aphorisms and generate new, never-before-seen aphorisms that are indistinguishable from Nietzsche's (at least in principle). Hopefully, these aphorisms will be able to provide stark insights into the human condition.


Human, all too Human - Tutorial for Few-shot Learning of Nietzschean Aphorisms I used Nietzsche's <Human, all too Human> as a source for few-shot samples since it takes the form of a collection of aphorisms. The full English-translated text is available at Project Gutenberg, which is a great resource consisting of numerous classical works of literature. Since the text file isn't formatted in a way that is suitable for NLP operations, a preprocessing step was necessary. After parsing the text file, I yielded two lists; one for the aphorisms themselves and the other for the corresponding section titles. The section titles will be used as aphorism topics later on. The following is the code for converting the entire text file into lists of aphorisms and their topics. The generation process is two-fold. First, the LLM generates a new aphorism topic given a list of randomly selected topic examples. Second, the LLM generates a new aphorism given a prompt consisting of randomly selected topic-aphorism pairs and the topic generated in the previous step. Therefore, two prompts are required: the topic prompt and the aphorism prompt. The following is the code for prompt engineering. In this project, I used the open-source 176B parameter LLM BLOOM, which is freely accessible via the Huggingface Inference API. Note that any sufficiently powerful LLM (e.g., GPT-3, PaLM, etc.) is able to perform few-shot learning. As mentioned above, the generation process is two-fold - first, we yield the topic, then the aphorism. The following is the code for generating Nietzschean aphorisms with BLOOM.


Results The model yielded impressive results. It was able to successfully generate aphorisms that conform to the style and topics of Nietzsche's writing in a few minutes. Although the generated aphorisms were by no means perfect, I believe that they succeeded in portraying "Nietzschean" characteristics. Here is an example of a prompt and its completion (text in bold): More aphorisms are presented in the appendix. There were a few limitations in the project, the biggest one being that I had to use only three few-shot samples due to BLOOM's context length constraints. I'm hoping to improve upon the current framework by fine-tuning a smaller language model rather than relying on BLOOM's few-shot learning capabilities.


This AI newsletter is all you need #22

What happened this week in AI by Louis One word: Galactica. Galactica, Meta's most recent large language model that can store, combine and reason about scientific knowledge was shut down after many users reported results that were misleading or incorrect. There's a lot of controversy going on around this model, mostly to do with the gap between Meta's confidence over the model and its rightfully questionable results. The demo was not as catastrophic as Microsoft's Tay incident of 2016, but it too quickly found the line between fun experimental tool and dangerous propagator of misinformation. Galactica represents a big advancement for large language models, but given that it was intended for scientific use, the level of rigor was far from met. On my end, I really liked a tweet shared by my friend Lior, which greatly summarizes my thoughts. I'd like to quote here: "The drama surrounding Galactica baffles me. Let's remember we're all on the same team trying to make our tiny field progress." Was Galactica perfect? No. But GPT3, StableDiffusion, and Dall-E weren't either. It's by releasing it into the world that the feedback loop starts, and these insights help us build better tools over time. To add the ethical perspective from Lauren, let's not forget what effects this might leave on the world and our responsibility as AI co-creators to handle those effects, whether they range from negative to positive. This is neither the first nor the last language model to accidentally spread falsehoods, but understanding and learning from these mistakes ensures that the progress we work toward in AI forges the future we want.


This AI newsletter is all you need #22

Hottest News Achieving Individual - and Organizational - Value With AI: A reportThe report has many interesting findings and suggests that employees tend to underestimate how much they use AI technologies at work. Some key findings are that a majority of individual workers personally obtain value from AI and regard AI as a coworker, not a job threat. Requiring individuals to use AI encourages its use more than building trust in AI does, and mandatory use, despite seeming oppressive, still leads to individual value. Organizations get value when individuals get value, not at the expense of individual value.Design app Canva released a beta version of its own text-to-image generatorYes, another one! I actually like this news. I create all my YouTube thumbnails using Canva and I really like their product. They also have a background removal tool that works quite well and other AI-based tools. This new one might be really powerful too and useful for AI-related thumbnails More layoffs...Twitter, Meta, and now Amazon are planning to lay off approximately 10,000 employees, one of the largest cuts in the company's history! For those of you looking for a job, please be patient and try not to be discouraged - you will find something! In the meantime, my best recommendation is to suggest work on your portfolio. Build a cool little app, implement Stable Diffusion, and join one or more Kaggle competitions! Try enjoying the "free time" you have and leverage it to improve your chances of finding your future dream job :)


This AI newsletter is all you need #22

Most interesting papers of the week Galactica: A Large Language Model for ScienceGalactica: a large language model that can store, combine and reason about scientific knowledge.Latent-NeRF for Shape-Guided Generation of 3D Shapes and TexturesAn efficient NeRF approach based on Latent Diffusion Models.Extreme Generative Image Compression by Learning Text Embedding from Diffusion Models"We propose a generative image compression method that demonstrates the potential of saving an image as a short text embedding which in turn can be used to generate high-fidelity images which is equivalent to the original one perceptually." Enjoy these papers and news summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #22

Featured Community post from the Discord JacobBum#7456 just published "Breaking it Down: K-Means Clustering". This is a great article which explores and visualizes the fundamentals of K-means clustering with NumPy and scikit-learn. If you write articles and publish them on your blog or on our Medium publication, share them on our discord server and you might get a chance to be featured here too!


This AI newsletter is all you need #22

Article of the week 6 Tips Save Me Time & Memory When Training Machine Learning Models by Youssef Hosni Training machine learning models can be time and memory-consuming, especially if your data is large. It is important to optimize the workflow to save computational time and memory consumption, especially while training the model multiple times with different hyperparameters to find the best hyperparameters for your model. This article shares six practical tips to decrease computational time and memory consumption while training a machine learning model.


This AI newsletter is all you need #22

Our must-read articles In-depth Azure Machine Learning Model Train, Test, and Deploy Pipelines on Cloud With Endpoints for Web APIs by Amit Chauhan META's PEER: A Collaborative Language Model by Salvatore Raieli If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.


This AI newsletter is all you need #22

Job offers Senior Data Scientist / AI Developer @ Uniphore (Spain, Hybrid Remote) Data Scientist @ SwissBorg (Europe, Remote) Lead Data Engineer, Data Platform @ Tubi (Remote) Data Scientist @ Alethea Group (Remote, US) Machine Learning Engineer, Infrastructure @ Earnin (Remote) AI Content Fellowship @ Deepgram (Remote) Interested in sharing a job opportunity here? Contact sponsors@towardsai.net.


Transformers for Multi-Regression - [PART2]

Fine Tuning    In my last post, I showed you how to use a pre-trained transformer to extract context-capturing embeddings and use them for training a multi-regressor. This time I will show you how to train end-to-end the whole transformer, which also means updating the parameters of the pre-trained model. Also, I will show you how to use the weights and biases platform: from the login with the wandb API, to creating and using model artifacts and passing through the model track. All the code sources can be retrieved from my Kaggle notebook Credit: In this Part, I borrowed @debarshichanda model's architecture.


Imports and Config First of all, we will define the CONFIG dictionary and the transformer-related imports that we will be using all over the project:


Custom Dataset Iterator: As explained in the previous post, we will define a subclass of torch.utils.data.Dataset and override __init__ , __len__ and __getitem__ special methods as follows:


Fine Tune Transformer With this approach, the hidden states are not fixed but trainable: for this reason, it requires the classification head to be differentiable. Usually, we use a neural network for the classifier. In this section, we will see how to fine-tune an encoder transformer based on microsoft/deberta-v3-base pretrained model, using a simple and feature-complete training and evaluation API provided by HuggingFace : the Trainer. We will define a custom model that extends microsoft/deberta-v3-base with a trainable neural network head. The custom model will be composed of: Pre-trained Baseline model : load the pre-trained microsoft/deberta-v3-base with the AutoModel.from_pretrained functionMean pooling Layer: we need to add some changes to the previous Mean Pooling function (see part one post): inherit from torch.nn.Module the pooling class and define the mean pooling function within a forward method (see code below)Dropout layer: add a dropout layer for regularizationLinear layer: input size = hidden_state_dim, output size = number of target features (6) The logits output of the linear layer is returned through a SequenceClassifierOutput - a subclass of ModelOutput class- on the forward method (All models must have outputs that are instances of subclasses of ModelOutput: reference here )


Loss & Metric As we will be using a Trainer, we need to define a new loss function corresponding to the target evaluation Metric (in our cases MCRMSE) this loss function will be used to train the transformer. The way to implement this is to define a subclassing Trainer and override the compute_loss() method. In the same way, we want to get the local evaluation for each target class during the evaluation step, so we will provide the Trainer a custom compute_metrics() function that allows calculating the RMSE of each of the six targets (Otherwise, the evaluation would just have returned just the loss evaluation - the MCRMSE).


Weights & Biases Even though HuggingFace Transformers provide a wide range of training checkpointing facilities. W&B provides powerful experiment tracking and model versioning tools with friendly and interactive dashboards. Each experiment project is partitioned on its own. Check out this excellent notebook that describes in detail how to use W&B in kaggle: To connect to Weights & Biases, we need the Access your API key from https://wandb.ai/authorize.There are two ways you can log in using a Kaggle kernel: Run the wandb.login(key=your-api-key) cmd: It will ask for the API key: you can copy/paste it from https://wandb.ai/authorize. Feel free to skip this part if you are not using Kaggle Use Kaggle secrets to store your API key: and use the code snippet below to log in. Click on the Add-ons menus in the Notebook Editor then Secrets: 2. Store the api-key as key-value pair that will be attached to the current Notebook: 3. Copy and paste the code snippet to access to the api-key then use wandb.login() to connect to W&B:


Wandb Arguments For each CV iteration 'i' we will create a new run called FB3-fold-i with 'i' = iteration's number, within a single project that we'll call Feedback3-deberta. Some other parameters: group: the group parameter is especially used to organize individual runs into a larger experiment, here are some use cases examplestags: we will add the model name and the metric tags. as explained in W&B doc Tags are useful for organizing runs together or applying temporary labels like "baseline" or "production". It's easy to add and remove tags in the UI or filter down to just runs with a specific tag.job_type: usually it's either "train" or "eval". Later, it would allow filtering and grouping of similar runs. We will set the job_type to "train"anonymous: this parameter allows controlling anonymous logging. we will set it to "must" which would send the run to an anonymous account instead of to a signed-up user account. For the other options, you can check the documentation For each CV iteration we can instantiate a wandb run as follows: Now let's define the training arguments that will be used by the Huggingface Trainer


Training Arguments Before instantiating our custom Trainer, we will create a TrainingArguments to define the training config.We will set the following parameters: output_dir : The output directory where the model predictions and checkpoints will be written: each CV iteration would have its own directory with a name equal to the number of iterations prefixed with "output-"evaluation_strategy : set to "epoch" which means the Evaluation is done at the end of each epoch.per_device_train_batch_size: The batch size for training. We will set it to 8per_device_eval_batch_size: The batch size for evaluation. We will set it to 16 (to speed up the time execution)num_train_epochs: number of training epochs. As a reminder, during one epoch each sample of the training dataset has been seen by the modelgroup_by_length: as long as we will be using dynamic padding, we will set this parameter to True to group together samples of roughly the same length in the training dataset (to minimize padding applied and be more efficient)max_grad_norm: Maximum gradient norm (for gradient clipping).learning_rate: The initial learning rate for AdamW optimizer. As a reminder, the AdamW optimizerweight_decay: The weight decay to apply to AdamW optimizer: in our case, we will apply the weight decay to all layers except biases and normalization layers Note :Weight decay is a regularization technique that adds a small penalty to the loss function (usually the L2 norm of the weights).loss = loss + weight_decay_parameter * L2_norm_of_the_weights Some implementations apply weight decay only to the weights and not the bias. On the other hand, PyTorch applies weight decay to both weights and bias. gradient_accumulation_steps: Number of steps the gradients should be accumulated across before performing a backward pass: When using gradient accumulation, gradient calculation is done in smaller steps rather than all at once for a batch; 1 (means no gradient accumulation)) load_best_model_at_end: we will set it to True to load the best model found during training at the end of the training: in this case, the save_strategy must be the same as evaluation_strategy: epoch in our casemetric_for_best_model: we will set to the competition metric MCRMSE or eval_MCRMSE (with eval_ prefix)greater_is_better: set it to False, because we want to get the model with lower MCRMSEsave_total_limit: we will set it to 1 to always keep one checkpoint a time (older checkpoints in output_dir will be deleted).report_to: as we are connected to W&B we will set the report_to logs to "wandb"label_name: set The list of label_name parameters to ["labels"], which corresponds to the previously defined field yielded by our custom Dataloader corresponding to the target classes Furthermore, we will define some other parameters for the Trainer: data collator: we need to define how to create a single batch from the list of data inputs returned by the DataloaderWe will use DataCollatorWithPadding which will dynamically pad the received inputs.optimizer: we will use AdamW with decay in all layers except bias and normalization layersscheduler: we will use the get_linear_schedule_with_warmup to create a schedule with a warmup period during which the learning rate increases linearly from 0 to the initial lr (set in the optimizer) and then decreases linearly from the initial lr set in the optimizer to 0.The Scheduler allows us to keep control of the learning rate if, for example, we want to make sure that every update of the learning rate doesn't exceed a lambda value (check this Stackoverflow discussion about the utility of optimizer scheduler) To launch the cross-validation training, first, we have to create the CV fold columns as explained in the PART1 post: The CV training workflow can be implemented as follows: You can access my public W&B dashboard that I created for this project: https://wandb.ai/athena75/FB3-deberta-v3?workspace=user-athena75


Create W&B artifacts For Later use, W&B is very convenient to create model artifacts once the model is fine-tuned. We can use them later and create new versions of our models. To create a model artifact, all you have to do is : Create a wandb.Artifact object with a clear and consistent name, and you have to specify the type parameter it can be a dataset or model, in our case modelAdd the local directory to the artifact: in fact, once you instantiate the model and start to fine-tune it, it creates a local checkpoint containing the model bin as well as the model state and configuration. You have to add it to the artifact.Once the artifact has all the desired files, you can call wandb.log_artifact() to log it. Here is a snip-code example of creating an artifact for each CV model:


Use W&B artifacts for inference: Once the training is accomplished, we can use the stored artifacts on the Weights & Biases server, in our case, to generate model predictions and generate an aggregated output prediction. PS: you can extract the usage code directly from the W&B interface https://wandb.ai/athena75/Feedback3-deberta/artifacts/model/FB3-fold-0/93c08783e5b7c696451a/usage login to your wandb account and Instantiate a default run with wandb.init()Indicate to the use_artifact() method the path to your artifact as well as the type (the model in our case) to retrieve the artifactDownload the artifact directory locally using the download() methodLoad the local model and use it to make predictions Example of implementation :


Credits: This Work is a singular genesis of these excellent resources, do not hesitate to upvote the Kaggle resources: @rhtsingh notebook: the different ways to explore transformer representations@debarshichanda work: from which I borrowed the trainer hyper-parameters and the model architecture@Y.NAKAMA notebook: the loss function and the multi-label - Stratified cross-validation@shreydan notebook: the notebook's global style


Conclusion: Thanks a lot for reading my posts! , I share with you all my work: this Kaggle notebook, as well as my public W&B dashboard. I hope it was clear, and feel free to ask me questions. My address mail is: schopenhacker75@gmail.com For a later post, I intend to address the subject of how to deploy a Transformer in production. Or how to build an MLOps Pipeline for NLP Transformers, I haven't decided yet...


Transformers for Multi-Regression - [PART1]

Transformers as Feature Extractor    All this work is resumed in my Kaggle notebook


Introduction We don't all have the talent of Flaubert, nor the clarity of Bergson, nor the genius of Proust, nor the style and finesse of Zweig, nor the skill of Voltaire, nor the clairvoyance of Schopenhauer... And the list is far from being exhaustive, and thank God that there are writers and philosophers of genius that allow us to escape for a few moments from this materialist world. But as far as we common people are concerned, literary or not, we can hope at least to do our best to respect the rules of the language and write "correctly". The teachers have helped us to learn the basics of the language, so why not, in our turn we would, help them, using our knowledge, to save time to correct the essays of their students, helping them to distinguish the strong and weak points of each of their students and better adapt their pedagogy to the level of each one. In this competition, we are asked to create an efficient model using the pre-scored argumentative essays written by 8th-12th grade English Language Learners based on six analysis metrics: cohesion, syntax, vocabulary, phraseology, grammar, and conventions. The scores range from 1.0 to 5.0 in increments of 0.5. We will describe how to use a hugging-face model to address this type of problem, I chose deberta-v3-base model (here is the corresponding model card), and I will show how we can use it in two efficient ways:  Feature extraction: We use the hidden states as features and just train a classifier on them without modifying the pre-trained model. For this section @cdeotte proposed a brilliant use with this method consisting in using multiple non-finetuned transformers embeddings, then concatenating them and training a standalone classifier: I strongly invite you to see the related discussion and notebook  Fine-tuning: We train the whole model end-to-end, which also means updating the parameters of the pre-trained model. This approach will be discussed in a later post In this part, we will go through the first approach step by step: Introduction to the encoder-based transformer The idea is to use BERT-based models, which are pretrained to predict masked elements of texts, along with a custom classifier: the workflow is the following: 1. Generate token encoding: First, the tokenizer generates one hot encoding called token encoding: each vector has a dimension equal to the tokenizer vocabulary [batch_size, vocab_size] . The AutoTokenizerclass of HuggingFace will automatically load the tokenizer corresponding to the checkpoint name (for this notebook, we will use deberta-v3-base), the tokenizer would generate a dictionary composed of: input_ids : the indices corresponding to each token in the sentencetoken_type_ids: identifies which sequence a token belongs to when there is more than one sequenceattention_mask: identifies the padded elements from real tokens Then the model would take the token encodings and proceed as follows: 2. Generate embeddings: The model converts the token encodings to dense embeddings. Unlike the token encodings, the embeddings are dense= non-zero values. The token encodings are generated with the tokenizer. -> We get a tensor with dimensions [batch_size, max_len] with the model's maximum context size 3. Generate hidden states: The model feeds the embeddings through the encoder stack to return the hidden states for each token input. we obtain a final tensor [batch_size, max_len, hidden_states_dim]We load an AutoModel object to initialize a model with all the weights of the checkpoint ( microsoft/deberta-v3-base in our case) Let's see now how to prepare our dataset to be processed by the transformer:


Prepare Dataset We will proceed by mini-batch, and for that, we will use the torch.utils.data.DataLoader and torch.utils.data.Dataset (you can check the reference here). The Dataset allows returning the dataset samples with their corresponding labels. The DataLoader wraps an iterable around the Dataset to enable easy access to the samples, offering many utilities such as reshuffling the data at every epoch to reduce model overfitting or allowing the use of multiprocessing to speed up data retrieval. To develop our custom Datasetwe have to override __init__ , __len__ and __getitem__ functions.The most important function is __getitem__: it returns a sample from the dataset at a given index idx. The output format of the function must respect the expected format of the model : input_ids : List of token ids to be fed to the model.attention_mask: List of indices specifying which tokens should be attended to by the modellabels: In our case, we deal with a multi-class regression problem, the label is a vector of the six analytics scores. For the training method, we will refer to the undeniable classic cross-validation scheme Multi-label data stratification: In our community of data scientists, it is evidence that the way how to split cross-validation folds has a direct impact on the model performance.Usually, for a one-class problem, the folds are stratified along with the single target (class distribution in case of a discrete target or bins distribution for a continuous target). Well, many works have been carried out to deal with this problem, for example: 2011 : Sechidis - On the stratification of multi-label data2017 : Szymański - Proceedings of the First International Workshop on Learning with Imbalanced Domains Here is a gentle presentation to explain the algorithm. We will use the iterative approach algorithm using the iterative-stratification implementation: Now let's implement the dataset iterator as described previously: This custom Dataset will be used later either to fine-tune the transformer or just use it as a feature extractor. PS: I added the is_train parameter to whether return the "labels" field or not (only the training dataset contains the labels field)


Transformers as Feature Extractors With this method, the encoder weights are frozen, and the hidden states are used as standalone features by the multi-regressor. Since the hidden states are computed once, this method is the best choice if we do not dispose of GPUs : The only axe of freedom that remains to us is how to reduce the hidden states tensor [batch_size, max_len, hidden_states_dim] to a single vector representation: I urge you to consult the invaluable @rhtsingh notebook that enumerates "exhaustively" the different ways to "pool" the hidden states encodings. I tested out the following techniques:  CLS embedding: BERT introduces a [CLS] token tag standing in the first position of each sentence that captures the whole sentence context. The cls embedding simply consists in selecting the first element of each hidden state vector, to get down to [batch_size, 1, hidden_states_dim] vector  Mean pooling: rather than selecting the first element we will consider the average of max_len dimensional embeddings for each hidden state dimension: we obtain a tensor of[batch_size, 1, hidden_states_dim] or just the unsqueezed form : [batch_size, hidden_states_dim]  Max Pooling: to get the max pooling we will take the max across max_len embeddings at each hidden state dimension, the result is a tensor of [batch_size, hidden_states_dim] dimensions  Mean-Max Pooling: we apply the mean pooling and the max pooling, then concatenate the two to get[batch_size, 2*hidden_states_dim] dimensional tensor Here a code sample to get all the embeddings: Now let's see how to generate the embeddings for both train and test datasets: