This AI newsletter is all you need #54

What happened this week in AI by Louie This week we were excited to read Demis Hassabis discussing Deepmind's upcoming new Gemini Large Language model. Historically DeepMind has primarily dedicated its efforts to reinforcement learning (RL) and remained relatively quiet in pursuing large language models (LLMs). Nevertheless, DeepMind was behind the Chinchilla paper which has since become a benchmark for training LLMs and also introduced Sparrow in 2022. There have been many changes at Deepmind this year with Deepmind recently merging with Google Brain and combining resources on AI efforts, and we have been looking forward to updates from Deepmind. In a recent development, DeepMind has unveiled its latest project named Gemini, an upcoming competitor to ChatGPT. DeepMind's previous emphasis on reinforcement learning (AlphaGo) has proven to be highly advantageous, as the RLHF (Reinforcement Learning from Human Feedback) approach served as the secret ingredient behind the impressive performance of chat agents such as ChatGPT. According to Demis Hassabis, the CEO of DeepMind, Gemini combines the capabilities of AlphaGo-type systems with the profound language understanding found in LLMs. The model is still in the development stage and is expected to remain so for another few months. As Hassabis noted, this project's completion could require an investment ranging from tens to hundreds of millions of dollars. We are thrilled to witness the continuous innovation at the forefront of LLMs, such as the GPT-4, and other companies' efforts to push the boundaries of these models' capabilities even further. Moreover, there is growing anticipation to witness a new wave of breakthroughs in LLMs, particularly with a huge increase in training compute available to leading AI companies after recent deliveries of Nvidia's H100 Tensor Core GPUs. We are interested to see if increased compute budgets for these models will be used primarily for more complex training steps and architectures, larger training data sets, or increased model parameters. - Louie Peters - Towards AI Co-founder and CEO


This AI newsletter is all you need #54

Hottest News DeepMind Claims Its Next Chatbot Will Rival ChatGPT DeepMind is developing a new chatbot named Gemini, intended to compete with, and potentially surpass, OpenAI's ChatGPT. By leveraging the achievements of AlphaGo and the language capabilities of LLMs, DeepMind aims to establish dominance in the generative AI market. 2. ElevenLabs Introduces Its Voice Library ElevenLabs has recently introduced Voice Library, a community platform integrated with a multilingual model that facilitates the development of lifelike synthetic voices with consistent primary speech characteristics for commercial applications. The Voice Design tool allows users to customize age, gender, and accent to generate unique and natural-sounding voices. 3. MosaicML Agrees to Join Databricks to Power Generative AI for All MosaicML, a startup dedicated to democratizing large-scale neural network training and inference, has announced a collaboration with Databricks in a $1.3B deal. The partnership aims to drive advancements in generative AI software expertise and extend customer reach while enhancing engineering capabilities. 4. Advancing Innovation With Open-Source AI: Hugging Face CEO Testifies Before the US Congress The CEO of Hugging Face, Clement Delangue, recently testified before the US Congress on Open-Source AI. In his testimony, he highlighted the importance of open-source AI in advancing innovation, promoting fair competition, and ensuring responsible development. Delangue emphasized that open-source principles democratize AI and foster a more inclusive and collaborative future in the field. 5. Adobe Indemnity Clause Designed To Ease Enterprise Fears About AI-Generated Art Adobe offers an indemnity clause to address copyright concerns surrounding their generative AI tool Firefly. By training the model on legal content and promising to cover any copyright claims, Adobe aims to ease enterprise users' worries and ensure the legality and safety of AI-generated artwork.


This AI newsletter is all you need #54

Five 5-minute reads/videos to keep you learning Announcing the First Machine Unlearning Challenge Google has announced the first Machine Unlearning Challenge, a collaboration between academic and industrial researchers. This new area of machine learning focuses on eliminating the impact of certain training examples from a model to protect privacy rights. The challenge, held on Kaggle, aims to evaluate unlearning models' forgetting quality and model utility, providing insights for improvement. 2. The Rise Of The AI Engineer The article delves into the emergence of AI engineering as a specialized field and outlines the necessary skills for success in this domain. It emphasizes the importance of possessing a solid comprehension of machine learning algorithms, data manipulation, and programming languages, as well as the ability to bridge the gap between research and implementation to create practical AI solutions. 3. The Generative AI Revolution: Exploring the Current Landscape This article provides an overview of the present state of Generative AI, emphasizing its capacity to generate coherent text, images, and code. It discusses notable models such as the Transformer, GPT family, Palm models, Chinchilla model, Megatron Turing model, and LlaMa models. The post also explores the potential impact of Generative AI in various fields, including animation, gaming, art, movies, and architecture. 4. AI and the Automation of Work ChatGPT and generative AI are poised to revolutionize the way we work. However, how distinct is this transformation compared to previous waves of automation spanning the last two centuries? Furthermore, what implications does it hold for employment? This article explores paradigm-breaking technologies from the past and endeavors to envision the future impact of AI on the nature of work. 5. What Is Langchain and Why Should I Care as a Developer? Langchain is experiencing remarkable growth as one of the fastest-growing open-source projects in history. This post provides a spotlight on how LangChain empowers developers to undertake incredible projects, offering a high-level overview of its capabilities. The author also shares a personalized account of their experimentation journey with the framework.


This AI newsletter is all you need #54

Papers & Repositories The Curse of Recursion: Training on Generated Data Makes Models Forget This paper explores the potential consequences for GPT-{n} as LLMs increasingly contribute a significant portion of the language available online. It reveals that utilizing model-generated content during training leads to irreversible defects in the resultant models, resulting in the disappearance of the tails of the original content distribution. If the output is not curated diligently, one may encounter a phenomenon known as "model collapse." 2. BradyFU/Awesome-Multimodal-Large-Language-Models The repository features a curated collection of papers and datasets on Multimodal Large Language Models (MLLMs). It offers valuable insights into various aspects such as multimodal instruction tuning, in-context learning, chain-of-thought, and LLM-aided visual reasoning. 3. Towards Language Models That Can See: Computer Vision Through the LENS of Natural Language The paper introduces LENS (Language Models ENhanced to See), a modular approach aimed at addressing computer vision problems by harnessing the capabilities of large language models (LLMs). The system employs a language model to engage in reasoning over outputs from a collection of independent and highly descriptive vision modules, which collectively offer comprehensive information about an image. 4. A More Efficient Way to Train a CLIP Model Expanding on the recent work of CLIPA, which introduces an inverse scaling law for CLIP training, this paper introduces CLIPA-v2. CLIPA-v2 enhances the efficiency of image-text matching models, such as CLIP, by utilizing shorter sequences. It achieves a zero-shot ImageNet accuracy of 81.1% while requiring only $10,000 in resources. 5. End-to-end Autonomous Driving: Challenges and Frontiers This survey offers a comprehensive analysis of over 250 papers, covering various aspects such as motivation, roadmap, methodology, challenges, and future trends in end-to-end autonomous driving. It delves into several critical challenges, including multi-modality, interpretability, causal confusion, robustness, and world models, among others. Enjoy these papers and news summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #54

Weekly AI Podcast In this week's episode of the "What's AI" podcast, Louis Bouchard interviews Petar Veličković, a research scientist at DeepMind and an affiliate lecturer at Cambridge. Petar shares insights on the value of a Ph.D., emphasizing its role as a gateway to research and the opportunities it provides for building connections and adaptability. He also highlights the evolving landscape of AI research, underscoring the importance of diverse backgrounds and contributions. The interview provides valuable perspectives on academia versus industry, the role of a research scientist, working at DeepMind, teaching, and the significance of curiosity in driving impactful research. Tune in on YouTube, Spotify, or Apple Podcasts if you are interested in AI research!


This AI newsletter is all you need #54

Upcoming Community Events The Learn AI Together Discord community hosts weekly AI seminars to help the community learn from industry experts, ask questions, and get a deeper insight into the latest research in AI. Join us for free, interactive video sessions hosted live on Discord weekly by attending our upcoming events. Reading group on Time Series: conformal prediction & its application in depth. In critical domains like medical diagnoses and safety-critical systems, quantifying prediction uncertainty in machine learning is crucial. Conformal prediction offers a robust framework for this purpose. It allows the quantification of uncertainty for any machine learning model as a post-processing layer, without requiring model refitting. Join us for an upcoming talk where we delve into the applications of conformal prediction. Attendees are encouraged to familiarize themselves with the insights shared on the MLBoost YouTube channel before the event. Join the event here and discover how conformal prediction enhances reliable decision-making by measuring uncertainty beyond traditional point predictions. Date & Time: 7th July 2023, 10:00 am EST 2. Reading Group: Segment Anything Model (SAM) Learn AI Together's weekly reading group presents informative presentations and discussions on the latest advancements in AI. This (free) event offers a wonderful opportunity to learn, ask questions, and engage with community members. This week, the focus will be on reviewing the "Segment Anything" paper, a recent publication from Meta research. You can access the paper here and join the discussion here. Date & Time: 8th July 2023, 10:00 pm EST Add our Google calendar to see all our free AI events!


This AI newsletter is all you need #54

Meme of the week! Meme shared by neon8052


This AI newsletter is all you need #54

Featured Community post from the Discord Akshitireddy has developed an open-source project called Interactive LLM Powered NPCs, which revolutionizes the way users interact with non-player characters (NPCs) in games. This project enables users to engage in immersive conversations with NPCs, leveraging their microphones to speak, listening to their voices, and witnessing realistic facial animations. The project focuses on enhancing the gaming experience in previously released titles such as Cyberpunk 2077, Assassin's Creed, GTA 5, and other popular open-world games. Check it out on GitHub and support a fellow community member. Share your feedback and questions in the thread here!


This AI newsletter is all you need #54

AI poll of the week! Join the discussion on Discord.


This AI newsletter is all you need #54

Article of the week Train and Deploy Custom Object Detection Models Without a Single Line Of Code by Peter van Lunteren Object detection is a computer vision technique used to identify specific objects within images. While numerous online tutorials cover object detection, none of them offer an automated method that eliminates the need for coding. In this tutorial, the author introduces EcoAssist, an open-source application hosted on GitHub, which simplifies object detection and makes it highly accessible to users.


This AI newsletter is all you need #54

Our must-read articles Gorilla: Everything You Need to Know by Muhammad Arham Google's Latest AI Model Enables Virtual Try On clothes with Unchanged Details and Flexible Poses by Shen Huang Meet vLLM: UC Berkeley's Open Source Framework for Super Fast and Chearp LLM Serving by Jesus Rodriguez 10 Sklearn Treasure Features Overlooked By 99% of Online Courses by Bex T. If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.


This AI newsletter is all you need #54

Job offers Research Engineer (Prototyping) GitHub Next @GitHub (Remote) Product Engineer, PropTech @Picket Homes (Nashville, TN, USA) Conversational AI Designer @Cresta (Remote) Forward Deployed Engineer @Cohere (Remote) Machine Learning Researcher @Shiru (Remote) Data Engineer Mid Level @pulseData (Remote) Machine Learning Engineer @Acentra Health (Remote) Interested in sharing a job opportunity here? Contact sponsors@towardsai.net. If you are preparing your next machine learning interview, don't hesitate to check out our leading interview preparation website, confetti!


Boost Your Fine-Tuning Performance with TPGM

Background At the recent CVPR 2023 conference in Vancouver, I had the privilege of exploring cutting-edge research in the field of transfer learning. Transfer learning is applicable to multiple domains, such as computer vision, natural language processing, and molecular modeling. Among the standout papers I encountered, one stood out for its innovative approach to fine-tuning and its potential to overcome existing limitations. In the past several years, fine-tuning large models on a specific task has gained popularity due to the high accuracy that is achievable with less training and less data. It has been shown that the initial layers of the network tend to learn more general information, and the final layers are more "task specific", therefore, we would like to retain that general information while targeting our own tasks. Methods have been proposed based on this knowledge. For example, it would make sense to choose a different learning rate for each layer (using smaller learning rates for the first few layers and larger ones for the final ones). The downside to this is that it adds several new hyper-parameters to the problem, and thus it is not feasible when trying to train larger models on sizable datasets. This leads to a reliance on manual heuristics and time-consuming hyper-parameter searches to find optimal learning rates.


The Newly Proposed Method In the paper "Trainable Projected Gradient Method for Robust Fine-tuning," the authors address the issues explained above through an exciting solution termed Trainable Projected Gradient Method (TPGM) [1]. By formulating fine-tuning as a bi-level constrained optimization problem, TPGM automates the process of learning fine-grained constraints for each layer. TPGM introduces a set of projection radii, representing distance constraints between the fine-tuned model and the pre-trained model, and enforces them using weight projections. What sets TPGM apart is its ability to "learn" these projection radii through a novel end-to-end bi-level optimization approach, eliminating the need for a manual search or slow non-derivative-based optimization techniques (e.g., grid searches). These radii are optimized based on the validation dataset, so it is important to make sure the rest of the parameters are frozen when conducting this portion of the optimization to avoid data leakage. Normally, the loss can be described as: This represents the traditional way of tuning hyper-parameters in machine learning. In this case, the objective is to minimize the loss function on a validation set, where: (x, y) - represents a pair of input dataL(·) - represents the task loss functionθt - represents the trainable model weightsλ - represents the hyper-parameters, such as the learning rateDval and Dtr - represent the validation and training datasets respectively The traditional process can be considered a bi-level optimization problem because it involves two steps. First, we adjust the hyper-parameters λ to reduce the error on the validation set, and then within this adjusted context, we tweak the model parameters θt to minimize the error on the training set. The loss function presented in Tian et al. [1], extends this formulation for fine-tuning a pre-trained model by introducing an additional constraint. This new formulation not only minimizes the loss function as before but also ensures the distance between the fine-tuned model parameters (θt) and the pre-trained model parameters (θ0) does not exceed a predefined limit γ. The additional parameters in this loss function include: γ - a scalar that represents the maximum allowed distance between the pre-trained model and the fine-tuned modelθ0 - representing the weights of the pre-trained modelθt-θ0 - represents the difference between the weights of the fine-tuned model and the pre-trained model, effectively measuring the 'distance' between them The addition of the constraint θt − θ0* ≤ γ aims to maintain the generalization and robustness of the fine-tuned model by ensuring it does not deviate too much from the pre-trained model (the amount of deviation allowed will be determined by the performance on the validation dataset). This forms a bi-level constrained minimization problem.


Conclusion The authors' experiments demonstrate that TPGM outperforms vanilla fine-tuning methods in terms of robustness to out-of-distribution (OOD) data while maintaining competitive performance on in-distribution (ID) data. For instance, when applied to datasets like DomainNetReal and ImageNet, TPGM showcases significant relative improvements in OOD performance. To delve deeper, the unique aspects of TPGM and its implications can be better understood through the following key points: TPGM presents a transformative solution for fine-tuning in transfer learning.TPGM formulates fine-tuning as a bi-level constrained optimization problem, which aids in automating the learning of fine-grained constraints for each layer.TPGM alleviates the need for task specific heuristics and time consuming hyper-parameter searches.A key finding is that different layers require different levels of regularization. The results show that the lower layers of the neural network are more tightly constrained, indicating their closer proximity to the ideal model. This is consistent with the common understanding that lower layers tend to learn more general features. As someone working in the field of deep learning, with previous research experience in optimization, I find this paper to be extremely impactful. The proposed method, TPGM, offers a significant leap forward in the world of transfer learning, potentially paving the way for more efficient, robust, and interpretable models in the future.


Citation [1] Tian, J., Dai, X., Ma, C-Y., He, Z., Liu, Y-C., & Kira, Z. (2023). Trainable Projected Gradient Method for Robust Fine-tuning. In Proceedings of the Conference on Computer Vision and Pattern Recognition (pp. TBD). doi:10.48550/arXiv.2303.10720


Resources The PaperThe Papers Code


Connect with Me! I'm an aspiring deep-learning researcher currently working as a Computer Vision Engineer at KEF Robotics in Pittsburgh! Connect with me, and feel free to reach out to chat about anything ML related! Follow me on MediumTwitterLinkedInGitHubBuy me a coffee?


The Generative AI Revolution: Exploring the Current Landscape

What is Generative AI? Generative AI is a subfield of machine learning that involves training artificial intelligence models on large volumes of real-world data to generate new contents (text, image, code,...) that is comparable to what humans would create. This is achieved by training algorithms on large datasets to identify patterns and learn from them. Once the neural network has learned these patterns, it can generate new data that adheres to the same patterns. However, this process is computationally intensive. Fundamentally, a generative AI for NLP applications will process an enormous corpus on which it has been trained and respond to prompts with something that falls within the realm of probability, as learnt from the mentioned corpus. For example, autocomplete is a low-level form of generative AI. Advanced models like ChatGPT and DALL-E take the concept to a whole new level. Different model architectures, such as diffusion models and Transformer-based large language models (LLMs), can be employed for generative tasks such as image and language generation. Diffusion models are a type of generative AI model that can be used for a variety of tasks, including image generation, image denoising, and inpainting. Similarly, the Transformer architecture revolutionized the language domain. The new era of language models are Transformer-based, which is a type of deep learning architecture for natural language processing (NLP) tasks. They utilize a self-attention mechanism to transform the input sequence into a set of context-aware high dimensional vectors (also known as embeddings) that can be used for a variety of NLP tasks, including language generation, machine translation, and text classification. The most well-known transformer-based LLMs are the GPT family, developed by OpenAI. The primary advantage of transformer-based LLMs over traditional NLP models is that they are highly parallelizable and can handle long-range dependencies between words in a sentence more effectively. This makes them more suitable for tasks that require a deeper understanding of the context, such as text summarization or generating a coherent and fluent text. Let's explore the history and current state of generative AI and the key players shaping its future.


The Generative AI Revolution Generative AI has been around for several years. One of the earliest examples is the Eliza chatbot developed by Joseph Weizenbaum in 1966. However, these early implementations relied on a rules-based approach that had several shortcomings, such as a limited vocabulary, lack of context, and overreliance on patterns. As a result, they were prone to frequent breakdowns, making it difficult to customize and expand these initial chatbots. Recently, significant progress has been made in AI and machine learning, resulting in the development of advanced generative AI systems. It's no coincidence that these breakthroughs have happened all at once. They're based on a new class of AI models that are incredibly flexible and powerful, surpassing anything we've seen before. In deep learning, there are three critical components that contributed the most to their recent success: scaling models, large datasets, and more compute power - all working together to bring us to this exciting point in AI advancement.


Progress in GPUs and their application to Machine Learning GPUs are designed for parallel processing, making them well-suited for the computationally intensive tasks involved in training deep neural networks. Unlike CPUs, which focus on sequential processing, GPUs have thousands of smaller cores that can handle multiple tasks simultaneously, allowing for faster training of large networks. A key breakthrough for machine learning was the intuition that GPUs could be used for Neural Networks, together with software progress such as Nvidia's release of CUDA in 2007, a programming language that allowed GPUs to be used as general-purpose computers.


Alexnet - 2012 - The Deep Learning Revolution The modern AI revolution began in 2012 with step change progress in deep learning and convolutional neural networks (CNNs), which were particularly effective in solving computer vision problems. Although CNNs had been around since the 1990s, they were not practical due to their intensive computing power requirements. However, In 2009, Stanford AI researchers introduced ImageNet, a labeled image dataset used to train computer vision algorithms, and a yearly challenge. In 2012, AlexNet combined CNNs trained on GPUs with ImageNet data to create the most advanced visual classifier at the time. The model outperformed the runner-up by a significant margin of nearly 11%! The success of CNNs, the ImageNet dataset, and GPUs drove significant progress in computer vision.


Transformers: Attention Is All You Need (Google) - 2017 One critical area where deep learning lagged was natural language processing (NLP), which involves getting computers to understand and hold a coherent conversation with humans rather than translation or classification. NLP breakthroughs were needed to bridge this gap. Previously, researchers relied on models such as recurrent neural networks (RNNs) and long short-term memory (LSTM) to process and analyze time-based data. These models were proficient at recognizing short sequences such as spoken words but struggled with longer sentences and paragraphs. The architectural flaws of these models was unable to capture the complexity and richness of ideas that arise when sentences are combined into larger bodies of text. A significant breakthrough in AI was the development of the "Transformer" model by Google with the very popular paper "Attention Is All You Need". This model represented a major milestone as it revolutionized the approach to translation problems by utilizing a mechanism called "attention": a particular neural network that allowed the model to analyze the entire input sequence and determine relevance to each component of the output. In the years to come, Transformers have been found to be state-of-the-art models for many other NLP tasks as well, and recently also in other domains such as computer vision.


Next word prediction, scale and fine tuning - BERT (Google) and GPT (OpenAI) family - 2018 With the advancement of Transformers, a key further breakthrough finding was the potential to train on unstructured data via next word prediction objective on website contents. It introduced the models such as BERT and GPT-2. This delivered surprising capabilities and "zero shot" performance at completing new tasks the model hadn't been trained for. OpenAI also continued to probe the ability for the performance of these models to continue increasing with more scale and more training data. One of the major challenges faced by researchers was acquiring the right training data. ImageNet, a collection of one hundred thousand labeled images, required a significant human effort. Despite the abundance of text available on the Internet, creating a meaningful dataset for teaching computers to work with human language beyond individual words is a time-consuming process. Additionally, labels created for one application using the same data may not apply to another task. With the advancements of BERT and first iteration of GPT, we started to harness the immense amount of unstructured text data available on the internet and the computational power of GPUs. OpenAI further advanced this approach with their development of GPT-2 and GPT-3 models, which are short for "generative pre-trained transformer." These models are specifically designed to generate new words in response to input and are pre-trained on vast amounts of text using the next world prediction objective. Another key breakthrough in these large transformer models is the concept of "fine tuning" - or adapting a large model to new more specific tasks or a new smaller and targeted data set - to improve performance in a particular domain with far lower compute cost than training a new model from scratch. For example, a foundational language model like GPT-3 may be fine-tuned on a dataset of medical documents to create an instruction-tuned model for medical document processing. This model will be better at understanding medical terminology, identifying medical entities, and extracting relevant information from medical texts.


Instruction Tuning - Instruct GPT and ChatGPT (OpenAI) - 2022 The most recent advancement which has led to the Generative AI landscape today is the concept of Instruction Tuning - taking a model which has just been trained to predict the next word of a text document - and teaching it (via fine tuning) to actually follow human instructions and preferences. This made it far easier to interact with these LLMs and to get them to answer questions and perform tasks without getting sidetracked by just trying to predict the next word. A fortunate feature of instruction tuning is that not only it helps to increase the accuracy and capabilities of these models, but they also help align them to human values and helps prevent them from generating undesired or dangerous content. OpenAI's specific technique for instruction tuning is called reinforcement learning with human feedback (RLHF) where humans are used to train the model by ranking its responses. Building on top of Instruction Tuning, OpenAI released ChatGPT - which reorganized instruction tuning into a dialogue format and created an easy to use interface for interacting with the AIs. This has catalyzed the mass awareness and adoption of Generative AI products and has led to the landscape we have today.


The Current LLM Landscape The breakthroughs in Generative AI have left us with an extremely active and dynamic landscape of players. This consists of 1) AI hardware manufacturers such as Nvidia and Google, 2) AI cloud platforms such as Azure, AWS, Nvidia and Google, 3) open source platforms for accessing the full models, such as Hugging Face, 4) access to LLM models via API such as OpenAI, Cohere and Anthropic and 5) access to LLMs via consumer products such as ChatGPT and Bing. Additionally, there are many more breakthroughs happening each week in this universe such as the release of multi modal models (that can understand both text and image), new model architectures (such as Mixture of Experts), Agent Models (models that can set tasks and interact with each other and other tolls). This all leads to many questions such as; How will most people interact with LLMs?Who will be the leading players going forward?How fast will the capabilities of these models keep progressing?Are open source models dangerous because of the lack of control of their outputs and use, or are they beneficial due to democratizing access to this technology?


1. OpenAI's GPT Models Notable Models Task specific models Find model information here: https://platform.openai.com/docs/models/gpt-3 Image & Audio Models OpenAI, the company behind the GPT models, is an AI research and deployment company. The San Francisco-based lab was founded in 2015 as a nonprofit with the goal of building "artificial general intelligence" (AGI), which is essentially software as smart as humans. OpenAI conducts innovative research in various fields of AI, such as deep learning, natural language processing, computer vision, and robotics, and develops AI technologies and products intended to solve real-world problems. OpenAI transitioned into a for-profit company in 2019. The company plans to cap the profit of the investors at a fixed multiple of their investment (noted by Sam Altman as currently ranging between 7x and 100x depending on the investment round date and risk). As per the WSJ OpenAI was initially funded by $130m of charity funding (Elon Musk tweeted he contributed $100m) and has since raised at least $13bn led by Microsoft (where OpenAI makes use of Azure cloud credits). With the Microsoft partnership, OpenAI's ChatGPT, along with Microsoft's own search AI, created an improved version of Bing and transformed Microsoft's Office productivity apps. In 2019, OpenAI released GPT-2, a model that could generate realistic human-like text in entire paragraphs with internal consistency, unlike any of the previous models. The next generation, GPT-3, launched in 2020, was trained with 175 billion parameters. GPT-3 is a multi-purpose language tool that users can access without requiring them to learn a programming language or other computer tools. In November 2022, OpenAI released ChatGPT, which is a superior version of the company's earlier text generation models with the capability to generate humanlike prose. After the success of ChatGPT (GPT 3.5), Open AI released GPT-4 in March 2023, which has multimodal capabilities. The model processes both image and text inputs for text generation. The model has a maximum token count of 32,768 capable of generating around 25,000 words as compared to GPT-3.5 which has 4,096 tokens context size. GPT-4 produces 40% more factual responses and its response rate for disallowed content is down by 82% as compared to previous models. (reported by OpenAI)


2. Google's Palm Models Google AI, formerly known as Google Research, is the AI research and development arm of Google. It was unveiled at Google I/O 2018. Google has contributed many of the most significant papers in breakthroughs in modern machine learning. Google's largest publicly disclosed model is its Pathways Language Model (PaLM) which has likely recently been rolled out in its Bard chatbot. PaLM has been used as a foundation model in several Google projects including the instruction tuned PaLM-Flan, and the recent PaLM-E (the first "embodied" multimodal language model). The pre-training of PaLM involved self-supervised learning drawing from a large text corpus that included multilingual web pages (27%), English books (13%), open-source code repositories, and source code from GitHub (5%), multilingual Wikipedia articles (4%), English news articles (1%), and other social media conversations (50%). PaLM excelled in 28 out of 29 NLP tasks in the few-shot performance, beating the prior larger models like GPT-3 and Chinchilla. PaLM variants scale up to 540 billion parameters (vs GPT-3 at 175 billion) and trained on 780 billion tokens (vs GPT-3 300bn) - totalling around 8x more compute training than GPT-3 (but likely considerably less than GPT-4). PaLM was trained across multiple TPU v4 pods. Being a dense decoder-only Transformer model, PaLM is trained on two TPU V4 pods connected over a data center network and uses a combination of model and data parallelism. Researchers used 3072 TPU v4 chips in each pod, attached to 768 hosts. This large TPU configuration allows for efficient scale training without using pipeline parallelism. The Pathways system allows for scaling a model across Google's thousands of Tensor Processing Unit chips.


3. DeepMind's Chinchilla Model DeepMind Technologies, founded in 2010, is a British AI research laboratory. It became a wholly owned subsidiary of Alphabet Inc., in 2015 after its acquisition by Google in 2014. DeepMind has created a neural network or a Neural Turing machine that tries to replicate the short-term memory of the human brain. In 2016, DeepMind's AlphaGo program defeated a human professional Go player, and their program AlphaZaro defeated the most powerful programs in the games of Go and Shogi. The program acquired competence using reinforcement learning. In 2020, DeepMind's program AlphaFold started making advances in the problem of protein folding and by July 2022, it had predicted over 200 million protein structures. In April 2022, Flamingo, a single visual language model program capable of describing any picture, was launched. Three months later, in July 2022, DeepNash was announced; as a model-free multi-agent reinforcement learning system. DeepMind developed a language model called Chinchilla AI in March 2022, which claimed to outperform GPT-3. A key breakthrough in the Chinchilla paper was that previous LLMs had been trained on too little data - for a given parameter size the optimum model should use far more training data than GPT-3. While more training data takes more time to gather, and leads to more training costs, achieving more capable models for a smaller parameter size has huge benefits for inference costs (the costs needed to run and use the finished model which scale with parameter size). Chinchilla has 70B parameters (60% smaller than GPT-3) and was trained on 1,400 tokens (4.7x GPT-3). The average accuracy rate of Chinchilla AI is 67.5% on Measuring Massive Multitask Language Understanding (MMLU) and outperforms other large language model platforms like Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (300 parameters and 530B parameters) on a large range of downstream evaluation tasks.


4. Microsoft & Nvidia's Megatron Turing Model Nvidia is a company that designs GPUs and APIs for data science and high-performance computing, and SoCs for mobile computing and the automotive market. The company is a leading supplier of AI hardware and software. Additionally, Nvidia's CUDA API enables the creation of massively parallel programs that leverage GPUs. Developed by NVIDIA's Applied Deep Learning Research team in 2021, the Megatron-Turing model consists of 530 billion parameters and 270 billion training tokens. Nvidia has provided access via an Early Access program for its managed API service to its MT-NLG model. Nvidia has made many of its LLM and Generative AI models and services available through its new DGX Cloud platform.


5. Meta's LlaMa Models Meta AI, formerly known as Facebook Artificial Intelligence Research (FAIR), is an artificial intelligence laboratory that aims to share open-source frameworks, tools, libraries, and models for research exploration and large-scale production deployment. In 2018, they released the open-source PyText, a modeling framework focused on NLP systems. Then, in August 2022, they announced the release of BlenderBot 3, a chatbot designed to improve conversational skills and safety. In November 2022, Meta developed a large language model called Galactica, which assists scientists with tasks such as summarizing academic papers and annotating molecules and proteins. Released in February 2023, LLaMA (Large Language Model Meta AI) is a transformer-based foundational large language model by Meta that ventures into both the AI and academic spaces. The model aims to help researchers, scientists, and engineers advance their work in exploring AI applications. It will be released under a non-commercial license to prevent misuse, and access will be granted to academic researchers, individuals, and organizations affiliated with the government, civil society, academia, and industry research facilities on a selective case-by-case basis. The sharing of codes and weights allows other researchers to test new approaches in LLMs. The LLaMA models have a range of 7 billion to 65 billion parameters. LLaMA-65B can be compared to DeepMind's Chinchilla and Google's PaLM. Publicly available unlabeled data was used to train these models, and training smaller foundational models require less computing power and resources. LLaMA 65B and 33B have been trained on 1.4 trillion tokens in 20 different languages, and according to the Facebook Artificial Intelligence Research (FAIR) team, the model's performance varies across languages. The data sources used for training included CCNet (67%), GitHub, Wikipedia, ArXiv, Stack Exchange, and books. LLaMA, like other large scale language models, has issues related to biased & toxic generation and hallucination.


6. Eleuther's GPT-Neo Models Founded in July 2020 by Connor Leahy, Sid Black, and Leo Gao, EleutherAI is a non-profit AI research lab The organization has emerged as a leading player in large-scale natural language processing research, with a focus on interpretability and alignment of large models. Their mission is to ensure that the ability to study foundation models is not limited to a few companies, promoting open science norms in NLP, and creating awareness about capabilities, limitations, and risks around these models. In December 2020, EleutherAI curated a dataset of diverse text for training LLMs called the Pile, which consisted of an 800GiB dataset. Subsequently, in March 2021, they released GPT-Neo models. EleutherAI also released GPT-J-6B in June 2021, which is a 6 billion parameter language model, making it the largest open-source GPT-3 like model at the time. Additionally, they combined CLIP with VQGAN to develop a free-to-use image generation model, which guided the foundation of Stability AI. EleutherAI also trains language models in other languages, such as Polyglot-Ko, which were trained in collaboration with the Korean NLP company TUNiB. EleutherAI used Google's TPU Research Cloud Program, but by 2021, they took funding from CoreWeave. The company also uses TensorFlow Research Cloud for cheaper computing resources. In February 2022, EleutherAI released the GPT-NeoX-20b model, which became the largest open-source language model of any type at the time. In January 2023, the company was formally incorporated as a non-profit research institute. EleutherAI's NLP model, GPT-NeoX-20B, is trained on 20 billion parameters using the company's GPT-NeoX framework and GPUs from CoreWeave. The GPT-NeoX-20B model has a 72% accuracy on LAMBADA sentence completion. When measured for zero-shot accuracy for Stem using Hendrycks Test Evaluation, it had an average of 28.98%. The model uses the Pile dataset for training and consists of data from 22 sources that falls under the following 5 categories: academic writing (Pubmed Abstracts and PubMed Central, arXiv, FreeLaw, USPTO Backgrounds, PhilPapers, NIH Exporter), web-scrapes and Internet resources (CommonCrawl, OpenWebText2, StackExchange, Wikipedia-English), prose (BookCorpus2, Bibliotik, Project Gutenberg), dialogue (Youtube subtitles, Ubuntu IRC, OpenSubtitles, Hacker News, EuroParl), and miscellaneous (GitHub, the DeepMind Mathematics dataset, Enron Emails). GPT-NeoX-20B is publicly accessible and a pre-trained general-purpose autoregressive transformer decoder language model. It is a powerful few-shot reasoner with 44 layers and a hidden dimension size of 6144 and 64 heads. Additionally, it uses 1.1. Rotary Positional Embeddings instead of learned positional embeddings, as found in GPT models.


7. Cohere's XLarge Founded in 2019 by Aidan Gomez, Ivan Zhang, and Nick Frosst, Toronto-based Cohere specializes in natural language processing (NLP) models. Cohere has improved human-machine interactions and aided developers in performing tasks such as summarizing, classification, finding similarities in content, and building their own language models. Cohere's API helps users design tools for language comprehension and offers a backend toolkit for integration in multiple ways. Cohere provides two types of large language models: Generation Language Models and Representation Language Models. The company uses a foundation model to train AI systems on large-scale data, enabling them to learn from new data to perform various tasks. Generative AI aims to develop human-like creations through coding, and Cohere competes with similar model providers like OpenAI and Anthropic, with the point of differentiation being the focus on serving enterprise users in incorporating generative AI. Cohere's goal is to make NLP accessible to all while building machines that are safe to use. In September 2021, Cohere raised $40 million, and a few months later, in November 2021, Google Cloud announced its partnership with Cohere. The company intends to use Cloud's TPU for the development and deployment of its products, and Sagemaker by Amazon also gives access to Cohere's language AI. Cohere powers Hyperwrite, which helps in quickly generating articles. AWS has also announced a partnership with Cohere AI. To date, Cohere has raised $170 million, and with the ongoing rush of funding in AI platforms, the Canadian startup is expected to be valued at $6 billion. Cohere is set to introduce a new dialogue model to aid enterprise users in generating text while engaging with the model to fine-tune the output. Cohere's Xlarge model resembles ChatGPT but provides developers and businesses with access to this technology. Cohere's base model has 52 billion parameters compared to OpenAI's GPT-3 DaVinci model, which has 175B parameters. Cohere stresses on accuracy, speed, safety, cost, and ease of use for its users and has paid much attention to the product and its design, developing a cohesive model.


8. Anthropic AI's Claude Anthropic is an American AI startup and public benefit corporation founded in 2021 by Daniela Amodei and Dario Amodei, former members of OpenAI. The company specializes in developing AI systems and language models, with a particular focus on transformer architecture. Anthropic's research on the interpretability of machine learning systems covers fields ranging from natural language and interpretability to human feedback, scaling laws, reinforcement learning, and code generation, among others. The company stresses the application of responsible AI and presents itself as an AI safety and research company working towards building reliable, steerable, and interpretable AI systems. By 2022, Google had invested nearly $400 million in Anthropic, resulting in a formal partnership between the two companies and giving Google a 10% stake in Anthropic. Outside backing amounted to $580 million, with total investments in Anthropic exceeding $1 billion to date. Anthropic has developed a conversational large language model AI chatbot named Claude, which uses a messaging interface and a technique called constitutional AI to better align AI systems with human intentions. AnthropicLM v4-s3 is a 52-billion-parameter, autoregressive model, trained unsupervised on a large text corpus. The ten principles used by Anthropic are based on the concepts of beneficence, non-maleficence, and autonomy. Claude is capable of a variety of conversational and text-processing tasks, such as summarization, search, creative and collaborative writing, Q&A, and coding. It is easy to converse with, more steerable, and takes directions on personality, tone, and behavior. Anthropic offers two versions of Claude - Claude (Claude-v1) and Claude Instant. Claude-v1 is a powerful, state-of-the-art high-performance model capable of handling complex dialogue, creative content generation, and detailed instructions. Claude Instant is lighter, less expensive, and much faster, making it suitable for handling casual dialogues, text analysis, and summarization. However, Claude is an expensive platform compared to ChatGPT. Anthropic vouches for Claude to be an honest, helpful, and harmless AI system, and much less likely to produce harmful outputs than present chatbots, which have been known to be toxic, biased, use offensive language and hallucinate. According to Anthropic, Claude cannot access the internet and is designed to be self-contained and trained to avoid sexist, racist, and otherwise toxic outputs, along with preventing human engagement in illegal and unethical activities. However, compared to ChatGPT, Claude is poor at math and programming. Still, the platform has also been seen to hallucinate and provide dubious instructions. Another major concern is that it is possible to intrude upon Claude's built-in safety features through clever prompting. The embargo on media coverage of Claude was lifted in January 2023, and a waiting list of users who wanted early access to Claude was released in February. Claude is now available and accessible to users through the Poe app by Quora. Also, Discord Juni Tutor Bot, an online tutoring solution, is powered by Anthropic. Additionally, Claude has found integration with Notion, DuckDuckGo, RobinAI, Assembly AI, and others.


9. AI21's Jurassic Models AI21 Labs specializes in Natural Language Processing to develop generative AI models that can understand and generate text. The Tel Aviv-based startup was founded in 2017 by Yoav Shoham, Ori Goshen, and Amnon Shashua. AI21 has emerged as a rival to OpenAI. In 2019, the startup raised $9.5 million, and in October 2020; it launched Wordtune which was an AI-based writing app. AI21 Labs launched AI21 Studio and Jurassic-1 in August 2021. This was followed by Walden Catalyst investing $20 million in AI21 Labs in November, soon after which the company completed a $25 million series A round led by Pitango First. AI21 raised $64 million in the next round of funding. In January, AI21 Labs launched Wordtune Spices and Jurassic-2 in March 2023. The Jurassic-1 model by AI21 Labs generates human-like texts and performs complex tasks like question answering, text classification, and others. The Jurassic-1 model comes in two sizes. Jurassic-1 Jumbo contains 178 billion parameters. The model uses a unique 250,000 token vocabulary and includes multi-word tokens, reducing the model's need to use a large number of tokens and thus improving the computational efficiency and reducing latency. Jurassic-1 allows developers to train custom versions of the model with just 50100 training examples helping users to build customized applications and services. Jurassic-1 has been notably used by Latitude to scale production of its gaming world, by Harambee to create a custom chatbot to increase sign-ups for its youth employment programs, and by Verb to build a writing tool for authors. The next iteration of Jurassic (Jurassic-2) is a highly customizable language model. It has comprehensive instruction tuning on proprietary data, which gives it advanced instruction following capabilities. The model supports languages like Spanish, French, German, Portuguese, Italian, and Dutch. Compared to the Jurassic-1 model, it has up to 30% faster response time, significantly reducing latency. Jurassic-2 has three sizes, with each one having a separate instruction-tuned version - Large, Grande, and Jumbo. Jurassic-2 helps users to build virtual assistants and chatbots and helps in text simplification, content moderation, creative writing, etc. Jurassic-2 also has zero-shot instruction capabilities. The model boasts of the most current knowledge and up-to-date database, with training being based on data updated in the middle of 2022, as compared to ChatGPT, which had closed its database by the end of 2021. Jurassic-2 comes with five APIs built for businesses that want specifically tailored generative AI features. The APIs include tools for paraphrasing, summarizing, checking grammar, segmenting long texts by topic, and recommending improvements. On Stanford's Holistic Evaluation of Language Models (HELM), Jurassic-2 Jumbo ranks second with an 86.8% win rate. Jurassic-2 is available for free till May 1st, 2023.


10. Baidu's ERNIE Model Baidu, based in Beijing, is a prominent Chinese company that specializes in artificial intelligence. In 2019, Baidu launched a powerful AI language model named Ernie (Enhanced Representation through Knowledge Integration), which has been open-sourced along with its code and pre-trained model based on PaddlePaddle. Since its inception, Ernie has undergone significant improvements and can now execute a diverse array of tasks, such as language comprehension, language generation, and text-to-image generation. ERNIE was designed to enhance language representations by implementing knowledge masking strategies, such as entity-level masking and phrase-level masking. Baidu launched ERNIE 2.0 in July 2019, which introduced a continual pre-training framework. This framework incrementally builds and learns tasks through constant multi-task learning. ERNIE 3.0 was unveiled in early 2021 and introduced a unified pretraining framework that allows collaborative pretraining among multi-task paradigms. Unlike other models such as GPT-3, ERNIE 3.0 showcased task-agnostic zero-shot and few-shot learning capabilities and could be easily tailored for natural language understanding and generation tasks with zero-shot learning, few-shot learning, or fine-tuning. In late 2021, Baidu released ERNIE 3.0 Titan, a pre-training language model with 260 billion parameters that were trained on massive unstructured data. Baidu developed ERNIE Bot, its latest large language model (LLM), and generative AI product. It is designed to serve as a foundational AI platform that can facilitate intelligent transformations in various industries, including finance, energy, media, and public affairs. Access to ERNIE Bot is currently limited to invited users, with the API expected to be available to enterprise clients through Baidu AI Cloud after application (as of March 16th). Baidu aims to use the capabilities of ERNIE Bot to revolutionize its search engine, which holds the dominant position in China. Moreover, it is anticipated that ERNIE Bot will improve the operational efficiency of various mainstream industries, including cloud computing, smart cars, and home appliances.


Hardware and Cloud Platforms Nvidia's H100 Tensor Core, their ninth-generation data center GPU, contains 80 billion transistors and is optimized for large-scale AI and high-performance computing (HPC) models. The A100, Nvidia's predecessor to the H100, is one of the best GPUs for deep learning. There is also Google's Tensor Processing Units (TPUs) which are custom-designed accelerator application-specific integrated circuits (ASIC) used for efficient machine learning workloads and are tightly integrated with TensorFlow, Google's machine learning framework. Google Cloud Platform has opened availability of TPU v4 on Cloud, specifically designed to accelerate NLP workloads, and has also developed TPU v5 for use internally. Microsoft Azure also offers GPU instances powered by Nvidia GPUs, such as the A100 and P40, that can be used for various machine learning and deep learning workloads. Another key development is the partnership between Microsoft Azure and OpenAI, which has given OpenAI the resources to train both GPT-3 and GPT-4 that resulted in the availability of these models for developers in their applications through Azure's cloud infrastructure. AWS provides access to GPUs such as the Amazon Elastic Compute Cloud (EC2) P3 instances, which offer up to 8 Nvidia V100 GPUs with 5,120 CUDA cores and 300 GB of GPU memory. AWS has also developed its own chips for inference(Inferentia) and training (Trainium). Several advanced models have been developed on these computing and cloud systems, including BERT, RoBERTa, Bloom, Megatron and the GPT family. BERT is one of the first pre-trained models that incorporated transformer architecture and resulted in state of the art scores in many NLP tasks. RoBERTa is a variant of BERT, trained on a much larger dataset with a more efficient training procedure. Lastly, Bloom is an open-access multilingual language model, containing 176 billion parameters and was trained on 384 A10080GB GPUs. The increasing availability of specialized hardware for NLP tasks represents a significant development in cloud computing programs. With the availability of these tools, companies can now train and run models that were previously impossible to build.


A note on Open Source Open-source LLMs efforts have been progressing, both in terms of open data sets and open source models available for anyone to fine tune and use. The overall potential of open source models are very promising. They provide a more in-depth access to LLMs for everyone, not just by using an API. However there are definitely questions on the increased risks of models that haven't been aligned - and are more flexible to adapting for nefarious use cases such as misinformation. AI efforts like Eleuther's "The Pile" and LAION's LAION-5B dataset have facilitated rapid progress in text and image modeling. Many companies and groups are also making foundational models accessible with open-source data sets, such as Big Science's Bloom model and the strategic partnership between Hugging Face and Amazon Web Services (AWS), which increases the availability of open-source data sets and models hosted on Hugging Face. Stability AI also supports EleutherAI's work studying Large Language Models, while Laion's project involves crowdsourcing annotations for its OpenAssistant ChatGPT replication project. Additionally, Carper has developed open-source RLHF workflows ranging from human annotation with CHEESE to do RLHF training using trlX package.


Generative AI applied to other modalities By some measures, consumer facing Generative AI has become the fastest growing technology trend of all time, with various models emerging for image, text, and code generation. For example, MidJourney's Discord has attracted around 13 million members for Image Generation, while ChatGPT has reportedly gained over 100 million users within a few months of release. Software development use cases have also seen a significant rise with over 1.2 million developers using GitHub Copilot's technical preview as of September.


1. Image Generation: Dall-E  MidJourney  Stable Diffusion  DreamStudio The combination of models, data, and computing has provided an incredible set of tools for working with images. OpenAI's DALL-E is an AI system that uses deep learning and transformer language models to generate digital images from natural language descriptions. It employs a decoder-only transformer model that models text and images as a single data stream containing up to 256 tokens for text and 1024 for images. The neural network then autoregressively models them. DALL-E is a 12-billion parameter version of GPT-3. The model uses a causal mask for text tokens and sparse attention for image tokens. DALL-E 2 is capable of producing higher-resolution images and uses zero-shot visual reasoning. It can create anthropomorphized versions, fill in the blanks, and transform existing images. However, DALL-E uses public datasets as training data, which can affect its results and often leads to algorithmic biases. Midjourney is an artificial intelligence program developed by Midjourney, Inc., an independent research lab. The platform uses natural language descriptions to generate images, and users can create images by using Discord bot commands on the official Discord server. On March 16, 2023, beta version 5 was released. Users can generate images by typing the /imagine command followed by the prompt, and the bot generates four images, from which the user selects the image they want to upscale. Midjourney Inc. is also developing a web interface. Stable Diffusion is an open source image model funded by Stability AI that generates images from text and performs tasks like inpainting, outpainting, and generating image-to-image translations. It uses a latent diffusion model supported by EleutherAI and LAION. It requires a minimum of 8GB VRAM making it independent of needing cloud services. Stable Diffusion 2.0 was released in November 2022 and trained on pairs of images and captions from LAION-5B and its subsets. DreamStudio is the official online implementation and team interface API for Stable Diffusion, developed by Stability AI. DreamStudio and Stable Diffusion have slightly different interfaces even as they are applications of the same technology. The web app was launched in August 2022, replacing the free Discord bot. The web app offers better functionality and stability, using the Stable Diffusion algorithm to generate images based on the user's prompt. DreamStudio API Access has an access fee. One of the key features of DreamStudio is its support for negative prompting. It also allows users to overpaint, copy, modify, and distribute images for commercial purposes.


2. Audio Generation: Whisper  AudioGen  AudioLM Whisper, developed by OpenAI, is a versatile automatic speech recognition system that supports multilingual speech recognition, speech translation, and language identification. It has been trained on 680,000 hours of multilingual and multitask supervised data using Python 3.9.9 and PyTorch 1.10.1, and the codebase is expected to be compatible with Python 3.83.10 and recent PyTorch versions. It deploys an encoder-decoder transformer model that uses 30-second chunks of input audio converted to log-Mel spectrograms, which are then passed to an encoder. The decoder predicts the corresponding text caption and intermixes special tokens to perform various tasks. Whisper provides an open-source model and inference codes for speech processing research and new application development. With nearly one-third of its dataset being non-English, Whisper outperforms the supervised state-of-the-art on CoVoST2 to English translation zero-shot. Google's AudioLM is a pure audio model that uses language modeling to generate high-quality audio without annotated data. It generates speech continuations that preserve the identity, prosody, and accent of the speaker and recording conditions, and can also generate coherent piano music continuations. The model demonstrates long-term consistency in syntax, harmony, rhythm, and melody, and has the potential for extension to multilingual speech, polyphonic music, and audio events. AudioLM uses a hybrid tokenization scheme and a SoundStream neural codec to improve fidelity. The model achieved a 51.2% success rate from human raters and an audio classifier with 98.6% accuracy was trained to detect synthetic speech generated by AudioLM. Currently, AudioLM is only available for research purposes and is not publicly available. Meta's AudioGen AI converts text prompts into audio files. It is the audio parallel of image-generating AI like DALL-E. It uses a language AI model and approximately 4000 hours of training data to generate ambient sounds, sound events, and their composition. Additionally, it can extend existing audio to create rudimentary music. The quality of the audio output has been rated at 70% via Amazon's Mechanical Turk platform. However, AudioGen currently cannot sequence sounds through time, and the ownership rights of the generated audio are unclear.


3. Search Engines: Neeva  You Neeva is an AI-powered search engine that provides ad-free and private searches. It achieves this through its in-house LLMs and search stack, while also blocking third-party website trackers and not sharing user information. Neeva's unique feature is its AI summaries, which provide synthesized answers backed by cited authority. It also allows users to search personal email accounts, calendars, and cloud storage platforms. This feature combines the best aspects of LLMs, like ChatGPT, with authority and timeliness. However, it only functions with question queries and has limitations on the free version (the premium plan is priced at $4.95/mo). Neeva has over 2 million users and local language versions in Germany, France, and Spain. You.com is a California-based search engine that uses multimodal conversational AI to group web results into website categories sorted by user preferences. It was launched for public beta in November 2021 with a focus on privacy and personalization. It offers YouWrite, a text generator, and YouChat, a chatbot with community-built apps and blended LLMs. You.com does not collect users' personal information and offers personal and private search modes. The search results allow users to create content directly from the search results, building trust and reliability.


4. Code Generation: Copilot  Codex GitHub Copilot is a tool that assists developers in programming by using AI to convert natural language into coding suggestions. It is powered by OpenAI Codex, which allows it to understand the developer's coding style and suggest context-specific solutions. When developers input their desired logic into the system, GitHub Copilot can generate code suggestions automatically. However, it is important to note that these suggestions are just that, suggestions, and it is up to the developer to decide whether to use them or not. OpenAI Codex is a natural language processing model that is based on GPT-3 and can generate working code in multiple programming languages such as Python, JavaScript, and Ruby, among others. To train Codex, billions of lines of source code from public sources, as well as natural language data, including code from GitHub repositories, were used. It has a memory of 14KB for Python code and is a powerful, transformer-driven system that can effectively and efficiently fulfill developers' tasks.


5. Text Generation: Jasper Jasper.AI is a subscription-based text generation model that requires minimal input from the user and searches the web to generate the desired output. It is particularly useful for generating short copy text where character limitations are important. The platform offers over 50 templates, including product descriptions, email subject lines, and Facebook headlines, among others. Additionally, it can help with generating ideas for blog posts and creating better outlines. However, Jasper.AI does have some drawbacks, such as the absence of fact-checking and citation of sources, which can lead to hallucinations. Additionally, learning the command input to achieve the desired output may take some time.


Conclusion Generative AI is a revolutionary technology that has the ability to transform many aspects of our lives. Keep in mind that there are still challenges in developing these models such as massive datasets, compute power, high training cost, and accessibility. Studies have revealed that many large language models are not adequately trained. Additionally, smaller datasets are still crucial for enhancing LLM performance in domain-specific tasks. Compute cost optimization is also essential since generative models, especially large language models, are still expensive to both train and serve for inference. Big players in the industry are working on optimizing compute costs at every level. Safety and security remain pressing concerns in the development of generative AI, and key players are incorporating human feedback to make the models safer from the outset. Open-source alternatives are also necessary to increase access to the next-generation LLM models for practitioners and independent scientists to push the boundaries forward.


Diffusers: Python Library for AI-Generated Images

Introduction The Diffusers library maintained by HuggingFace is a go-to library for Generative AI that provides multiple stable diffusion pipelines for images, audio, and several other useful functionalities. SD-WebUI is deployed on Automatic1111 and is available on GitHub for people who prefer GUI. However, for deployment purposes, GUI may not be the best option, and people have been using the diffuser library to deploy full-blown Generative AI applications. This article will showcase the setup and usage of a diffuser library for generating images using Stable Diffusion.


Pre-requisites Firstly, you need to set up a fresh environment for the use of the diffuser library. A fresh environment is not a necessity but it helps avoid dependency clashes between pre-installed packages and versions required by diffusers and associated libraries. You can use either a Python virtual environment or a Conda environment. In the new environment, run the following commands to set up the required packages. For documentation and codebase, refer to the GitHub links for Diffusers and Transformers library. Both are currently maintained by HuggingFace, with new functionality and updates being pushed frequently.


Code Basic text-to-image pipelines are provided by the Diffusers library. Other available pipelines, including Image-to-Image, Inpainting, and ControlNet, are used similarly. For this article, we will focus on the basic Stable Diffusion text-to-image pipeline.


Setup Hyperparameters and Constants Device is used to select hardware devices. If an Nvidia GPU is detected in the system, it will be selected, which can speed up inference. In case a GPU is not available, inference can still be done on CPU hardware. The prompt is the textual prompt that will be passed to the pipeline. MODEL_ID is the pre-trained model that will be fetched from HuggingFace. Multiple models are provided for Stable Diffusion available for use on HuggingFace Model Hub. Stable Diffusion 2.1 is the most recent release that provides 768x768 output results. Other models available are Stable Diffusion 1.5, and 2.0. Moreover, other fine-tuned models are also available that are trained for specific styles, such as anime or realistic images.


Initialize Pipeline We use torch float16 precision instead of the default float32. Using Half precision, we can reduce GPU utilization that provides efficient inference. The from_pretrained method fetches the pre-trained model from HuggingFace. It downloads and caches all required modules such as Text Encoder, Unet, and Variational AutoEncoder, and returns a StableDiffusionPipeline object. We then convert to pipe to the dedicated hardware that is to be used for inference. It will be either GPU or CPU.


Inference We pass the required parameters for the forward call of the StableDiffusionPipeline object. It returns an object of type StableDiffusionOutput that is built in the diffusers library. It contains a list of generated images. For our use case, we only fetch the first generated image. The num_inference_steps argument sets the total denoising steps used. A higher number provides better results as the Unet can denoise an image for longer. The guidance_scale argument controls the prompt conditioning on the output image. A lower guidance scale means the model pays less attention to the prompt, so the output may not reflect the prompt provided. However, the model has more creative freedom so the generated image can showcase more variations. A higher guidance scale focuses more on the prompt provided. The generated image is a PIL Image object. So the Pillow library can be used to save or post-process the image further.


Output The result is a 768x768 dimension image based on the prompt provided. The exact dimension is based on the default size the pre-trained model was trained on. However, the dimension can be changed by using the height and width keyword argument during inference.


Conclusion The article highlighted the basic usage of the diffusers library that can be used for the deployment of applications related to Generative AI. There are multiple other pipelines available for different use cases. The API of each pipeline is similar, with simple changes in inference arguments making the required changes.


Gorilla: Everything You Need to Know

Introduction LLMs suffer from outdated information and they require re-training to keep up-to-date with recent changes. With limited context and weights, LLMs cannot store data for accurate responses. Therefore, LLMs are augmented by the use of numerous tools and plugins that use external APIs for better answers. Gorilla introduces self-instruct fine-tuning and retrieval training on a large corpus of APIs that provides better results than leading LLMs, including but not limited to ChatGPT4 and Claude.


Gorilla: Everything You Need to Know

Dataset The authors created APIBench, an exhaustive set of API corpus, scraped from the three major model hubs. TorchHub: 94 API callsTensorHub:696 API callsHuggingFace: Chose only the 20 most downloaded models totaling 925 API calls. Then using Self-Instruct, the authors used GPT 4 to generate realistic prompts that use the APIs and constructed 10 different instruction-API pairs to finetune LLMs.


Gorilla: Everything You Need to Know

Finetuning Gorilla is a LLaMA-7B fine-tuned model on the APIBench dataset. The authors use a chat-style training where each API call is used, similar to a real-world conversation between a user and a chatbot. The LLaMA model is then finetuned with and without a retriever.


Gorilla: Everything You Need to Know

API Call with Constraints Consider the following prompt: LLMs face issues when generating responses for such prompts, as the user adds constraints on an API call and its associated parameters.


Gorilla: Everything You Need to Know

Retreiver-Aware Training In certain scenarios, the model is constrained by a specific API call or documentation that can degrade responses provided by the LLMs. Consider the following prompt: During finetuning, Gorilla aims to parse the second half to fetch the required API specification to answer the first part of the question.


Gorilla: Everything You Need to Know

Inference Gorilla can be used in two different modes, similar to the two methods mentioned above. A user provides a prompt in natural language, and it is parsed by Gorilla in two ways. Zero-shot passes the same prompt with no additions to finetuned Gorilla LLM, which returns an API response. In Retrieval-mode, Gorilla uses GPT-Index or BM25 to first retrieve the most appropriate API documentation, which is added to the user prompt. The message Use this API documentation for reference: is concatenated with the user prompt before sending it to Gorilla LLM. The methodology is summarized by the following image: The authors used the large API data corpus to generate 16450 prompts, and response pairs generated using self-instruct. These were used to finetune the Gorilla LLM. At inference time, the image highlights the two inference methods, Zero-shot, and Information retrieval methods. When using zero-shot, the prompt is passed as is to the Gorilla LLM without any preprocessing. When using the information retrieval method, the natural language prompt is sent to a retriever that compares the user prompt with an existing API database. Relevant API documentation is returned and concatenated with the user prompt before sending it to Gorilla LLM.


Gorilla: Everything You Need to Know

Results The image from the paper highlights the improvements of Gorilla over other LLM models. The paper summarizes the results of evaluating Gorilla on the collected dataset with different retrieval methods. The model performs better than other LLMs when queried with API-specific prompts. The graphical illustration highlights such results:


Gorilla: Everything You Need to Know

Reduced Hallucinations Hallucinations refer to code generated with the wrong API or an API that does not exist. This can lead to runtime errors and non-functional code. Gorilla is specifically finetuned on a large corpus of API calls reducing the chances of hallucination errors. An example is provided in the results section of this article.


Gorilla: Everything You Need to Know

Improved Code Generation With more structured responses that use the most up-to-date API documentation fetched at runtime, the responses are better suited for code generation. This can save a lot of cost and time in debugging generated code, as the responses are dependable.


Gorilla: Everything You Need to Know

Usage The code is open-source and can be accessed on GitHub. For inference, the Gorilla can be accessed using the provided Colab notebook or accessed with a Command Line Interface through code. First, create a new environment and install all dependencies. Finetuned delta weights for Gorilla are available on HuggingFace. Original LLaMA weights are available here. After downloading weights, apply the weights to the original LLaMA model using the following code: The model can then be run with the given command:


Gorilla: Everything You Need to Know

Conclusion The article introduced Gorilla LLM, the LLaMA model specifically finetuned for API calls. It provides better responses than GPT-4 on all 3 datasets collected and is better suited to adapt to run-time API usage changes. The correct usage of Gorilla can improve the performance of LLMs using a wide variety of tools and plugins.


This AI newsletter is all you need #53

Our in-depth practical Generative AI course has reached thousands of sign-ups! We launched our practical Generative AI course last week and we are excited with the positive reception. Thousands of people have already signed up for the course and some people are already putting their new skills into practice, deploying their models on behalf of customers! This course is the first of many by Towards AI and builds on our four-year history of providing educational and tutorial AI content. This first course was released in collaboration with Activeloop and the Intel Disruptor Initiative. The course focuses on LangChain and Deep Lake, a vector database for all AI data in production, and stands out with its depth and its focus on practical skills. With over 50 lessons including 10 practical projects across 8 modules and 5 full days of study time, the course offers a deep dive into AI technology, guiding you through creating innovative and production-grade AI tools and solutions. Enroll today for free at learn.activeloop.ai.


This AI newsletter is all you need #53

What happened this week in AI by Louie This week in AI developments, we highlight the acquisition of MosaicML by Databricks which we think demonstrates the rapid potential speed of startup maturity in AI as well as the potential for open-source models for LLMs. Databricks, a data and AI company, announced its definitive agreement to join forces with MosaicML, a leading generative AI platform. This collaboration aims to democratize generative AI and empower organizations to build, own and secure generative AI models with their own data. MosaicML is behind leading open source LLMs such as MPT-7B and MPT-30B, and is well known for its cost effective model training. The transaction was valued at approximately $1.3 billion and stands as a testament to the rapid pace of AI startups as the company has achieved an exit status just 2.5 years after it was founded and after raising just ~$64 million in capital. This acquisition follows recent significant AI startup funding rounds, such as Adept AI raising $350 million and Claude raising $450 million, and further emphasizes the substantial investments in this space and amount of capital being invested to remain competitive. This acquisition also underscores how a small number of individuals can create wealth in the AI paradigm, as demonstrated by MosaicML with its 62 employees. We believe Databricks' acquisition of MosaicML serves as validation for the open-source model of LLMs and where companies may choose to train their own models using customer-specific data to maintain full understanding of the model training set and control of deployment. - Louie Peters - Towards AI Co-founder and CEO


This AI newsletter is all you need #53

Hottest News Introducing Voicebox: The first generative AI model for speech to generalize across tasks with state-of-the-art performance Meta AI has created Voicebox, a new model that employs a Flow Matching approach to train on vast and varied datasets. This enables it to produce top-notch synthesized speech without the need for specific training. The model can match various audio styles, read text passages in multiple languages, and even edit speech segments within audio recordings. While the research paper and audio samples are accessible, the model and code are kept private to prevent misuse. 2. ElevenLabs Launches New Generative Voice AI Products and Announces $19m Series A Round ElevenLabs, a voice technology research company and global leader in audio AI software, has secured $19 million in Series A funding for their AI-powered synthetic voice technology. This technology empowers content creators to effortlessly manage AI-generated audio content. The company has further plans to launch an AI dubbing tool and an AI Speech Classifier to ensure transparency and safety in generative media. 3. RoboCat: A Self-improving Robotic Agent DeepMind's AI agent, RoboCat, can quickly adapt and enhance its skills with various robotic arms by generating new training data. In just 100 demonstrations, it can learn to operate new robotic arms within hours. The latest version of RoboCat has significantly improved, with its success rate on novel tasks more than doubling compared to its initial version, thanks to its growing experience. 4. Microsoft advances state of the art for LLM training and serving DeepSpeed is an algorithm and system employed in training massive open models. To overcome existing limitations, Microsoft has introduced ZeRO++ with enhancements in memory, throughput, and usability. 5. Amazon's Vision: An AI Model for Everything Amazon aims to establish a unified gateway for businesses to access both open-source and closed-source generative AI models. In an interview with Matt Wood, AWS VP of Product, he delves into Amazon's perspective on the AI market, their strategy to outperform other tech giants in the AI competition, the future of the internet, and more.


This AI newsletter is all you need #53

Five 5-minute reads/videos to keep you learning LangChain & Vector Databases in Production Course Last week, we announced our collaboration with Activeloop and the Intel Disruptor Initiative to create LangChain's free AI course, "LangChain & Vector Databases in Production." The course is designed to make AI practical and accessible, tailored to both experienced developers and enthusiasts. It offers 50+ lessons and 10+ projects covering API integration, prompt engineering, and production use. 2. Market Map and Analysis: Gen AI Search Companies Generative AI firms are enhancing their services to compete with Google, Microsoft, and Baidu. The article discusses the leading AI search companies, such as You.com and Perplexity.ai, in the consumer search space, and Vectara, Dashworks, Nuclia, Metaphor, and Hebbia in the enterprise space. 3. Emerging Architectures for LLM Applications The article introduces a reference architecture for the LLM apps stack using in-context learning. It consists of three parts: data preprocessing and embeddings, prompt construction and retrieval, and prompt execution. In-context learning simplifies AI and is particularly useful for smaller datasets, enabling real-time data incorporation. 4. How GPT works: A Metaphoric Explanation of Key, Value, Query in Attention, using a Tale of Potion The article explains the concept of Key, Value, and Query in Attention using a potion metaphor to illustrate their role in predicting the next word in a text within the functioning of GPT. It aims to help readers develop a more intuitive understanding of GPT's inner workings from end to end. 5. AI and The Burden of Knowledge Researchers are shifting focus from larger models to making AI models more efficient, considering the high making and deploying costs involved. This could lead to a new era of AI development, where models can achieve superhuman performance without requiring massive amounts of resources.


This AI newsletter is all you need #53

Papers & Repositories Unifying Large Language Models and Knowledge Graphs: A Roadmap This work suggests a roadmap to unify Language Models (LLMs) and Knowledge Graphs (KGs) in AI. It introduces KG-enhanced LLMs, LLM-augmented KGs, and Synergized LLMs + KGs frameworks to enhance knowledge representation and reasoning for improved downstream tasks. 2. Textbooks Are All You Need This paper presents Phi-1, a new coding language model that leverages "textbook quality" data. Despite its smaller size, Phi-1 demonstrates remarkable accuracy on HumanEval and MBPP. It excels in Python coding and outperforms larger models by utilizing high-quality examples. 3. A Simple and Effective Pruning Approach for Large Language Models This paper introduces Wanda, a novel pruning approach for LLMs. Wanda achieves sparsity without requiring retraining or weight updates. It efficiently identifies efficient sparse networks from pre-trained models, surpassing pruning approaches and matching or exceeding the performance of other recent methods while minimizing computational costs. 4. FastSAM: A Faster Alternative to the Segment Anything Model This repository presents a fast method that matches the performance of the recently proposed Segment Anything (SAM) model. SAM is utilized in computer vision for tasks like image segmentation and captioning. However, this new technique employs a standard approach called instance segmentation, delivering comparable results at a speed 50 times faster. 5. Tart: Boosting the Reasoning Abilities of LLMs This study reveals that while LLMs excel in various tasks, they face challenges with probabilistic reasoning, leading to performance limitations. To address this, the paper introduces TART, a solution that enhances an LLM's reasoning capabilities through a synthetically trained Transformer-based reasoning module. Enjoy these papers and news summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #53

Weekly AI Podcast In this week's episode of the "What's AI" podcast, Louis Bouchard interviews Jay Alammar, widely known in the AI and NLP field for his exceptional blog on transformers and attention. They delve into the world of Transformers, AI evolution, training steps of LLMs, and more. Jay shares insights on building LLM Apps and the challenges that accompany them. Tune in to the episode on YouTube, Spotify, or Apple Podcasts.


This AI newsletter is all you need #53

Upcoming Community Events The Learn AI Together Discord community hosts weekly AI seminars to help the community learn from industry experts, ask questions, and get a deeper insight into the latest research in AI. Join us for free, interactive video sessions hosted live on Discord weekly by attending our upcoming events. 1. Reading group on Time Series: conformal prediction & its application in depth In critical domains like medical diagnoses and safety-critical systems, quantifying prediction uncertainty in machine learning is crucial. Conformal prediction offers a robust framework for this purpose. It allows the quantification of uncertainty for any machine learning model as a post-processing layer, without requiring model refitting. Join us for an upcoming talk where we delve into the applications of conformal prediction. Attendees are encouraged to familiarize themselves with the insights shared on the MLBoost YouTube channel before the event. Join the event here and discover how conformal prediction enhances reliable decision-making by providing a measure of uncertainty beyond traditional point predictions. Date & Time: 7th July 2023, 10:00 am EST Add our Google calendar to see all our free AI events!


This AI newsletter is all you need #53

Meme of the week! Meme shared by bigbuxchungus


This AI newsletter is all you need #53

Featured Community post from the Discord Marcklingen has contributed to the development of langfuse, an open-source experimentation platform for LLM-based applications. Langfuse allows users to stay updated on output quality and feature trends, track token usage, segment executions conveniently, compare low-quality executions side-by-side, and more. Check it out on GitHub and support a fellow community member. Share your questions and feedback in the thread here.


This AI newsletter is all you need #53

AI poll of the week! Join the discussion on Discord.


This AI newsletter is all you need #53

Article of the week (Vision) Transformers: Rise of the Chimera by Quadric This article tries to answer questions such as: What makes Vision Transformers (ViT) so special in comparison to their CNN counterparts, why these unique architectural features of ViTs are "breaking" almost all NPU and AI hardware accelerators targeting the high-performance edge market, and how Quadric's Chimera GPNPU architecture can run ViTs today with real-time throughput at ~1.2W.


This AI newsletter is all you need #53

Our must-read articles Graph-Based Learning: Part 1 by Anurag Tangri PatchTST - A Step Forward in Time Series Forecasting by M. Haseeb Hassan Bootstrap: A Beginner-Friendly Introduction With a Python example by Janik and Patrick Tinz 5 Powerful Cross-Validation Methods to Skyrocket Robustness of Your ML Models by Bex T. If you want to publish with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.


This AI newsletter is all you need #53

Job offers Principal AI Engineer, Deep Learning/LLMs @Cynch AI (San Francisco, CA, USA) Senior Python and Machine Learning Developer @Patona (Remote) API Engineer @Move.ai (London, UK) Data Analyst @Kido (Mumbai, India) Lead Data Scientist @Rising Academies (Remote/Freelancer) Data Analyst (Internship) @Infosys (Singapore) Junior ML Engineer @SWORD Health (Remote) Interested in sharing a job opportunity here? Contact sponsors@towardsai.net. If you are preparing your next machine learning interview, don't hesitate to check out our leading interview preparation website, confetti!


Can ChatGPT beat DeepPavlov in Natural Language Understanding tasks?

Introduction ChatGPT, which has been released by OpenAI, has revolutionized the way artificial intelligence interacts with humans. Although there is limited information available on how the model was constructed, its exceptional quality has made it extremely popular. It is noteworthy that the base model of ChatGPT only has 3.5B parameters, yet in conversation tasks, it outperforms the GPT3 model with 175B parameters. This shows the high relevance of collecting data from humans for supervised model fine-tuning. There are several publications on ChatGPT evaluation that test its effectiveness on well-known natural language processing (NLP) tasks. In this article, I am going to compare the performance of ChatGPT with that of the supervised transformer-based models from the DeepPavlov library. DeepPavlov is a conversational open-source library for Natural Language Processing (NLP) and Multiskill AI Assistant development. DeepPavlov is based on PyTorch and leverages Transformer and Datasets packages from HuggingFace to train various transformer-based models on hundreds of datasets.


Install DeepPavlov Library DeepPavlov Library is an open-source NLP framework. It contains all essential state-of-the-art NLP models that can be used alone or as a part of DeepPavlov Dream - an open-source Multiskill AI Assistant Platform. The library contains various text classification models for topic classification, insult detection, and intent recognition. Before comparing the models, let's first install DeepPavlov Library by running the command: The DeepPavlov Library supports Python 3.6 - 3.9 .


How to use DeepPavlov Library The DeepPavlov framework is implemented in Python and uses PyTorch as the base machine learning framework. DeepPavlov supports HuggingFace transformers which allows using all AutoModel transformer-based models from the HuggingFace Hub. DeepPavlov provides a way to interact with models via a command-line interface (CLI), an application programming interface (API), or via Python pipelines. The list of the CLI - supported commands is as follows: install model's specific requirements, run python -m deeppavlov install <config_name>, where <config_name> is the name of the configuration file.interact with the model through CLI, run python -m deeppavlov interact <config_name> [-d] [-i], where -d downloads files from the metadata of the configuration file (optional), and -i installs model requirements (optional).train the model, run the following command: python -m deeppavlov train <config_name> [-d] [-i]. The dataset will be downloaded regardless of whether there was the -d flag or not. To train a model on your own data, you need to modify dataset_reader path in the model configuration file. The data format is specified in the corresponding model doc page. In order to change the backbone transformer architecture, you should modify the corresponding variable in the variables section.riseapi starts a REST API server with the selected model, which might be useful in production deployment scenarios, run python -m deeppavlov riseapi <config_name> [-d] [-i]. Alternatively, you can interact with DeepPavlov's models by using Python pipelines.


Comparing ChatGPT and DeepPavlov There is no easy way to compare ChatGPT with DeepPavlov-based models. You have to keep in mind that ChatGPT performance is conditioned on the prompt. Even the slightest modification of the prompt might change the result completely. Almost all prompts used here were adopted from [2] to be fully comparable with the paper results. Moreover, testing ChatGPT performance on well-established NLP benchmarks might lead to several problems. Evaluating ChatGPT performance on thousand test examples might be long and expensive. Therefore I decided to either evaluate the GLUE test set of the corresponding task if the size of the test set is less than 1000 samples or follow the approach proposed in [2] by evaluating a sample of the dev set and comparing it with ChatGPT. In the following comparisons, I included the ChatGPT performance, BERT-based model performance (from [2]), and of DeepPavlov-based models. It is worth mentioning that according to [1], ChatGPT might remember many of the evaluation datasets from the pre-training phrase. OpenAI is continuously improving ChatGPT by releasing new versions. Since the initial release on Dec 15, 2022, there have been seven major releases. All results mentioned here are based on the ChatGPT version of March 23, 2023.


Recognizing Textual Entailment (RTE) The Recognizing Textual Entailment (RTE) task is a natural language processing task that involves determining whether a given piece of text (the hypothesis) can be inferred from another piece of text (the premise). The task requires understanding the meaning and context of the text and making a judgment about the logical relationship between the two pieces of text. RTE is often used in applications such as question answering, information retrieval, and text summarization. For example, given the sentence "Cyprus, divided or not, joins the EU on the 1st of May." determine if the following statement is entailed: "Cyprus was divided into two parts on May 1." The correct answer is false (not entailment). To access the pre-trained model from the DeepPavlov library, one should use: To test ChatGPT performance, I used the following prompt: "Given the sentence '[text_1]', determine if the following statement is entailed: '[text_2]'". The RTE test set contains thousands of examples. So to compare ChatGPT with DreepPavlov RTE pretrained models, I used the same 25 sampled instances for each class from the dev set as in [2]. On this sampled set DeepPavlov's RTE model gains an accuracy of 90, whereas ChatGPT is only 88. Here are three examples where ChatGPT fails to correctly label the pair of sentences:


Winograd NLI (WNLI) ChatGPT has difficulty with coreference resolution that deals with resolving ambiguous pronouns. Winograd Schema Challenge assesses a machine learning model's capability to refer pronouns to the corresponding noun. It involves resolving ambiguous pronouns in a sentence by using the context provided in the preceding sentence. The challenge is considered difficult for NLP models as it requires a deep understanding of natural language and context. The WNLI task represents the mix of natural language inference with coreference resolution. To access the pre-trained model from the DeepPavlov library, one should use: To solve this problem using ChatGPT, the query based on [2] was used as input: "Having the premise ['sentence1'], tell me if the following hypothesis ['sentence2'] is entailed with the premise? Answer 'entailment' if yes, or 'not_entailment' if no". Mainly, ChatGPT manages to choose the correct answer. But when it needs to determine which person is substituted for the pronoun or the solution involves the application of physical laws and concepts, ChatGPT incorrectly determines the answer option based on the given context. Primarily, ChatGPT is able to select the accurate response. However, sometimes it struggles in logical or physical reasoning to select the answer based on the provided context. The unlabeled test set of WNLI contains 146 pairs of sequences. Therefore, it's feasible to label the entire test set with ChatGPT and complete a fair comparison between ChatGPT and DeepPavolv pretrained WNLI model. DeepPavlov's WNLI model gains an accuracy of 81.5, whereas ChatGPT is only 79.5. Here are some examples where ChatGPT fails to label the pair of sentences correctly:


Word in Context (WiC) One more task where I tested the work of ChatGPT is the word sense disambiguation (WSD) task and the WIC dataset. The Word-in-Context (WIC) dataset is a benchmark dataset used to evaluate the ability of machine learning models to determine the correct sense of a word in a given context. It consists of pairs of sentences where the target word has different senses, and the task is to predict whether the word has the same sense in both sentences or not. The dataset is designed to evaluate models' ability to capture contextual information and make accurate sense predictions. As the ChatGPT input I used the following query from [2]: "Are the given sentences expressing the same sense of the word ['verb/noun']: ['sentence1'] and ['sentence2']?" According to the results in [1], the SOTA accuracy exceeds the ChatGPT accuracy by 9.4: 74.00 vs. 64.58. And our experiments confirm this: ChatGPT doesn't solve this task well if a word in two different sentences has the same meaning. In this case, ChatGPT uses synonyms to explain the meanings of the studied word in each sentence. And sometimes, these synonyms are different, which leads to the wrong decision. Here are some examples of mistakes: