Improving Language Understanding by Generative Pre-Training GPT, or Generative Pre-trained Transformer, was published by Alec Radford from OpenAI in 2018. This work very cleverly and optimally used all the findings of the previous development and merged them with a massive parameterized model and well-curated training data, to achieve huge performance increases. The aim of GPT is to learn a universal representation that transfers with little adaptation to a wide range of tasks. It is another transformer decoder-based, auto-regressive model, that was trained using a two-step method. The first step is an unsupervised pre-training phase, followed by task-specific fine-tuning. The authors have used the methods suggested by ULMFit to fine-tune the models to achieve better performances. Pre-training: The authors have broken down the pre-training document sequence, into long single vector sequences, and pre-trained using language modeling objectives. The length of the sequences is maximized as far as processing power supports under the claims that using longer sequences trains the model to handle long-term dependencies. Also, the authors have added some auxiliary goals, like POS tagging, chunking, and named entity recognition, during pre-training, which have helped by (a) improving the generalization of the supervised model, and (b) accelerating convergence. 2. Fine-Tuning: For fine-tuning, the authors have used a traversal model, which means they have attached the input and output on a single vector; for the tasks which require generations and for the tasks like similarity comparison, pairs of sentences are concatenated separated by a unique separator character. These are the ways the fine-tuning is arranged. Model Specifications and Data: To pre-train the model using language modeling, the authors have used the BooksCorpus dataset for training the language model. It contains over 7,000 unique unpublished books from a variety of genres, including Adventure, Fantasy, and Romance. This section describes the model architecture. I am directly quoting from the paper to avoid discrepancies and it is pretty simple to understand with the knowledge of the real transformer architecture. Link: https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf


Modern NLP: A Detailed Overview. Part 2: GPTs

The Improvements Over The Years Till now we have discussed the working of the first GPT model. Since then, OpenAI has been working to improve the performance, performing numerous tweaks and using the improvements in the big data world.


Language Models are Unsupervised Multitask Learners This paper was released by OpenAI in 2019, which unveiled GPT-2 as a successor model of GPT, the basic version. OpenAI released a paper and a smaller model compared to the actual model, which they didn't release as they expected people to misuse the model's capabilities. OpenAI said that the sole aim of this model is its generative capability improvement, that is, it can predict the next words optimally. The authors observed all the state-of-art models followed a semi-supervised learning approach but were very brittle and sensitive to the target task, that is, they will diverge if trained to do a different target task. To mitigate this, the authors wanted something more generalized. The authors basically wanted to set up a Zero-shot learning setting for downstream tasks, which means no extra dataset should be required for fine-tuning, and it will be completely unsupervised. Multitask learning or meta-learning supposedly emerged as a stable framework to achieve such a goal. The works on the topic have achieved decent performance on 17 tasks using an amalgamation of 10 datasets. The authors realized that the difference between specific task learning and multi-task learning is basically the probability the model has to predict, which is P(output/input) for a specific task learning but P(output/input, task) for multitask learning as it might vary. From different task-conditioning works (that is, when a single objective task is represented as a chain of different target tasks, and the model is tested to perform on them), it was observed that the model performed best if the global minima of the unsupervised pre-training task that is Language modeling, coincides with the global minima of the supervised task of fine-tuning, so the ultimate target was to optimize the unsupervised task and find the best global minima. It was established from ablations that this can be achieved given a sufficiently large dataset. Dataset: The authors leveraged the knowledge from a web scraper named Common Crawler used for similar work and devised their own crawler to create a dataset named WebText, containing the text subset of these 45 million links, 8 million text documents resulting in 40GB of data. The authors also used BPE or Byte-Pair Encoding, which draws a line between word-level and character-level encoding to enable the model to train on any type of data. Model Modifications: The authors slightly modified the transformer decoder block to add a normalization layer before the block as a kind of pre-activation and another normalization and residual connection after the final self-attention layer. The model overall contained 1.5 billion parameters and was ten times more than the GPT-1 model. Finding: Pre-Training this huge model on a huge amount of generic data like Wikipedia extracts, the researchers have observed the generative model works fairly well without any sort of fine-tuning. These gave rise to the concept called Zero-Shot Learning, as the models do not require any amount of labeled dataset to train on and can still deliver performance. Paper link: https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf


Conclusion In recent years, OpenAI has developed GPT3 and much more using several concepts like multitask learning which is another story to tell, as this is already too long. In this article, we have seen the evolution of the NLP world till GPT-2. The further developments we will see in the next articles of the series. [Update] The next part of this series talks about BERT. Please visit this page, if interested. Happy Reading!


Logistic Regression: Intuition & Implementation

Introduction Logistic Regression is a fundamental binary classification algorithm that can learn a decision boundary between two different sets of data attributes. In this article, we understand the theoretical aspect behind the classification model and implement it using Python.


Dataset Logistic Regression is a supervised learning algorithm so we have data feature attributes and their corresponding labels. The features are independent variables, denoted by X, and are represented in a one-dimensional array. The class labels, denoted by y, are either 0 or 1. Therefore, Logistic Regression is a binary classification algorithm. If the data has d-featuers:


Cost Function Similar to Linear Regression, we have associated weights for the feature attributes and a bias value. We have a weight value associated with each feature attribute and a single bias value. We randomly initialize these values and optimize them using Gradient Descent. During training, we take a dot product between weights and features and add the bias term. But because our target labels our 0 and 1, we use a non-linear Sigmoid function, represented by g, that pushes our values between the required range. The sigmoid function plotted is as follows: To predict values in Logistic Regression, we pass our predictions to the sigmoid function. So, the prediction function is: Now that we have predictions from our model, we can compare them with the original target labels. The error between the predictions is calculated using the Binary Cross Entropy Loss. The loss function is as follows: where y is the original target label, and p is the predicted value between [0,1]. The loss function aims to push the predicted values toward the actual target labels. If the label is 0 and the model predicts 0, the loss is 0. Similarly, if both predicted and target labels are 1, the loss is 0. Else, the model tries the converge to these values.


Gradient Descent We use the Cost function and obtain the derivative concerning weights and bias. Using chain rule and mathematical derivation, the derivatives are as follows: We obtain a scalar value that we can use to update the weights and biases. This updates the values against the loss gradient, so after multiple iterations, we gradually converge toward the optimal values of weights and biases. During inference, we can then use the weights and bias values to obtain predictions.


Implementation We utilize the mathematical formulas mentioned above to code the Logistic Regression model and will evaluate its performance on some benchmark datasets. Firstly, we initialize the class and parameters. We require the weights and bias values that will be optimized, so we initialize them here as object attributes. However, we can not set the size here as it depends on the data passed during training. Thus, we set them to None for now. The learning rate and number of iterations are hyperparameters that can be tuned for performance.


Training The training function initializes the weights and bias values. We then iterate over the dataset multiple times, optimizing these values towards convergence, such that the loss minimizes. Based on the above equations, the sigmoid function is implemented as follows: We then use this function to generate predictions using: We calculate loss over these values and optimize our weights:


Inference Once we have fit our data during training, we can use the learned weights and bias to generate predictions similarly. The output of the model is in the range [0, 1], as per the sigmoid function. We can then use a threshold value such as 0.5. All values above this probability our tagged as positive labels and all values below this threshold our tagged as negative labels.


Evaluation We can use the above script to test our Logistic Regression model. We use the Breast Cancer dataset from Scikit-Learn for training. We can then compare our predictions with the original labels. We can use different dimensionality reduction techniques such as PCA, to visualize the decision boundary. After reducing our features to two dimensions, we obtain the following decision boundary.


Conclusion In conclusion, this article explored the mathematical intuition of Logistic Regression and demonstrated its implementation using NumPy. Logistic Regression is a valuable classification algorithm that utilizes the sigmoid function and gradient descent to find an optimal decision boundary for binary classification. For code and implementation, refer to this GitHub repo. Follow me for more articles on deep learning architectures and research advancements.


Data Scraping in the Spotlight: Are Language Models Overstepping by Training on Everyone's Content?

What are Large Language Models and why do they need data? We'll start with a simple explainer of how Machine Learning models work - let's say you want to predict how late your upcoming flight's arrival time will be. A very basic version can be human guess work (eg. if weather sucks or if the airline sucks, it's likely late). If you want to make that more reliable, you can take real data on flight arrivals times and pattern match it again various factors (eg. how arrival times related to airline, destination airport, temperature, rainfall etc.). Now you can take this one step further, use the data and create a math equation to predict this. For example: Delay minutes = A * airline reliability score + B * busy-ness of an airport + C * amount of rainfall. How do you calculate A, B, C? By using the large volume of past arrival time data you have and doing some math on it. This equation in math terms is called a "regression" and is the one of most commonly use basic machine learning models. Note that the model is basically a math formula comprising of "features" (eg. airline reliability score, busy-ness of an airport, amount of rainfall) and "weights" (eg. A, B, C which show how much weight each variable adds to the prediction). The same concept can be extended to other more complex models - like "neural networks" (that you might have heard in the context of deep learning) or Large Language Models (often abbreviated to LLMs and are the underlying models for all text-based AI products such as Google Search, ChatGPT and Google Bard). We won't go into too much detail but each of these models, including LLMs, are a combination of "features" and "weights". The most performant models have the best combination of features and weights, the way to get to that combination is through training with a TON of data. The more data you have, the more performant the model. Therefore, having a massive volume of data is critical and companies that train these models need to source this data.


Where are they getting this data from? Broadly, data sources can be broadly categorized into: Open Source Data: These are high volume data sources that are typically available for commercial purposes, including LLM training. Examples of large open source data include Wikipedia, CommonCrawl (an open repository of web crawl data), Project Gutenberg (free eBooks), BookCorpus (free books written by unpublished authors) to name a few.Independent Content Websites: These include a broad set of websites such as news publications (think Washington Post, the Guardian), creator-specific platforms (think Kickstarter, Patreon, Medium) and user-generated content platforms (think Reddit, Twitter). These typically have more restrictive policies when it comes to scraping their content, especially if it's used for commercial purposes. In an ideal world, LLM companies would explicitly list out all the data sources they have used / scraped and do so in compliance with the policies of whoever owns the content. However, several of them have been non-transparent about it, the biggest offender being OpenAI (maker of ChatGPT). Google published one dataset it used for training, called C-4. The Washington Post put together a neat analysis of this data, here are the top 30 sources based on their analysis: Most of this data was acquired from scraping and content platforms contend that this data was scraped in violation of their terms of use. They are clearly unhappy about it, especially given the amount of upside the LLM companies are able to capture from the data.


Why should companies like OpenAI, Google care about how they source data? Okay, content providers are complaining. So what? Should companies with LLM products care about this, besides wanting to be "fair" out of the goodness of their hearts? Data sourcing is becoming increasingly critical for two major reasons. Legal Complications: Companies developing LLMs are starting to find themselves embroiled in lawsuits from content creators and publishers who believe their data was used without permission. Legal battles can be costly and tarnish the reputation of the companies involved. Case in point: Microsoft, GitHub, and OpenAI are being sued for allegedly violating copyright law by reproducing open-source code using AIGetty Images sues AI art generator Stable DiffusionAI art tools Stable Diffusion and Midjourney targeted with copyright lawsuit [side note: Stable Diffusion, Midjourney are AI image generators and not language generators and therefore not "LLMs" but the same principles of what constitutes a model and how they are trained is the same] Making headway with Enterprise Customers: Enterprise customers employing LLMs or their derivatives need to be assured of the legitimacy of the training data. They do not want to face legal challenges due to the data sourcing practices of the LLMs they use, especially if they cannot pass on the liability of those lawsuits to the LLM providers. Can you really build effective models with all of these messy data sourcing constraints? That's a fair question. A masterclass in applying these principles is the recent announcement of Adobe Firefly (it's a cool product and in open beta, you can play around with it) - the product has a wide set of features including Text to image, i.e. you can type a line of text and it will generate an image for you. What makes Firefly a great example is: Adobe only uses images that are part of Adobe Stock that they already have the licenses for, plus open source images that are not license restricted. In addition, they have also announced that they want to build generative AI in a way that enables creators to monetize their talents and that they will announce a compensation model for Adobe Stock contributors once Firefly is out of betaAdobe will indemnify its customers for Firefly outputs (starting with the text to image feature) - if you haven't heard the term "indemnify" before, in simple terms, Adobe is saying they are confident they have cleanly sourced the data going into their models and are therefore willing to cover any legislation that might come up if someone sues an Adobe customer for using Firefly output. One criticism of the clean data sourcing approach has been that it will hurt the quality of output generated by the models. The opposite side of that argument is that high quality data owned by content providers can provide better quality input to model training (garbage in, garbage out is real when it comes to model training). In the image below, left is an output from Adobe Firefly, right is from OpenAI's Dall-E. If you compare the two, they are quite similar and Firefly's output is arguably more realistic, which goes to show that high quality language models can be built off of just cleanly sourced data.


What strategies are content platforms adopting to respond to this? Several companies that have large volume of content have come out strongly expressing that they intend to charge AI companies for using their data. It's important to note that most of them have not come up with an anti-AI stance (i.e. they are not saying AI is going to take over our business, so we are shutting down access to content). They are mostly pushing for a commercial construct that defines how the access of this data will occur and how they will get compensated for it. StackOverflow, arguably the most popular forum that programmers use when they need help, plans to begin charging large AI developers for access to the 50 million Q&A content on its service. StackOverflow CEO Prashanth Chandrasekar laid out some reasonable arguments: The additional revenue will be vital to ensuring StackOverflow can keep attracting users and maintaining high-quality information, which will also help future chatbots by generating new knowledge on the platformStackOverflow will continue to license data for free to some people and companies, and only looking to charge companies developing LLMs forcommercial purposesHe argues that LLM developers are violating Stack Overflow's terms of service, which he believes falls under a Creative Commons license that requires anyone later using the data to mention where it came from (which LLMs don't do) Reddit came out with a similar announcement (alongside their controversial changes to API pricing that shut down several third party apps). Reddit CEO Steve Huffman told the Times "The Reddit corpus of data is really valuable but we don't need to give all of that value to some of the largest companies in the world for free". Twitter stopped free access to their APIs earlier this year, and also announced a recent change that limits the number of tweets a user can see in a day, in an attempt to prevent unauthorized scraping of data. Though the execution and rollout of the policies leave much to be desired, the intent is clear that they do not intend to provide free data access for commercial purposes. Another group that has come out with a united front and critique of LLMs is news organizations. The News/Media Alliance (NMA), which represents publishers in print and digital media in the US, has published what they are calling AI principles. While there isn't much tactical detail here, the message they are trying to get across is clear: Again, their arguments have not been to shut these down but to have commercial agreements in place to use this data in compliance with copyright law, and they also make the argument that compensation frameworks (for example, licensing) already exist in the market today and therefore will not slow innovation.


Conclusion This is just the beginning. Platforms with high volume of content are likely to seek compensation for their data. Even companies that have not yet announced this intent but already have other forms of data licensing programs (eg. LinkedIn, Foursquare, Reuters) are likely to adapt them for AI/LLM companies. Though this development may seem like a hindrance to innovation, it is a necessary step for the long-term sustainability of content platforms. By ensuring they are compensated fairly, content creators can continue to produce quality content, which in turn will feed into making LLMs more effective.


EU AI Act: A Promising Step or a Risky Gamble for the Future of AI?

I. The First Leap at AI Regulation is Here The EU finished drafting its AI Act this year on June 14th. And this is an initial leap towards a safe and better AI. As you may know, the EU's AI Act is the first international regulation law made on 'Artificial Intelligence. This Act lays down rules and regulations for companies (called providers) to build ethical AI systems. European countries, as well as major AI companies like OpenAI, have presented their own suggestions to the parliament. The Artificial Intelligence Act (AIA for short) urges companies to follow certain requirements while working with AI systems. This includes doing KYC of the providers, disclosing the architecture of the system, details on data privacy, et Cetra. This act was the culmination of three years of intense policymaking, though the joint efforts by the EU in this area date back to 2017. However, it was only recently that the need for AI regulations and ethics became evident. People realised the power of AI wizardry - with tools like ChatGPT and Dalle - that worked terrifically fast, making everyone artists and poets overnight. People marvelled at the capacity of AI tools. And later on, they became concerned about their safety. It became evident that these machines could generate convincing misinformation. Plagiarism and copyright issues by these wizards are an endless debate in itself, on the creative front in particular. While on the one side, people sold courses on AI prompts and 'the top X AI tools you need to know before the world ends...' kinda posts, the other side was concerned over fast-paced AI. These concerns were predominantly raised by AI experts. And they worried about the safety from these wizards running on algorithms. As I am writing this, the legislation of AIA is still under process. Its enforcement would be done likely by 2023 or 2024. On top of it, there will be a grace period of potentially 2436 months before the Act will come into force. Assuming the AI Act gets drafted by the end of the year, it still won't be in full action until the end of the next two-three years. Now that the Act is still in making, this is a great time to analyse it and see if this act can indeed tame these AI wizards, or if there are some shady under-the-table loopholes in them.


II. Wizards and their Spell Companies betting on AI are leading the future, and the fortune at the same time. The global generative AI market is worth over $13 billion. And whenever you try to regulate an industry of that magnitude - which would surpass $22 Billion by 2025 - the regulations become daunting. Not to mention, there's always a loophole in a project of this size. And such is the case with AIA. AIA is important, and in the coming years, other unions would try to build their own laws on top of this framework - making them consistent globally. Companies like OpenAI and Anthropic, the big tech giants on the AI front, both courted a universal regulations body that makes laws on AI. While the AI Act kept social safety in mind, it did create regulations to support companies under the regulations - making them flourish as well, without increasing the competition. This is because the AI Act requires companies to share the information of their system with other providers. Accountability is one major issue while using such algorithmic beasts. This created a gap - as in who will be accountable if these wizards go wild. The AIA addressed this gap by clarifying the roles and responsibilities of AI providers. With the Magnitude of AI startups rising up this year, the question of computational resources, as well as the amount of training data is crucial to address. AI act is looking to put a cap on the training of new models, ensuring the safe development of these AI systems. AIA entertains the interests of both the providers (AI Companies) and the consumers (the users). Innovations in AI must go on, while tackling the associated risks. Part of making AI safe is to develop them in such a way that it becomes safer. You need to train a beast in order to make it socialise. AIA made sure that regulation does not choke the big corporations, or put a dead end in the way of future technological advancement. Of course, the AI draft can't make every party satisfied. This was the case, obviously, for the big companies. The big corporations allege that the EU is over-regulating. Moreover, the mighty OpenAI CEO Sam Altmann, also expressed his solicitude by stating he would have to leave the EU if the rules are too strict. Under AIA, both ChatGPT and its large language model GPT-4 could be labelled as 'high-risk'. I will show you what 'high-risk' means in a bit. Moreover, according to Time, the OpenAI CEO said "If we can comply, we will, and if we can't, we'll cease operating... We will try. But there are technical limits to what's possible. Since the AI act in itself is comprehensive, I referred to the analysis made by a non-profit organisation called Future Life Institute (FLI). If you remember, this was the same organisation that issued an open letter calling for a 6-month pause in the development of AI - which was signed by 1,125 tech leaders and AI experts. The other day, I found FLI had their weekly newsletter of EU's AI Act on Substack maintained by Risto Uuk where he posts highlights, major events and FLI analysis of the EU act. FLI has done a pretty good job in analysing AIA. FLI has done research in the field of AI ethics itself. FLI developed the Asilomar AI principle - which is a set of 23 guidelines for the safe development of Artificial Intelligence.


III. High-Risk Systems and evading as exceptions The AIA is set up on a risk-based approach. Under this approach, some AI uses - that are deemed harmful - are prohibited; some uses need to follow strict rules; and some non-harmful AI uses are not caught by the AIA. The second one, i.e. those systems that are subjected to the strict requirement, falls under the category of 'high-risk' systems. According to the EU, 'high-risk' systems are those systems that will negatively affect you and your fundamental rights. EU classifies high-risk systems broadly into two categories : 1) AI products that fall under the EU's product safety legislation, which are the products that the EU banned or recalled, or 2) AI systems that fall into the following eight specific areas: Biometric identification and categorisation of natural personsManagement and operation of critical infrastructureEducation and vocational trainingEmployment, worker management and access to self-employmentAccess to and enjoyment of essential private services and public services and benefitsLaw enforcementMigration, asylum and border control managementAssistance in legal interpretation and application of the law. If an AI system falls under any of the above given bulleted points - they are labelled high risk. And yes, if you were wondering, ChatGPT (especially the GPT4 version) seems to fall under all of them. And this has caused OpenAI to fret over the AI Act. The alleged lobbying efforts by OpenAI could be one of the reasons for it. Evidently, many of its apps - majorly ChatGPT and their model GPT4 - would be classified under high risk. Let's take this case and ask: Can ChatGPT and GPT4 be considered high risks? And before you answer that, let me frame the question the OpenAI way: Are these tools inherently high risk? The answer seems clear from what we observed in the past few months. AI tools like ChatGPT are high-risk. There have been occasions where these tools have posed significant risks. Misinformation by these tools is still a thing, they hallucinate and spit convincing lies, Leaders apparently ending war thanks to deep fakes, churning out your data on the internet, countless intellectual property rights saga ... and many more. The list goes on and on. I tried to be fair and showed the broad scenarios of these risks. However, I am not blind to the fact that these machines have proved to be straight-up harmful at an individual level. Some bad things happened because of AI. For example, one person killed themselves and I wrote about it before. Another example is when some lawyers got in trouble for using ChatGPT - they did not know that the chatbot made up fake sources. These are some examples of the dark side of AI. As you can see, I can keep on going on. By the time I hit publish on this post, there would creep up another incident where people faced trouble. Someone out there in the vast world ends up a victim of these spells by the AI wizard. This shows the bounds of harm that these algorithmic machines have. These bounds are virtually boundless. FLI noticed that systems like ChatGPT can avoid the label of high risk, thanks to the Article 4c of the AIA.


IV. Article 4c - The un4seen clause of AIA Article 4c is the exception of Article 4b of the AIA. Article 4b lays out rules for the development of the AI systems, and the providers of such systems. According to this article, if you intend to create a high-risk system, you have to provide details of your system to 'other' providers like you. This is frowned upon by the AI companies as they would need to disclose the information about their own system. This also would lead to heavy competition in the market. The exception to this rule creeps up in the definition in Article 4c. According to this, you (as a provider) would NOT have to give out information about your system IF your system is not classified as 'high-risk'. This could be done by stating that you removed all the high-risk elements of your system. This act is a provision that allows providers to evade their obligations. If they can convince their systems are 'not high risk' - and that they won't be misused - they won't have to disclose their AI system. Disclosure on how an AI system is made is vital, to ensure that the providers are developing it ethically, and safely. This becomes the backdoor of evading responsibilities for big players in the AI market. AI experts, and FLI, want this article to be removed, as this could allow large tech giants to evade their obligations and fair play. This could be a major loophole, but I won't consider this as a big flaw in the AIA. What is needed right now is a better revision of this article, or better still discard this exception overall. The good news is, AIA is still under draft. FLI noted this loophole and covered it in its analysis. One notable aspect of the AIA, despite the backlash from big corporations concerned about over-regulation, is that the act aims to ensure that European organisations can remain competitive. Especially with other major players in countries like the United States and China. For instance, OpenAI and Anthropic are based in the USA, while Baidu Ernie is based in China. All of them are strong players in the AI race (that later one not so much). So, in terms of competition, European companies have a long way to go to be on par with the competition.


V. The Fast-paced future, with a pinch of reality Going by the trend, with a pinch of optimism, AI would definitely shape the tech of the future. AI is already used in multiple fields in some form or the other. Whether it is using AI to solve protein structures, using AI simulations in various parts of engineering, modernising education using AI, or even using AI to brush your teeth, or for other bizarre use cases, AI would become the new normal - just like smartphones. With so much dependence under one umbrella, safety and ethics is also a major feature that should be addressed as soon as possible. Could an AI-powered lifestyle be the new normal? Probably. That's debatable. It is important that big corporations address these issues and play their roles sincerely with combined efforts. And yes, the big tech giants hold accountability because they are the ones steering the world towards the new AI era. Addressing the loopholes, and changing them early in the draft phase is better; rather than wasting time amending them later. However, not all tech leaders exhibit arrogance, nor is the concept of AI ethics purely imaginary. Of course, it is not. Almost all big AI companies have their own safety charter. But the AI industry is largely unregulated as of now. AI models carry the inherent risk that one could not foresee. The regulatory efforts came in too late, as AI flourished more than ever. But it's better late than ever - and the EU, thankfully, took the first major step to keep the measure, While it may be a small step for a union of 28 countries, it represents a significant leap for the future of individuals online in this new AI era.


This AI newsletter is all you need #56

What happened this week in AI by Louie This week we saw several new competitors in the world of LLMs, in both open-source and closed models. Despite its impressive capabilities, the first version of the LLaMA model had licensing limitations that restricted its use exclusively to academic scenarios, which was considered one of its drawbacks. Now, Meta has recently unveiled LLaMA 2, which comes with a commercial use license. The latest model has been trained on an additional 40% of data, amounting to 2 trillion tokens, and possesses twice the context length, reaching an impressive 4096 tokens. LLaMA 2 surpasses models such as MPT and Falcon when evaluated against the chosen benchmarks. The model will be available for the public through Amazon AWS, Microsoft Azure, and Huggingface in three different sizes. (7B, 13B, 70B) In other news, after conducting a lengthy private test of their initial Claude model, Anthropic has now made the Claude 2 language model publicly available. It is a ChatGPT competitor which is accessible through the claude.ai address. Their model reportedly scored 76.5% on the Bar exam (a 3.5% improvement) and placed in the 90th percentile on the GRE. Furthermore, the model is now accessible through API access, enabling businesses and individuals to leverage it as a foundation for their own applications and projects. The evaluation of the model reveals that it excels in providing harmless responses, exhibiting a twofold improvement. This achievement aligns with the vision of Anthropic's founders upon their departure from OpenAI.As part of their ongoing exploration, they are currently investigating the feasibility of training-aligned models. To this end, they are conducting an experiment involving the training of a "Decepticon" variant of Claude. The objective is to identify the specific source of deception within models. At the beginning of the week, Elon Musk announced the establishment of xAI, a new company that directly competes with OpenAI. The primary objective of xAI is to develop Artificial General Intelligence (AGI) to gain a deeper understanding of the true nature of the universe. The company has assembled an impressive team of talented individuals from renowned organizations such as DeepMind, OpenAI, Google, and Microsoft. At present, there is a lack of specific information about the company. Nevertheless, we think its strong team, financial backing, and ambition mean it is likely to become an important contributor to the field. - Louie Peters - Towards AI Co-founder and CEO


This AI newsletter is all you need #56

Hottest News Anthropic Releases Claude 2 Anthropic has released Claude 2, an advanced AI model that outperforms Claude 1.3 in various evaluations, achieving impressive scores on Codex HumanEval and GSM8k. Claude 2 excels in coding, math, and even scored higher on the Bar exam. Additionally, it offers improved effectiveness in generating harmless responses and can handle inputs up to 100K tokens. 2. Elon Musk Launches AI Firm xAI As He Looks To Take On OpenAI Elon Musk's new AI startup, xAI, is recruiting top engineers from tech giants like Google and Microsoft to develop a "maximally curious" AI. Although separate from X Corp, xAI will closely collaborate with companies like Twitter and Tesla, aiming to bring about breakthroughs and innovation in the field of AI through synergistic efforts. 3. Bard's Latest Update: More Features, Languages and Countries Bard, a language model, has expanded its availability worldwide and now supports multiple languages. New features include the ability to listen to Bard's responses, customize the tone and style of its output, pin and rename past conversations, export Python code to Replit and Google Colab, and utilize images in prompts with the help of Google Lens integration. 4. Programs To Detect AI Discriminate Against Non-Native English Speakers Stanford researchers found that over 50% of essays written by non-native speakers were flagged as AI-generated, emphasizing the need to address the discrimination faced by non-native writers using AI detectors. This has implications for college and job applications as well as search engine algorithms, potentially harming academic careers and psychological well-being. 5. Shutterstock Expands Deal With OpenAI To Build Generative AI Tools Shutterstock announced its plans to expand its existing deal with OpenAI to provide the startup with training data for its AI models. In turn, Shutterstock will gain "priority access" to OpenAI's latest tech and new editing capabilities. Additionally, Shutterstock is working towards becoming a leader in generative AI by collaborating with top AI companies and compensating artists for their contributions to training the AI.


This AI newsletter is all you need #56

Five 5-minute reads/videos to keep you learning Train LLMs using QLoRA on Amazon SageMaker This guide explains how to use QLoRA on Amazon SageMaker for fine-tuning large language models. It highlights the usage of tools like Hugging Face Transformers, Accelerate, and the PEFT library for adapting pre-trained language models to different applications without fine-tuning all parameters. It also emphasizes the advantages of QLoRA. 2. The ultimate guide to LLMs and NLP for content marketing NLP plays a crucial role in content marketing as it automates content generation, optimizes it for search engines, gauges sentiment, segments audiences, powers chatbots, and virtual assistants, conducts social listening, and aids in content curation. This article provides an overview of how to effectively utilize NLP to scale content marketing. 3. A developer's guide to prompt engineering and LLMs This article shares GitHub's efforts with LLMs to assist developers in harnessing the technology to its fullest potential. It provides a high-level overview of how LLMs operate and offers guidance on constructing LLM applications. It utilizes GitHub Copilot code completions as a prime illustration of an LLM-based application. 4. Can A.I. Invent? The article explores the recent issuance of a patent to an AI system, prompting inquiries into the legal and ethical implications surrounding the granting of intellectual property rights to non-human entities. 5. How to Use AI to Do Stuff: An Opinionated Guide Increasingly powerful AI systems are being released at an accelerating rate. This article serves as an orientation to the current state of AI. It is an opinion piece that draws from the author's experience and focuses on selecting the appropriate tools for various tasks.


This AI newsletter is all you need #56

Papers & Repositories Instruction Mining: High-Quality Instruction Data Selection for Large Language Models This paper proposes Instruct Mining, a linear rule for evaluating the quality of instruction-following data. With a 42.5% improvement over models that use unfiltered data, this approach emphasizes the importance of data quality in fine-tuning LLMs for effective interpretation of instructions. The selection process employs natural language indicators such as naturalness, coherence, and understandability. 2. Generative Pretraining in Multimodality The paper introduces Emu, a Transformer-based multimodal foundation model capable of seamlessly generating images and texts in a multimodal context. It can handle diverse types of data input, including images, text, and videos, and surpasses other large multimodal models in tasks such as image captioning, visual question answering, and text-to-image generation. 3. Becoming Self-Instruct: Introducing Early Stopping Criteria for Minimal Instruct Tuning This paper introduces the Instruction Following Score (IFS), a metric that detects the ability of language models to follow instructions. It aids in distinguishing between base and instruct models, thereby preventing unnecessary fine-tuning that could alter a model's semantics. Furthermore, researchers have observed that significant semantic shifts occur when the IFS plateaus, underscoring the relationship between instruction following and model semantics. 4. Provably Faster Gradient Descent via Long Steps This work establishes provably faster convergence rates for gradient descent through the utilization of a computer-assisted analysis technique. The convergence rate of optimization methods represents the mathematical limit of how quickly a model can identify the optimal solution under the guidance of that particular optimization method. The cyclical learning rates can be enhanced by occasionally taking substantial steps. 5. GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest GPT4RoI is an innovative model that enhances vision-language tasks by incorporating regions of interest. This incorporation enables precise alignment between visual features and language embeddings, thereby empowering users to interact with the model through language and spatial instructions. Enjoy these papers and news summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #56

Weekly AI Podcast In this week's episode of the "What's AI" podcast, Louis Bouchard interviews Aleksa Gordić, a former research engineer at DeepMind who has embarked on his startup journey. They explore various aspects of his professional life, discussing topics such as his current priorities, his work at DeepMind, dropping out of his master's program, and securing a software engineer role in machine learning without a formal degree. Aleksa shares his experiences and provides valuable insights for anyone interested in AI. He highlights the importance of practical experience, participation in competitions, and self-learning. Furthermore, he discusses the diverse roles within companies like DeepMind and emphasizes the value of personal drive and a strong project portfolio. To gain a deeper understanding of Aleksa's journey and explore the world of AI, tune in on YouTube, Spotify, or Apple Podcasts.


This AI newsletter is all you need #56

Meme of the week! Meme shared by dimkiriakos


This AI newsletter is all you need #56

Featured Community post from the Discord Operand has released an open-source Python library for agent integration, aimed at complementing existing libraries such as HF Agent API and LangChain. This library serves as a framework for connecting agents, software systems, and human users. It achieves this by defining actions, callbacks, and access policies that facilitate the connection, monitoring, control, and interaction with agents. Check it out on GitHub and support a fellow community member. Share your thoughts on this project in the thread here.


This AI newsletter is all you need #56

AI poll of the week! Join the discussion on Discord.


This AI newsletter is all you need #56

Article of the week Top Computer Vision Papers During Week From 3/7 To 9/7 by Youssef Hosni This article offers a comprehensive overview of the most noteworthy papers published in the first week of July 2023, focusing on the latest research and advancements in computer vision. Whether you are a researcher, practitioner, or enthusiast, this article aims to provide valuable insights into state-of-the-art techniques and tools within the field of computer vision.


This AI newsletter is all you need #56

Our must-read articles ChatGPT Code Interpreter Is Now Available for All Plus Users by Gencay I. Stop Ignoring Julia! Learn It Now And Thank Your Younger Self in the Future by Bex T. Markov Chain Attribution Modelling by Snehal Nair If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.


This AI newsletter is all you need #56

Job offers Software Development Team Lead @Computronics Solutions (Sofia, Bulgaria) Junior Software QA Engineer (Manual) @CoverGo (Remote) Software Engineer @Toku (Santiago, Chile) Java Developer (m/f/d) @UBench (Turnhout, Belgium) Software Engineer (Java) @Compass Education (Hawthorn, Australia) Senior Software Developer @TherapyNotes.com (Remote) Software Developer @Servus Credit Union (Remote) Interested in sharing a job opportunity here? Contact sponsors@towardsai.net. If you are preparing your next machine learning interview, don't hesitate to check out our leading interview preparation website, confetti!


How I Created an Animation Of the Embeddings During Fine-Tuning

Preparation: Fine-tuning The first step is to fine-tune the google/vit-base-patch16224-in21k Vision Transformer (ViT) model [1], which is pre-trained. We use the CIFAR-10 dataset [2] for this, containing 60,000 images classified into ten different classes: airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. You can follow the steps outlined in the Hugging Face tutorial for image classification with transformers to execute the fine-tuning process also for CIFAR-10. Additionally, we utilize a TrainerCallback to store the loss values during training into a CSV file for later use in the animation. It's important to increase the save interval for checkpoints by setting save_strategy="step" and a low value for save_step in TrainingArguments to ensure enough checkpoints for the animation. Each frame in the animation corresponds to one checkpoint. A folder for each checkpoint and the CSV file are created during the training and are ready for further use.


Creation of the Embeddings We use the AutoFeatureExtractor and AutoModel from the Transformers library to generate embeddings from the CIFAR-10 dataset's test split using different model checkpoints. Each embedding is a 768-dimensional vector representing one of the 10,000 test images for one model checkpoint. These embeddings can be stored in the same folder as the checkpoints to maintain a good overview.


Extracting Outliers We can use the OutOfDistribution class provided by the Cleanlab library to identify outliers based on the embeddings for each checkpoint. The resulting scores can then identify the top 10 outliers for the animation.


Applying PCA and Procrustes Analysis With a Principal Component Analysis (PCA) for the scikit-learn package, we visualize the embeddings in a 2D space by reducing the 768-dimensional vectors to 2 dimensions. When recalculating PCA for each timestep, large jumps in the animation due to axis-flips or rotations can occur. To address this issue, we apply an additional Procrustes Analysis [3] from the SciPy package to geometrically transform each frame onto the last frame, which involves only translation, rotation, and uniform scaling. This enables smoother transitions in the animation.


Review in Spotlight Before finalizing the entire animation, we conduct a review in Spotlight. In this process, we utilize the first and last checkpoints to perform embedding generation, PCA, and outlier detection. We load the resulting DataFrame in Spotlight: Spotlight provides a comprehensive table in the top left, showcasing all the fields present in the dataset. On the top right, two PCA representations are displayed: one for the embeddings generated using the first checkpoint and one for the last checkpoint. Finally, in the bottom section, selected images are presented. Disclaimer: The author of this article is also one of the developers of Spotlight.


Create the animation For each checkpoint, we create an image, which we then store alongside its corresponding checkpoint. This is achieved through the utilization of the make_pca(...) and get_ood(...) functions, which generate the 2D points representing the embedding and extract the top 8 outliers, respectively. The 2D points are plotted with colors corresponding to their respective classes.The outliers are sorted based on their score, and their corresponding images are displayed in a highscore leaderboard. The training loss is loaded from a CSV file and plotted as a line graph. Finally, all the images can be compiled into a GIF using libraries such as imageio or similar.


Conclusion This article has provided a detailed guide on how to create an animation that visualizes the fine-tuning process of a Vision Transformer (ViT) model. We've walked through the steps of generating and analyzing embeddings, visualizing the results, and creating an animation that brings these elements together. Creating such an animation not only helps in understanding the complex process of fine-tuning a ViT model but also serves as a powerful tool for communicating these concepts to others. The complete code for the animation is available in the accompanying notebook on GitHub.


References [1] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020), arXiv [2] Alex Krizhevsky, Learning Multiple Layers of Features from Tiny Images (2009), University Toronto [3] Gower, John C. Generalized procrustes analysis (1975), Psychometrika


Modern NLP: A Detailed Overview. Part 1: Transformers

Attention is All You Need In 2017, Ashish Vaswani, from Google Brains, along with colleagues from the University of Toronto, presented an idea for sequence-to-sequence tasks like neural language translation and paraphrasing, different from the existing one-word at a time-step approach followed by LSTMs and RNNs. The issue with the standing architecture of RNNs as detected were: As we added one word at a time, in the case of long sequences, it was hard to retain information. In encoder-decoder structured models using RNN and LSTM, the hidden vectors were passed from one time-stamp to the other. Then, in the final step, we pass the final context vector to the decoder. The hidden context vector passed to the decoder has a greater influence on the last few words as compared to the first few words of the sequence, as the information is faded over time.To remove the issue mentioned in the first point, an attention mechanism was introduced. This suggested that while decoding, we give separate attention to the words in the input sequence. Every word in the input sequence gets a specific attention weightage vector, which is then multiplied by the word vector to create a weighted sum of vectors. But the issue was, as we did this one step at a time, it took too long for the calculations and also didn't fully irradicate the information loss.


The Idea Transformers suggested using a concept called self-attention. The model will take in the entire sentence at a time, and then use self-attention to decide how important are the other words in the sentence in the context of the current word. So, compared to the standing recurrent architecture, it has a few advantages: While detecting the weights, we have all the words already so, there is no possibility of information loss, and also, we get the context from both sides. That is, we get to know both the words, previous to the selected word and words following it, which helps to form a better context, rather than the Recurrent structure (except the case of Bi-LSTM).As we have the availability of the full sentence and also, we need to find the importance of all the other words for each word in the sentence, we can do it parallelly for all the words. This saves a lot of processing time and leads to full utilization of processing power.


Self-Attention: The Building Block Self-attention mechanism tries to find how much other words are important with respect to a particular word, then create a combined context vector, to represent that word. Basically, it means, if you choose a word in the sentence, how much that is related to the other words in the sentence? As we all know, words define the context of the sentence, and the meaning of a word is dependent on that context more often than not. This is a way to find out the context of the sentence and the related words. To achieve this, it uses three vectors, namely, Query (Qi), Key(Ki), and Value (Vi), for each input word embedding (xi) in the sentence. The length of the embedding vector x is suggested to be 512, according to the paper. To obtain these vectors first three weight matrices are defined: Wq, Wv, and Wk. We multiply each input word vector Xi with the corresponding weight matrix to get the resulting key, query, and value vectors for the given word. In order to find out the importance of one word xi in the context of the word xj, we need to find the scalar dot product of the key vector Ki corresponding to the word xi and the query vector Qj for the word xj. The dot product result is then divided by the square root of the dimension of vector Ki, which is 8 as the dimension of k is 64 as given by the paper. As suggested by the paper, if we don't divide, the dot product value goes too big, which leads to softmax values becoming too steep, and thus producing bad gradients for smooth learning. Once we find out the importance of all the words for a given word, we use softmax on the results of all the words. The softmax provides the final importance of all the individual words such that they all sum up to 1. Next, comes the value Vi vectors for the words, we multiply the vectors Vi with their corresponding importance. The intuition is the value vectors that create the representation of the words, while the importance factors throw weightage for the context of the subject word. If a word has no relation to the context word, its importance value will be very very low, and thus the final product vector will be very low, and we can disregard its significance for the task. Finally, we take the sum of all these weighted value vectors to create the final context vector for that particular word, which we receive from the attention block.


Multi-Head Attention We have seen how attention works and produces context vectors for each word. The authors of the paper used multi-head attention in order to get an unbiased composite context vector. They have used 8 such attention heads, which give 8 different context vectors for a word. The idea is, as each of the internal weight matrices, namely Wq, Wv, and Wk are initialized randomly, the variation of the initialization points in each head may help to capture a range of different features in the context vectors. Finally, for each word, we have 8 context vectors, which we concatenate together to get the representative context vector for a given word.


The Self-Attention Block: Bringing It All Together Whatever we have discussed till now is based on one particular word in the sentence, but we do need to think about all the words in the sentence and make the system parallel. The paper suggests we use embeddings of length 512 to represent each word in the sentence. Now, we already know, for NLP tasks, we usually need to use zero-padding to equalize the sentence length. Next, we stack all the 512-dimensional word vectors on top of each other, and as there is a fixed number of words in a sentence, we get a fixed dimensional 2D vector to represent the entire sentence, which is sent through the entire attention mechanism.


Transformer: The Architecture The Transformer also follows a standard Encoder-Decoder architecture. For the sake of easy and better learning, the dimension of the word vectors and outputs of layers are all kept at 512. The learning of the model is done in a auto-regressive manner, i.e., words are produced one by one, and for prediction of the (t+1) th word, we attach the output of the t-th word with the input and feed it to the model. Encoder: The authors have used modules with 2 sub-layers. The first layer contains Multi-head Attention, which we have discussed above, and the second sub-layer is a fully connected feed-forward layer. The feed-forward layer consists of 2 connected normal neural network layers. The input and output of the feed-forward layers are of dimension 512, but the internal dimensionality is of 2048, i.e, the number of nodes in the internal layer is 2048. The fully connected layer uses ReLU activation. The authors have also used an addition and Normalization layer to smooth the learning and avoid information loss as we have seen in several NLP and computer vision cases. So, the equation becomes Output = Norm( x + f(x)), where x is the input and f() is the transformation of the layer, which can either be feed-forward or an attention block. There are 6 such modules in the encoder block. Decoder: This is very similar to Encoder Block. This also has 6 modules and a similar architecture. The only difference is, in addition to the 2 sublayers already present, the decoder block introduces a 3rd sublayer, which is also an attention layer, but the inputs are masked so that the model can't use (t+1)th time stamps word as input while predicting the t-th word. The multi-head attention sublayer, without the mask, takes in the value from the corresponding layer's encoder. So, the layer takes in input from the previous decoder layer and the corresponding encoder layer. The Attention layer has been used by the model in 3 different ways, according to authors, while training. As we already know, we input three vectors for all the words, Key, Query, and Value, to the attention blocks, the authors have used this to train the model better. Option 1: In this case, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder. This allows every position in the decoder to attend to all positions in the input sequence. Option 2: In this case, values and queries come from the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder. Option 3: In this case, values and queries come from the output of the previous layer in the decoder. Each position in the decoder can attend to all positions in the previous layer of the decoder. Now, this one brings the importance of masked multi-head attention, as the model can't see words at t+1, the auto-regressive property is preserved. Finally, the authors have used a linear transformation layer followed by a softmax layer.


Positional Encoding Apart from the model, this paper also introduced the concept of positional encoding. The issue was, as this paper, does not use recurrent or convolutional networks and is not time step based either, the author felt, there should be something, to signify the positioning of the words, as it plays a major role in expressing the meaning of the sentence. In order to solve that, the authors brought in two estimations. where pos is the position of the word, i is the dimension, and dmodel = 512, the input dimension size. The size of the encodings is also kept at dimension 512, so they can be added to the word embeddings easily. The authors have selected these specific functions as, these functions give multiples of the same value after a certain offset, as a result, can be represented as linear functions.


Conclusion We have learned how transformers works; next, the second story of the series talks about the GPTs. We will learn about other evolutions as well as their implementations as well. Till then, Happy Reading!!!!


Boost Your Job Search With NLP

Introduction In the most common job platforms, search functionalities consist of filtering jobs based on a few input words plus some filters (such as location). These words generally indicate a domain or a desired position, and, in most cases, the exact input words "must" appear in the job descriptions that the search system retrieves. We're not exactly sure to what extent the latest NLP (Natural Language Processing) techniques are used to optimize our search. NLP has recorded remarkable breakthroughs in recent years, and we can leverage some techniques that are able to bring search to the next level. We can think of using semantic search based on a set of query-sentences indicating, e.g.: the tasks we would like to perform in the next jobthe ideal working context, such as the company culture(for tech jobs) the tech stack we wish to use In this article, we will recap how sBERT (sentence BERT) model works and will show how to apply a pre-trained model to improve daily tasks like searching for a job. A Dash app has been built, including these search functionalities; here's what it looks like:


Boost Your Job Search With NLP

Table of Content 1. What is sBERT?1.1 Quick review of NLP embeddings techniques1.2 Transformers overview1.3 From transformers to sBERT1.4 From BERT to sBERT2. Generating sentence embeddings2.1 Choosing among pre-trained models2.2 Using pre-trained models for semantic search3. Putting all together with Dash3.1 Data preparation3.2 App Input3.3 Matching mechanismReferences


Boost Your Job Search With NLP

1.1 Quick review of NLP embeddings techniques For any NLP tasks (like search) we need to represent our text into vectors, such that we can perform mathematical operations. There is now a vast history of embeddings techniques: "Bag of words": these traditional methods use word frequencies in documents or terms co-occurrences to create embeddings at the word (or document) level. Each vector consists of binary entries or frequency entries (there are several ways of measuring frequencies: tf-idf, ppmi, bm25, ...). The resulting vectors are often too big and sparse."Count-based models": these models overcome the limitations of size/sparsity vectors of the above approaches. Starting from term co-occurrences matrices (or document-term matrices), dimensionality reduction techniques are applied to create dense and smaller vectors. Some examples of these approaches are GloVe (which uses an enhanced version of SVD, singular-value decomposition, to reduce size), LSA (latent semantic analysis)."Predictive models": these models use simple (1 hidden layer) neural networks trained to predict a word. Word2Vec is a very famous model (more details here) that can be trained to predict a target word starting from some surrounding words (using a fixed window) or vice-versa (predicting the context from a given word). The weights learned by the hidden layer can be used as word embeddings."Contextual embeddings": the more recent research focused on creating word representations that take context into account. In fact, models like Word2Vec create a "global" representation of each word and there are some obvious limitations (think of polysemy, context-dependent terms, ...). Contextual embedding models can create dynamic word representations that change depending on the context, word type (noun/verb), grammar rules, etc. These models are based on deep neural networks. For some time, different types of RNN (recurrent neural networks) (LSTM, GRU) were proposed, with some limitations: (a) these architectures were computationally slow as they need to be trained sequentially; (b) they presented some limitations when relating words that are very far in the corpus. All these limitations were overcome by Transformers, a network architecture created by Google [1], that inspired the creation of many different models, such as BERT, GPT, etc.


Boost Your Job Search With NLP

1.2 Transformers overview Transformer neural networks were presented on machine translation tasks. They have an encoder architecture and a decoder architecture. Since in this post, we are interested in using Transformers for text embedding, we will be only focusing on the encoder part: Input Embedding: transformer architecture takes as input, for each word, an embedding vector of dimension 512 (the input embedding is an initialization that will be later improved).Positional encoding: since transformers take as input all the word embeddings at the same time (and not word by word like RNN), they need some extra information about the position of each word. This is why the input embedding vectors are summed to positional vectors. Each positional vector is built using Fourier series theory: the sin and cos functions are used to get a unique vector depending on the position of the word in the sentence (pos) and on the i-th dimension of the vector (i). For each word and dimension, the positional score is added to the corresponding embedding. (Watch this for a graphical and more exhaustive explanation).Self-attention mechanism: Intuitively, an attention mechanism allows the model to assign more weight to those parts of the input which are relevant to complete a task. There are clear examples of this mechanism in computer vision (e.g., object detection), where CNNs (Convolutional Neural Networks) learn "filters" to be applied to an image so that the network can focus on the relevant pixels. This concept was applied to NLP as well, using Q (query), K (keys), V (values) matrices. These matrices are initially just copies of the input vectors. The Q, and K matrices are multiplied to build a "mask" matrix, i.e., a matrix containing attention scores for each pair of terms. Thanks to the multiplication, similar pairs will get higher scores (the softmax function constrains these scores from 0 to 1). This attention filter is then multiplied by the V matrix, to get a weighted value for each pair of terms. Intuitively, this method allows to link each term to other parts of the input sentence to complete a task (e.g., machine translation) (watch this for a graphical and more exhaustive explanation): Multi-head attention: the above attention mechanism is repeated multiple times. Intuitively, this means that the model learns the tasks from different points of view: one layer may use attention with respect to vocabulary, another layer with respect to grammar, etc. As an output of the whole encoding layer, we obtain a matrix with all embedding vectors. These vectors store the word embedding, the positional encoding information, and the context information (i.e., what other words in the sentence got prioritized by the attention layers).


Boost Your Job Search With NLP

1.3 From transformers to BERT Since the Transformer paper, several models have been created using variations of this architecture for specific purposes. BERT (Bidirectional Encoder Representations from Transformers) was created by Google in 2018 [2] using only the encoder component of the transformer architecture. The idea of this model was to create a versatile pre-trained model that could produce contextual semantic embeddings that could be passed into a fine-tuning step. In simple terms: a user, depending on a specific task, could use a pre-trained BERT model and add a fine-tuning layer to complete the training (e.g., depending on a specific classification task). Compared to the original encoder, the BERT model: Adds some modifications to the architecture: it stacks a bigger number of transformer encoders (12 in the "BERT base" version and 24 in the "BERT large" version); it adds more multi heads in the self-attention mechanism (12 in the "BERT base" version and 16 in the "BERT large" version); the final embedding vectors have bigger sizes (768 in "BERT base" and 1024 in "BERT large").BERT is trained on two unsupervised tasks described below. The two tasks are: Masked LM (MLM): predict a word that was corrupted in the input. Practically, during training, 15% of the tokens from the input sentences were "masked" in three different ways (80% of the time, the token was replaced by a [MASK] token; 10% by a random word; 10% left unchanged). The encoder was trained to predict these tokens.Next sentence prediction (NSP): BERT was trained to learn the relationship between two sentences to improve question-answering or inference performances. When a pair of sentences A, B were given as input (separated by a special [SEP] token), 50% of the time, B was the natural sentence that followed A, while 50% of the time, B was a random sentence. The encoder was trained on predicting the "isNext" label relating A to B.


Boost Your Job Search With NLP

1.4 From BERT to sBERT A sBERT (sentence-BERT) model was created in 2019 [3] to improve the BERT model for specific tasks such as sentence similarity or sentence clustering. The sBERT paper highlighted the following "weak" points in the original BERT model: BERT was trained on NSP, but that is not exactly predicting how similar the two sentences A, and B are.BERT requires two input sentences in order to compare them, this would slow down the computation when looking for the most similar sentences in a corpus.BERT doesn't output a single sentence embedding. (Several strategies were tested to extract a sentence embedding from BERT, but a unanimous method wasn't found). The sBERT model was created with the following architecture: A BERT model is used to produce an embedding for each sentence. As discussed, since there's not a single way to extract sentence embeddings from BERT, the sBERT default way is to calculate a mean on all the transformer outputs. Then, a "pooling" layer is added to calculate fixed-size sentence embeddings (independently from the sentence length).sBERT uses Siamese networks (two copies of the same architecture), trained on different tasks (e.g., sentence classification and regression - the picture refers to this second case, i.e., predicting the sentence similarity). Depending on the task, the objective function at the top level of the architecture can be modified. Thanks to pre-trained sBERT models, we can create sophisticated sentence embeddings of fixed sizes in a computationally efficient way! Let's see an example below.


Boost Your Job Search With NLP

2.1 Choosing among pre-trained models Since the sBERT release, many pre-trained models were built and compared on several tasks. To pick a model, first of all, let's focus on the ones designed for our task. In this post, we want to find a model for semantic search, which can be of two types: Symmetric search: we have a symmetric search when the query and the result are approximately of the same size and of similar content.Asymmetric search: we have an asymmetric search when the query is short, and the expected result is a longer sentence answering the query. We'll be performing a symmetric semantic search: here's a list of the top-performing models pre-trained on this task. At the time of the article, the top-performing model is all-mpnet-base-v2 [4], which: is pre-trained on predicting sentence similarity using above 1 billion sentencesallows input sentences of max 384 tokens words (longer input is automatically truncated)outputs embeddings of size 768


Boost Your Job Search With NLP

2.2 Using pre-trained Models for semantic search Let's see the above model in action. Suppose we have extracted these two sentences from two job descriptions. One is looking for candidates with churn models experience, the other is looking for candidates with nlp experience: A user, who is looking for jobs aligned with his/her experience in customer analytics, types the following query: Note how there's no word overlapping between the query and the two sentences. Traditional models would fail to relate the query to one of the two sentences unless encoding some semantics and contextual information. Let's download the sBERT model from Huggingface and encode all three sentences: Alternatively, we can also use Huggingface API to request embeddings: We can now calculate the cosine similarity between the query and each job sentence: And then plot the embeddings (we'll use tSNE to reduce the embeddings to 3 dimensions): We can see how "job 0" was successfully selected as more similar to the query. The model "knows" that churn models are more related to customer satisfaction and retention, compared to nlp !!!


Boost Your Job Search With NLP

3. Putting all together with Dash We can now build a Dash web application that simulates a search system. This app has been deployed using Python anywhere, here.


Boost Your Job Search With NLP

3.1 Data preparation Before starting our search, we should prepare the data, clean it and produce embeddings for each sentence: We assume some job description data to be available. To get this data, we can scrape job platforms (here's one example).The data needs to be pre-processed, using classic NLP techniques, and tokenized into sentences (here's the code used for this project).In the app I have designed, two dataframes are used. One ("Job_Info") contains all job information (such as: job ID, location, employer, full description) and the other ("df_embeddings") contains sentences, one per row.Using the sentence dataframe, we produce sentence embeddings using a copy of the pre-trained sBERT model (The sentence embeddings are saved directly into "df_embeddings").


Boost Your Job Search With NLP

3.2 App Input The app requires user input to work. The user will need to: Write some query sentences. The app manages up to 10 query sentences.(Optional) Select among the available locations. Once the "Search" button is pressed, the app will use the Huggingface API to embed each query sentence (the model and code used are shown above). (If you want to set up a copy of this app locally, remember to input your own API token in ".assets/nlp_functions.py/embed_api"). Here's how the input interface looks like:


Boost Your Job Search With NLP

3.3 Matching mechanism To rank jobs, the app computes the cosine similarity of each job sentence against each query sentence. The maximum similarity for each job and query sentence will be saved. So, supposing we have 4 input query sentences, each job will have 4 similarity scores. Using these similarity scores, each job is ranked from 1 (top similarity) to n. Finally, the different ranks are combined using a weighted average to generate a single final ranking score per job. Here's why and how: Suppose we have two input query sentences. Across all job sentences, query-1 has a max of 0.2 similarity, while query-2 has a max of 0.8.If we pick a job (comparing cos similarities from all its sentences) that ranked 1st on query-1 but 10th on query-2, we may want to prioritize other jobs that scored worse in query-1 but better in query-2.So, a weighted average of individual job ranks is computed using the max similarities obtained in each query. Finally, the app produces two outputs: A box-plot showing the "strengths" of similarity scores between jobs and query sentences. One box plot is generated per query sentence, where individual data points are the max similarity scores of each job. With this chart, we can immediately see if our query sentence produced "strong" matches.Ranked results: one dbc.Card is generated per job, from most similar to least similar jobs. Each job card contains the sentences, extracted from the full job description, used by the system to rank the job. Here's how the output interface looks like: I hope this was useful and that you may apply this in your own job search. Thank you for reading!!!


Boost Your Job Search With NLP

References [1] "Attention is all you need" (2017) - Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Annotated version with implementation notes, code reviews, and examples.[2] "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (2018) - Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova[3] "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks" (2019) - Nils Reimers, and Iryna Gurevych[4] Huggingface "all-mpnet-base-v2" model


Harness the Power of Vector Databases: Influencing Language Models with Personalized Information.

How do vector databases work? First, these Databases store vectors, as their name suggests. We need to transform the text we have into information that can be stored in these tools. In other words, we need to convert our text into vectors. There are various approaches available, but in summary, they all convert a sequence of text, which can be words, syllables, or phrases, into vectors. The vectors we obtain are multidimensional, and naturally, we can calculate the difference between one vector and another or search for vectors that are closer to a specific one. With this information, we can understand how it works in a general way. We convert the text into vectors and store them.çWe convert the text to be searched into vectors and compare them.It selects the closest vectors.These vectors are then converted back into text and returned. Let's forget about text searches! It's all about vector comparisons. As you may have already guessed, the process of converting text to vectors should be the same for both the stored text and the text to be searched. Otherwise, the comparison would be meaningless. Vector Databases are gaining increasing importance, not only for cases like ours where we search for related news but also for any recommendation system. Indeed, vectors are essentially numerical representations, and they don't necessarily have to originate from text. We can store movies transformed into vectors, along with their metadata, and search for the most similar ones. We could even identify patterns that allow us to recommend movies based on user viewing habits. Wait a minute... could it be that Netflix is also using them for their recommendation system? Shall we make a bet?


What technology will we use? As for the database, I have chosen ChromaDB. It is one of the latest databases to emerge and has gained popularity rapidly. You will see that its usage is extremely straightforward! We won't need to worry about much because ChromaDB handles most of the work for us. It is an open-source solution that can be seamlessly integrated with LangChain, which is important because, in future articles, we will use LangChain to build increasingly complex solutions. We will obtain the model from Hugging Face. Specifically, I have used dolly-v23b. This is the smallest version of the Dolly Models family. I suggest using smaller versions of models whenever possible. Personally, I enjoy experimenting with different models whenever I have the opportunity, and Hugging Face offers a vast selection of models to choose from. If you want to try a different model, you can search for it on Hugging Face and make sure it is trained for text generation.


Let's get started with the project! You can find the code in a Notebook on Kaggle, where you can fork, run, and experiment with it. It is also available on GitHub in a repository where I store all the notebooks from the Large Language Models course, as well as their corresponding articles. If you are interested in following the complete course, it is best to subscribe to the GitHub Repository. This way, you will receive notifications of new lessons or modifications to the existing ones.


Let's import the necessary libraries. To get started, we'll need to install a few Python packages: sentence-transformers: This library is necessary for transforming sentences into fixed-length vectors, i.e., for embedding.transformers: This package provides various libraries and utilities that facilitate working with transformer models. Even though we may not use it directly, not installing it will result in an error message when working with the model.chromadb: Our vector database. It is easy to use, open-source, and fast. It is possibly the most widely used vector database for storing embeddings. You can install these packages using the following commands: The following two libraries are likely familiar to you: Numpy and Pandas. They are two of the most widely used Python libraries in data science. Numpy is a library for numerical computing that makes it easy to do math calculations. Pandas, on the other hand, is the go-to library for data manipulation and analysis.


Load the dataset. I have prepared the notebook to work with three different datasets available on Kaggle. All the datasets contain news articles but in different formats. Two of them only contain article summaries, while the third one includes the full text of the articles. The only reason for making the notebook work with three different datasets is to make it easy to experiment and see how the solution reacts to different inputs. Feel free to try as many datasets as you like. It's about looking at and understanding the solution's behavior and performance with different data sources. As we work with limited resources on platforms like Kaggle or Colab, I have set a limit on the number of news articles to load. This limit is defined by the variable MAX_NEWS. The field name that contains the news article has been assigned to the variable DOCUMENT, while what could be considered metadata or categories is stored in the variable TOPIC. This way, we isolate the rest of the notebook from the specific dataset we choose to use. You just need to remove the comment marker for the dataset you want to use it.


Let's import and configure the Vector Database. First, we will import ChromaDB, followed by its Settings class from the config module. This class allows us to modify the configuration of the ChromaDB system and customize its behavior. Now that we have imported the library, let's create a settings object by calling the imported Settings class. The settings object, with older versions of ChromaDB, will be created with two parameters. chroma_db_impl: We will indicate the implementation to use for the database and the format in which we will store the data. I won't go into the specifics of all the various choices, but I will explain the motivations behind the ones I chose: - For the implementation, we have selected "duckdb". It offers excellent performance as it operates mostly in memory. And is fully compatible with SQL. - As for the data format, we will be using "parquet". It is the optimal choice for tabular data. Parquet provides a good compression ratio and offers high performance for querying and processing data.persist_directory: This parameter contains the path where we want to save the information. If we do not specify it, the database will be in-memory and not persistent. However, working purely in-memory can cause issues in cloud or collaborative environments like Kaggle, as it may attempt to create a temporary file. From versión 0.4.0 you only need to call the function PersistentClient, indicating the path where store the data.


Working with the data in ChromaDB. Data in ChromaDB is organized into collections. Each collection must have a unique name, so if we try to create a collection using an existing name, it will throw an error. To achieve this, we will check if the collection exists in the list of ChromaDB collections. If it does, we will delete it before creating it again. It's important to note that this approach is suitable for testing and experimentation in this notebook. A different strategy should be implemented in a production environment. Alternatively, we could have created three separate collections, one for each dataset. I'll leave that idea here, in case you want to modify the notebook and adapt it to your preferences. After creating the collection, we are ready to add our data to the ChromaDB database. We can do this by calling the `add` function and providing the document, metadata, and a unique identifier for each record. The document can be of any length and will include the entire contents of our document. Depending on the length of the documents we want to store, we can consider splitting them into smaller parts, such as pages or chapters. We should keep in mind that the information returned by the database will be used to create the context of our prompt, and that these prompts do have limitations in terms of the length they can reach. Therefore, it's important to consider the trade-off between the length of the documents and the prompt's length limitations when designing our system. In this example, we will use the entire document information to create the prompt. However, in more advanced projects, we can employ another model to generate a summary of the returned information. This allows us to create a prompt with less content but with more relevance. We will explore this approach further when we dive into how LangChain works. The metadata is not used in the vector search itself. Metadata is used to store categories or additional information that can be used in post-filtering to refine the results. As for the unique identifier, we can easily generate it using Python. It can be as simple as generating numbers from 0 to MAX_RANGE. Once we have the information stored in ChromaDB, we can perform queries and retrieve documents that match the desired topic or search query. As mentioned earlier, the results are returned based on the similarity between the search terms and the content of the documents. It's important to note that metadata is not used in the search process. The comparison is performed solely based on the content of the document itself. In the `n_results` parameter, we specify the maximum number of documents we want to be returned. Let's see the response: As we can see, it has returned 10 news articles. They are all very short but related to laptops. Interestingly, not all of them contain the word "laptop." How is this possible? Imagine that vectors are represented in a multidimensional space, where each vector represents a point in that space. The similarity between vectors is determined by measuring the distance between these points. Let's imagine a two-dimensional space and take one of the returned phrases as an example to represent the words in that space. 'Acer Swift 3 featuring a 10th-generation Intel Ice Lake CPU, 2K screen, and more launched in India for INR 64999 (US$865)' The graph could look similar to this image, where we can see that words related to "notebook" are clustered closely together. By calculating the distance between them using vector arithmetic, we can retrieve sentences or documents that contain these words. Now that we have the data and a basic understanding of how the search works, we can start working with the model.


Let's load the model from Hugging Face and create the prompt. Now it's time to start working with the libraries from the Transformers universe. The immensely popular library maintained by Hugging Face provides access to an incredible number of models. Let's import the following utilities: AutoTokenizer: This tool is used for tokenizing text and is compatible with many of the pre-trained models available in Hugging Face's library.AutoModelForCasualLM: Provides an interface for using models specifically designed for text generation tasks, such as the ones based on GPT. In our mini-project, we are using the model databricks/dolly-v23b.Pipeline: This allows us to create a pipeline that combines different tasks. The model I have selected is dolly-v23b, which is the smallest model in the Dolly family. Even so, it still has 3 billion parameters. This model is more than sufficient for our small experiment, and based on the tests I have conducted, it seems to perform better in this case compared to GPT-2. However, I encourage you to experiment with different models yourself. My only recommendation is to start with the smallest model available in the family you choose. After these lines, we now have the tokenizer in the variable `tokenizer` and the model in `lm_model`. We will use these variables to create the pipeline. In the pipeline call, we need to specify the response size, which I will limit to 256 tokens. We also provide the value "auto" for the `device_map` field. This indicates that the model itself will decide whether to use CPU or GPU for text generation.


Creating the prompt. To create the prompt, we will use the result of the query we executed earlier on the database. In our case, it has returned 10 articles related to the word "notebook". The prompt will consist of two parts: 1. The context: This is where we will provide the information that the model needs to consider in addition to what it already knows. In our case, it will be the result obtained from the query to the database. 2. The user's question: This is the part where the user can input their specific question or query. Constructing the prompt is as simple as chaining together the desired texts to end with the desired prompt. Let's see how the prompt looks like: As you can see, everything is quite straightforward! There are no secrets. We simply tell the model: "Consider this context that I'm providing, followed by a line break, and the user's question is this." From here on, the model takes over and does all the work of interpreting the prompt and generating a correct response. Let's obtain the response. All we need to do is call the previously created pipeline and pass it the prompt recently created. Let's see the response from the model: Perfect! The model has considered the context we provided and has constructed the user's response correctly, utilizing not only its pre-training knowledge but also the information we passed in the prompt.


Conclusions, next steps. I suppose you have realized that everything has been much simpler than it seemed at the beginning. We have used a vector database to store our own information, which we used to construct the prompt for a large language model. The model has returned the correct response, taking into account the context we provided. You can imagine that this way of working opens up a world of possibilities and complements the fine-tuning of large language models perfectly. If you want to play around with the notebook, remember that it's available on Kaggle and GitHub. Here are some ideas: 1. Use all the datasets that the notebook is prepared for, and if possible, try incorporating a new dataset. 2. Explore different models on Hugging Face and compare the results. 3. Modify the prompt creation process. Feel free to experiment, iterate, and explore different possibilities. This will help you gain a more in-depth understanding of vector databases, large language models, and their application in natural language processing tasks.


Information Retrieval For Retrieval Augmented Generation

Tip 0.5 - Evaluate your IR performance Information retrieval refers to the process of finding information from a vast pool of data that is relevant to answer a retrieval query. In RAG applications, the retrieved texts will be attached to the question provided by the user to create the prompt that will be sent to the LLM. Before we can start talking about performance-improving tips, we first need a reliable way to measure our information retrieval performance. Building your evaluation set The evaluation set should contain a list of representable queries Q, a list of documents D, and a binary matrix stating which document is relevant to answer which query. Representable queries are indicative or typical of the different kinds of searches that users will commonly perform in the system. Tip: Utilize an LLM in order to build the matrix by asking it whether a specific document contains relevant information to answer a specific query. Choosing the right evaluation metrics We want to check whether the IR system returns all the relevant documents (high recall) but also that it only returns documents that are indeed relevant (high precision). Depending on your use case, one can be more important than the other. Another decision we have to make when building our IR system is what will be the number of returned documents, K. This is somewhat of an art. We want to have K as small as possible to reduce noise (and potentially costs) while making the result contain as much relevant information as possible. Hence the importance of measuring the expected quality for different K values. Based on that two considerations, the metrics I would recommend for most use cases are Precision at rank k (P@k) and Recall at rank k (R@k), which measure Precision and Recall for a variety of possible K values. It is important to spend time selecting the right evaluation metric for the use case. For additional details, I highly recommend the following blogpost by Pinecone. Now that we have the tools to establish a baseline and to check whether my tips are any good, let's get down to business.


Tip 1- Integrate TF/IDF for similarity distance Are you telling me there is something more to similarity distance than cosine similarity on the embedding space? Well, yes. Before all the fancy embedding-based distance calculations, IR systems were based on TF/IDF, the intuitive approach of calculating document similarity based on terms that occur frequently in a particular document but are rare in the overall document collection. As it turns out, integrating a variation of TF/IDF (In addition to the embedding-based search) in the similarity calculations can provide a significant performance boost.


Tip 2 - Don't embed your text as is Regardless of what embedding method or model we chose, its goal is to create a vector representation such that "similar" texts will be mapped to "similar" embedded vectors. The problem arises when our stored information has a completely different format than our retrieval query. For example, in question answering, the stored information is likely to be long informational documents, while the retrieval query is likely to be a short question. In order to mitigate the problem, we need to make sure that the stored information and the retrieval query are formatted to a common structure as the first step of the embedding process. This can be done either in the embedding procedure for the stored information, the retrieval query, or both. The formatting operations should be use-case and domain-specific. Tip: In many use cases, the formatting operation can be performed by a LLM. Examples: Question and answer - For each stored document, ask an LLM, 'What question does this document provide an answer for?' Use the embeddings of this response as the document's representation.Nontextual formats such as JSON can be converted to textual paragraphs describing the information stored within them.


Tip 3 - Embed paragraphs, not documents Focusing on smaller-sized textual objects is likely to improve the quality of the overall retrieved-context due to the following reasons: More accurate embeddings: The embedded vector will have better context preservation as paragraphs typically focus on a single topic, while documents could span multiple topics.Less noise: The returned results will be more focused and contain less off-topic information.Specifically for RAG - Ability to fit information from more documents in the context window. On the other hand, there are some notable drawbacks: Significant increase in the number of embedded vectors we need to store and calculate distance calculations on.Some paragraphs may contain information that cannot be understood without the document context.


Conclusion The efficiency of Retrieval Augmented Generation depends largely on the effectiveness of the Information Retrieval component in place. Thus, it is important to accurately evaluate the information retrieval component and work towards improving it. Small modifications in the way we store our information and perform our similarity search can dramatically improve the quality of our information retrieval. The three tips discussed in this post are battle tested and have been proven to increase the performance of the applications they were integrated into. Looking forward to hearing if integrating the tips delivered what was promised, as well as hearing additional tips that worked for you.


GPT4Readability - Never Write a README Again

GPT4Readability: Your AI-Powered Coding Assistant GPT4Readability is a powerful package that leverages large language models (LLMs) and a vector database to generate a README.md file and suggest code improvements for your Python code repositories. It's like having a personal assistant who not only understands and documents your code but also helps you make it better.


But Why Can't I just Use ChatGPT? ChatGPT is a powerful tool. However, it has a limitation when it comes to handling large volumes of text or code. You can only input a certain amount of text or code into ChatGPT at a time, which can be restrictive when dealing with extensive codebases. In contrast, GPT4Readability is designed to handle larger volumes of code. While it may still struggle with extremely large codebases, it can comfortably handle most normal-sized ones. This means by simply pointing it to the correct folder, it can parse your entire project, analyze the full context, and generate a comprehensive README or provide detailed code improvement suggestions. This is something that ChatGPT cannot do!


Getting Started with GPT4Readability I'll start by walking you through the setup. First, you'll need to install the package: Next, you'll need to create an account with OpenAI. This is because GPT4Readability uses the OpenAI API to access GPT-3.5 and GPT-4. Eventually, the hope is to also integrate open-source models from HuggingFace (stay tuned to the GitHub repository for updates). Once you've created an account on the OpenAI platform, proceed to add a payment method and generating an API key. Now, you'll need to set an environment variable named OPENAI_API_KEY equal to your newly minted API key. This allows GPT4Readability to authenticate with the OpenAI API. If you don't do this, you will be prompted to add the key when running the GPT4Readability. Using OpenAI's GPT-3 and GPT-4 LLMs through the API isn't expensive. The entire development of this tool costs just $1.50, which seems quite cheap compared to the $20/month ChatGPT plus users pay! However, you can always keep an eye on your usage of the OpenAI website to ensure you're not overspending.