4. Model implementation

Now it’s time to finally start coding. Our suggested starting point is the PyTorch file itself: copy the contents of
modeling_brand_new_bert.py inside src/transformers/models/brand_new_bert/ into
modeling_tf_brand_new_bert.py. The goal of this section is to modify the file and update the import structure of
🤗 Transformers such that you can import TFBrandNewBert and
TFBrandNewBert.from_pretrained(model_repo, from_pt=True) successfully loads a working TensorFlow BrandNewBert model.
Sadly, there is no prescription to convert a PyTorch model into TensorFlow. You can, however, follow our selection of
tips to make the process as smooth as possible:
Prepend TF to the name of all classes (e.g. BrandNewBert becomes TFBrandNewBert). Most PyTorch operations have a direct TensorFlow replacement. For example, torch.nn.Linear corresponds to
tf.keras.layers.Dense, torch


4. Model implementation

.nn.Dropout corresponds to tf.keras.layers.Dropout, etc. If you’re not sure
about a specific operation, you can use the TensorFlow documentation
or the PyTorch documentation. Look for patterns in the 🤗 Transformers codebase. If you come across a certain operation that doesn’t have a direct
replacement, the odds are that someone else already had the same problem. By default, keep the same variable names and structure as in PyTorch. This will make it easier to debug, track
issues, and add fixes down the line. Some layers have different default values in each framework. A notable example is the batch normalization layer’s
epsilon (1e-5 in PyTorch
and 1e-3 in TensorFlow).
Double-check the documentation! PyTorch’s nn.Parameter variables typically need to be initialized within TF Layer’s build(). See the following
example: PyTorch /
TensorFlow If the PyTorch mo


4. Model implementation

del has a #copied from ... on top of a function, the odds are that your TensorFlow model can also
borrow that function from the architecture it was copied from, assuming it has a TensorFlow architecture. Assigning the name attribute correctly in TensorFlow functions is critical to do the from_pt=True weight
cross-loading. name is almost always the name of the corresponding variable in the PyTorch code. If name is not
properly set, you will see it in the error message when loading the model weights. The logic of the base model class, BrandNewBertModel, will actually reside in TFBrandNewBertMainLayer, a Keras
layer subclass (example).
TFBrandNewBertModel will simply be a wrapper around this layer. Keras models need to be built in order to load pretrained weights. For that reason, TFBrandNewBertPreTrainedModel
will need to hold an example of inputs to the mo


4. Model implementation

del, the dummy_inputs
(example). If you get stuck, ask for help - we’re here to help you! 🤗
In addition to the model file itself, you will also need to add the pointers to the model classes and related
documentation pages. You can complete this part entirely following the patterns in other PRs
(example). Here’s a list of the needed manual
changes:
Include all public classes of BrandNewBert in src/transformers/__init__.py Add BrandNewBert classes to the corresponding Auto classes in src/transformers/models/auto/modeling_tf_auto.py Add the lazy loading classes related to BrandNewBert in src/transformers/utils/dummy_tf_objects.py Update the import structures for the public classes in src/transformers/models/brand_new_bert/__init__.py Add the documentation pointers to the public methods of BrandNewBert in docs/source/en/model_doc/brand_new_bert.md Add yoursel


4. Model implementation

f to the list of contributors to BrandNewBert in docs/source/en/model_doc/brand_new_bert.md Finally, add a green tick ✅ to the TensorFlow column of BrandNewBert in docs/source/en/index.md
When you’re happy with your implementation, run the following checklist to confirm that your model architecture is
ready:
All layers that behave differently at train time (e.g. Dropout) are called with a training argument, which is
propagated all the way from the top-level classes You have used #copied from ... whenever possible TFBrandNewBertMainLayer and all classes that use it have their call function decorated with @unpack_inputs TFBrandNewBertMainLayer is decorated with @keras_serializable A TensorFlow model can be loaded from PyTorch weights using TFBrandNewBert.from_pretrained(model_repo, from_pt=True) You can call the TensorFlow model using the expected input for


5. Add model tests

Hurray, you’ve implemented a TensorFlow model! Now it’s time to add tests to make sure that your model behaves as
expected. As in the previous section, we suggest you start by copying the test_modeling_brand_new_bert.py file in
tests/models/brand_new_bert/ into test_modeling_tf_brand_new_bert.py, and continue by making the necessary
TensorFlow replacements. For now, in all .from_pretrained() calls, you should use the from_pt=True flag to load
the existing PyTorch weights.
After you’re done, it’s time for the moment of truth: run the tests! 😬
  Copied NVIDIA_TF32_OVERRIDE=0 RUN_SLOW=1 RUN_PT_TF_CROSS_TESTS=1 \
py.test -vv tests/models/brand_new_bert/test_modeling_tf_brand_new_bert.py
The mos


5. Add model tests

t likely outcome is that you’ll see a bunch of errors. Don’t worry, this is expected! Debugging ML models is
notoriously hard, and the key ingredient to success is patience (and breakpoint()). In our experience, the hardest
problems arise from subtle mismatches between ML frameworks, for which we have a few pointers at the end of this guide.
In other cases, a general test might not be directly applicable to your model, in which case we suggest an override
at the model test class level. Regardless of the issue, don’t hesitate to ask for help in your draft pull request if
you’re stuck.
When all tests pass, congratulations, your model is nearly ready to be added to the 🤗 Transformers library!


6.-7. Ensure everyone can use your model

6. Submit the pull request
Once you’re done with the implementation and the tests, it’s time to submit a pull request. Before pushing your code,
run our code formatting utility, make fixup 🪄. This will automatically fix any formatting issues, which would cause
our automatic checks to fail.
It’s now time to convert your draft pull request into a real pull request. To do so, click on the “Ready for
review” button and add Joao (@gante) and Matt (@Rocketknight1) as reviewers. A model pull request will need
at least 3 reviewers, but they will take care of finding appropriate additional reviewers for your model.
After all reviewers are happy with the state of your PR, the fina


l action point is to remove the from_pt=True flag in
.from_pretrained() calls. Since there are no TensorFlow weights, you will have to add them! Check the section
below for instructions on how to do it.
Finally, when the TensorFlow weights get merged, you have at least 3 reviewer approvals, and all CI checks are
green, double-check the tests locally one last time
  Copied NVIDIA_TF32_OVERRIDE=0 RUN_SLOW=1 RUN_PT_TF_CROSS_TESTS=1 \
py.test -vv tests/models/brand_new_bert/test_modeling_tf_brand_new_bert.py
and we will merge your PR! Congratulations on the milestone 🎉
7. (Optional) Build demos and share with the world
One of the hardest parts about open-source is discovery.


 How can the other users learn about the existence of your
fabulous TensorFlow contribution? With proper communication, of course! 📣
There are two main ways to share your model with the community:
Build demos. These include Gradio demos, notebooks, and other fun ways to show off your model. We highly
encourage you to add a notebook to our community-driven demos. Share stories on social media like Twitter and LinkedIn. You should be proud of your work and share
your achievement with the community - your model can now be used by thousands of engineers and researchers around
the world 🌍! We will be happy to retweet your posts and help you share your work with the community.


Adding TensorFlow weights to 🤗 Hub

Assuming that the TensorFlow model architecture is available in 🤗 Transformers, converting PyTorch weights into
TensorFlow weights is a breeze!
Here’s how to do it:
Make sure you are logged into your Hugging Face account in your terminal. You can log in using the command
huggingface-cli login (you can find your access tokens here) Run transformers-cli pt-to-tf --model-name foo/bar, where foo/bar is the name of the model repository
containing the PyTorch weights you want to convert Tag @joaogante and @Rocketknight1 in the 🤗 Hub PR the command above has just created
That’s it! 🎉


Debugging mismatches across ML frameworks 🐛

At some point, when adding a new architecture or when creating TensorFlow weights for an existing architecture, you
might come across errors complaining about mismatches between PyTorch and TensorFlow. You might even decide to open the
model architecture code for the two frameworks, and find that they look identical. What’s going on? 🤔
First of all, let’s talk about why understanding these mismatches matters. Many community members will use 🤗
Transformers models out of the box, and trust that our models behave as expected. When there is a large mismatch
between the two frameworks, it implies that the model is not following the reference implementation for at least one
of the frameworks. This might lead to silent failures, in which the model runs but has poor performance. This is
arguably worse than a model that fails to run at all! To that end,


 we aim at having a framework mismatch smaller than
1e-5 at all stages of the model.
As in other numerical problems, the devil is in the details. And as in any detail-oriented craft, the secret
ingredient here is patience. Here is our suggested workflow for when you come across this type of issues:
Locate the source of mismatches. The model you’re converting probably has near identical inner variables up to a
certain point. Place breakpoint() statements in the two frameworks’ architectures, and compare the values of the
numerical variables in a top-down fashion until you find the source of the problems. Now that you’ve pinpointed the source of the issue, get in touch with the 🤗 Transformers team. It is possible
that we’ve seen a similar problem before and can promptly provide a solution. As a fallback, scan popular pages
like StackOverflow and


GitHub issues. If there is no solution in sight, it means you’ll have to go deeper. The good news is that you’ve located the
issue, so you can focus on the problematic instruction, abstracting away the rest of the model! The bad news is
that you’ll have to venture into the source implementation of said instruction. In some cases, you might find an
issue with a reference implementation - don’t abstain from opening an issue in the upstream repository.
In some cases, in discussion with the 🤗 Transformers team, we might find that fixing the mismatch is infeasible.
When the mismatch is very small in the output layers of the model (but potentially large in the hidden states), we
might decide to ignore it in favor of distributing the model. The pt-to-tf CLI mentioned above has a --max-error
flag to override the error message at weight conversion time.


Performance and Scalability

Training large transformer models and deploying them to production present various challenges.
During training, the model may require more GPU memory than available or exhibit slow training speed. In the deployment
phase, the model can struggle to handle the required throughput in a production environment.
This documentation aims to assist you in overcoming these challenges and finding the optimal setting for your use-case.
The guides are divided into training and inference sections, as each comes with different challenges and solutions.
Within each section you’ll find separate guides for different hardware configurations, such as single GPU vs. multi-GPU
for training or CPU vs. GPU for inference.
Use this document as your starting point to navigate further to the methods that match your scenario.


Training

Training large transformer models efficiently requires an accelerator such as a GPU or TPU. The most common case is where
you have a single GPU. The methods that you can apply to improve training efficiency on a single GPU extend to other setups
such as multiple GPU. However, there are also techniques that are specific to multi-GPU or CPU training. We cover them in
separate sections.
Methods and tools for efficient training on a single GPU: start here to learn common approaches that can help optimize GPU memory utilization, speed up the training, or both. Multi-GPU trainin


Training

g section: explore this section to learn about further optimization methods that apply to a multi-GPU settings, such as data, tensor, and pipeline parallelism. CPU training section: learn about mixed precision training on CPU. Efficient Training on Multiple CPUs: learn about distributed CPU training. Training on TPU with TensorFlow: if you are new to TPUs, refer to this section for an opinionated introduction to training on TPUs and using XLA. Custom hardware for training: find tips and tricks when building your own deep learning rig. Hyperparameter Search using Trainer AP


Inference

Efficient inference with large models in a production environment can be as challenging as training them. In the following
sections we go through the steps to run inference on CPU and single/multi-GPU setups.
Inference on a single CPU Inference on a single GPU Multi-GPU inference XLA Integration for TensorFlow Models


Training and inference

Here you’ll find techniques, tips and tricks that apply whether you are training a model, or running inference with it.
Instantiating a big model Troubleshooting performance issues


Contribute

This document is far from being complete and a lot more needs to be added, so if you have additions or corrections to
make please don’t hesitate to open a PR or if you aren’t sure start an Issue and we can discuss the details there.
When making contributions that A is better than B, please try to include a reproducible benchmark and/or a link to the
source of that information (unless it comes directly from you).


The Transformer model family

Since its introduction in 2017, the original Transformer model has inspired many new and exciting models that extend beyond natural language processing (NLP) tasks. There are models for predicting the folded structure of proteins, training a cheetah to run, and time series forecasting. With so many Transformer variants available, it can be easy to miss the bigger picture. What all these models have in common is they’re based on the original Transformer architecture. Some models only use the encoder or decoder, while others use both. This provides a useful taxonomy to categorize and examine the high-level differences within models in the Transformer family, and it’ll help you understand Transformers you haven’t encountered before.
If you aren’t familiar with the original Transformer model or need a refresher, check out the How do Transformers work chapter from the Hugging Face course.


Convolutional network

For a long time, convolutional networks (CNNs) were the dominant paradigm for computer vision tasks until the Vision Transformer demonstrated its scalability and efficiency. Even then, some of a CNN’s best qualities, like translation invariance, are so powerful (especially for certain tasks) that some Transformers incorporate convolutions in their architecture. ConvNeXt flipped this exchange around and incorporated design choices from Transformers to modernize a CNN. For example, ConvNeXt uses non-overlapping sliding windows to patchify an image and a larger kernel to increase its global receptive field. ConvNeXt also makes several layer design choices to be more memory-efficient and improve performance, so it competes favorably with Transformers!


Encoder

The Vision Transformer (ViT) opened the door to computer vision tasks without convolutions. ViT uses a standard Transformer encoder, but its main breakthrough was how it treated an image. It splits an image into fixed-size patches and uses them to create an embedding, just like how a sentence is split into tokens. ViT capitalized on the Transformers’ efficient architecture to demonstrate competitive results with the CNNs at the time while requiring fewer resources to train. ViT was soon followed by other vision models that could also handle dense vision tasks like segmentation as well as detection.
One of these models is the Swin Transformer. It builds hierarchical feature maps (like a CNN 👀 and unlike ViT) from smaller-sized patches and merges them with neighboring patches in deeper layers. Attention is only computed within a local window, and the window is shifted between attention layers to create connections to help the model learn better. Since the Swin Transformer


Encoder

 can produce hierarchical feature maps, it is a good candidate for dense prediction tasks like segmentation and detection. The SegFormer also uses a Transformer encoder to build hierarchical feature maps, but it adds a simple multilayer perceptron (MLP) decoder on top to combine all the feature maps and make a prediction.
Other vision models, like BeIT and ViTMAE, drew inspiration from BERT’s pretraining objective. BeIT is pretrained by masked image modeling (MIM); the image patches are randomly masked, and the image is also tokenized into visual tokens. BeIT is trained to predict the visual tokens corresponding to the masked patches. ViTMAE has a similar pretraining objective, except it must predict the pixels instead of visual tokens. What’s unusual is 75% of the image patches are masked! The decoder reconstructs the pixels from the masked tokens and encoded patches. After pretraining, the decoder is thrown away, and the encoder is ready to be used in downstream tasks


Decoder

Decoder-only vision models are rare because most vision models rely on an encoder to learn an image representation. But for use cases like image generation, the decoder is a natural fit, as we’ve seen from text generation models like GPT-2. ImageGPT uses the same architecture as GPT-2, but instead of predicting the next token in a sequence, it predicts the next pixel in an image. In addition to image generation, ImageGPT could also be finetuned for image classification.


Encoder-decoder

Vision models commonly use an encoder (also known as a backbone) to extract important image features before passing them to a Transformer decoder. DETR has a pretrained backbone, but it also uses the complete Transformer encoder-decoder architecture for object detection. The encoder learns image representations and combines them with object queries (each object query is a learned embedding that focuses on a region or object in an image) in the decoder. DETR predicts the bounding box coordinates and class label for each object query.


Encoder

BERT is an encoder-only Transformer that randomly masks certain tokens in the input to avoid seeing other tokens, which would allow it to “cheat”. The pretraining objective is to predict the masked token based on the context. This allows BERT to fully use the left and right contexts to help it learn a deeper and richer representation of the inputs. However, there was still room for improvement in BERT’s pretraining strategy. RoBERTa improved upon this by introducing a new pretraining recipe that includes training for longer and on larger batches, randomly masking tokens at each epoch instead of just once during preprocessing, and removing the next-sentence prediction objecti


Encoder

ve.
The dominant strategy to improve performance is to increase the model size. But training large models is computationally expensive. One way to reduce computational costs is using a smaller model like DistilBERT. DistilBERT uses knowledge distillation - a compression technique - to create a smaller version of BERT while keeping nearly all of its language understanding capabilities.
However, most Transformer models continued to trend towards more parameters, leading to new models focused on improving training efficiency. ALBERT reduces memory consumption by lowering the number of parameters in two ways: separating the larger vocabulary embedding into two smaller matrices a


Encoder

nd allowing layers to share parameters. DeBERTa added a disentangled attention mechanism where the word and its position are separately encoded in two vectors. The attention is computed from these separate vectors instead of a single vector containing the word and position embeddings. Longformer also focused on making attention more efficient, especially for processing documents with longer sequence lengths. It uses a combination of local windowed attention (attention only calculated from fixed window size around each token) and global attention (only for specific task tokens like [CLS] for classification) to create a sparse attention matrix instead of a full attention matri


Decoder

GPT-2 is a decoder-only Transformer that predicts the next word in the sequence. It masks tokens to the right so the model can’t “cheat” by looking ahead. By pretraining on a massive body of text, GPT-2 became really good at generating text, even if the text is only sometimes accurate or true. But GPT-2 lacked the bidirectional context from BERT’s pretraining, which made it unsuitable for certain tasks. XLNET combines the best of both BERT and GPT-2’s pretraining objectives by using a permutation language modeling objective (PLM) that allows it to lear


Decoder

n bidirectionally.
After GPT-2, language models grew even bigger and are now known as large language models (LLMs). LLMs demonstrate few- or even zero-shot learning if pretrained on a large enough dataset. GPT-J is an LLM with 6B parameters and trained on 400B tokens. GPT-J was followed by OPT, a family of decoder-only models, the largest of which is 175B and trained on 180B tokens. BLOOM was released around the same time, and the largest model in the family has 176B parameters and is trained on 366B tokens in 46 languages and 13 programming languages.


Encoder-decoder

BART keeps the original Transformer architecture, but it modifies the pretraining objective with text infilling corruption, where some text spans are replaced with a single mask token. The decoder predicts the uncorrupted tokens (future tokens are masked) and uses the encoder’s hidden states to help it. Pegasus is similar to BART, but Pegasus masks entire sentences instead of text spans. In addition to masked language modeling, Pegasus is pretrained by gap sentence generation (GSG). The GSG objective masks whole sentences important to a document, replacing them with a mask token. The decoder must generate the output from the remaining sentences. T5 is a more unique model that casts all NLP tasks into a text-to-text problem using specific prefixes. For example, the prefix Summarize: indicates a summarization task. T5 is pretrained by supervised (GLUE and SuperGLUE) training and self-supervised training (randomly sample and drop out 15% of tokens).


Encoder

Wav2Vec2 uses a Transformer encoder to learn speech representations directly from raw audio waveforms. It is pretrained with a contrastive task to determine the true speech representation from a set of false ones. HuBERT is similar to Wav2Vec2 but has a different training process. Target labels are created by a clustering step in which segments of similar audio are assigned to a cluster which becomes a hidden unit. The hidden unit is mapped to an embedding to make a prediction.


Encoder-decoder

Speech2Text is a speech model designed for automatic speech recognition (ASR) and speech translation. The model accepts log mel-filter bank features extracted from the audio waveform and pretrained autoregressively to generate a transcript or translation. Whisper is also an ASR model, but unlike many other speech models, it is pretrained on a massive amount of ✨ labeled ✨ audio transcription data for zero-shot performance. A large chunk of the dataset also contains non-English languages, meaning Whisper can also be used for low-resource languages. Structurally, Whisper is similar to Speech2Text. The audio signal is converted to a log-mel spectrogram encoded by the encoder. The decoder generates the transcript autoregressively from the encoder’s hidden states and the previous tokens.


Encoder

VisualBERT is a multimodal model for vision-language tasks released shortly after BERT. It combines BERT and a pretrained object detection system to extract image features into visual embeddings, passed alongside text embeddings to BERT. VisualBERT predicts the masked text based on the unmasked text and the visual embeddings, and it also has to predict whether the text is aligned with the image. When ViT was released, ViLT adopted ViT in its architecture because it was easier to get the image embeddings this way. The image embeddings are jointly processed with the text embeddings. From there, ViLT is pretrained by image text matching, masked la


Encoder

nguage modeling, and whole word masking.
CLIP takes a different approach and makes a pair prediction of (image, text) . An image encoder (ViT) and a text encoder (Transformer) are jointly trained on a 400 million (image, text) pair dataset to maximize the similarity between the image and text embeddings of the (image, text) pairs. After pretraining, you can use natural language to instruct CLIP to predict the text given an image or vice versa. OWL-ViT builds on top of CLIP by using it as its backbone for zero-shot object detection. After pretraining, an object detection head is added to make a set prediction over the (class, bounding box) pairs


Encoder-decoder

Optical character recognition (OCR) is a long-standing text recognition task that typically involves several components to understand the image and generate the text. TrOCR simplifies the process using an end-to-end Transformer. The encoder is a ViT-style model for image understanding and processes the image as fixed-size patches. The decoder accepts the encoder’s hidden states and autoregressively generates text. Donut is a more general visual document understanding model that doesn’t rely on OCR-based approaches. It uses a Swin Transformer as the encoder and multilingual BART as the decoder. Donut is pretrained to read text by predicting the next word based on the image and text annotations. The decoder generates a token sequence given a prompt. The prompt is represented by a special token for each downstream task. For example, document parsing has a special parsing token that is combined with the encoder hidden states to parse the document into a structured output format (JSON).


Decoder

The Decision and Trajectory Transformer casts the state, action, and reward as a sequence modeling problem. The Decision Transformer generates a series of actions that lead to a future desired return based on returns-to-go, past states, and actions. For the last K timesteps, each of the three modalities are converted into token embeddings and processed by a GPT-like model to predict a future action token. Trajectory Transformer also tokenizes the states, actions, and rewards and processes them with a GPT architecture. Unlike the Decision Transformer, which is focused on reward conditioning, the Trajectory Transformer generates future actions with beam search.


🤗 Transformers Notebooks

You can find here a list of the official notebooks provided by Hugging Face.
Also, we would like to list here interesting content created by the community.
If you wrote some notebook(s) leveraging 🤗 Transformers and would like to be listed here, please open a
Pull Request so it can be included under the Community notebooks.


Documentation notebooks

You can open any page of the documentation as a notebook in Colab (there is a button directly on said pages) but they are also listed here if you need them:
| Notebook                       | Description                                                    |   Unnamed: 2 |   Unnamed: 3 |
|--------------------------------|----------------------------------------------------------------|--------------|--------------|
| Quicktour of the library       | A presentation of the various APIs in Transformers             |          nan |          nan |
| Summary of the tasks           | How to run the m


Documentation notebooks

odels of the Transformers library task by task |          nan |          nan |
| Preprocessing data             | How to use a tokenizer to preprocess your data                 |          nan |          nan |
| Fine-tuning a pretrained model | How to use the Trainer to fine-tune a pretrained model         |          nan |          nan |
| Summary of the tokenizers      | The differences between the tokenizers algorithm               |          nan |          nan |
| Multilingual models            | How to use the multilingual models of the library              |          nan |          nan |


PyTorch Examples

Natural Language Processing
| Notebook                                         | Description                                                                                |   Unnamed: 2 |   Unnamed: 3 |
|--------------------------------------------------|-------------------------------------------------------------------------------------------------------------|--------------|--------------|
| Train your tokenizer                             | How to train and use your very own tokenizer                                                                |          nan |          nan |
| Train your language model                        | How to easily start using transformers                                                                      |          nan |          nan |
| How to fine-tune a model on text classification  | Show how to preprocess the data and fine-tune a pretrain


PyTorch Examples

ed model on any GLUE task.                          |          nan |          nan |
| How to fine-tune a model on language modeling    | Show how to preprocess the data and fine-tune a pretrained model on a causal or masked LM task.             |          nan |          nan |
| How to fine-tune a model on token classification | Show how to preprocess the data and fine-tune a pretrained model on a token classification task (NER, PoS). |          nan |          nan |
| How to fine-tune a model on question answering   | Show how to preprocess the data and fine-tune a pretrained model on SQUAD.                                  |          nan |          nan |
| How to fine-tune a model on multiple choice      | Show how to preprocess the data and fine-tune a pretrained model on SWAG.                                   |          nan |          nan |
| How to fine-tune a model on translation          |


PyTorch Examples

Show how to preprocess the data and fine-tune a pretrained model on WMT.                                    |          nan |          nan |
| How to fine-tune a model on summarization        | Show how to preprocess the data and fine-tune a pretrained model on XSUM.                                   |          nan |          nan |
| How to train a language model from scratch       | Highlight all the steps to effectively train Transformer model on custom data                               |          nan |          nan |
| How to generate text                             | How to use different decoding methods for language generation with transformers                             |          nan |          nan |
| How to generate text (with constraints)          | How to guide language generation with user-provided constraints                                             |          nan |          nan


PyTorch Examples

 |
| Reformer                                         | How Reformer pushes the limits of language modeling                                                         |          nan |          nan |
 Computer Vision
| Notebook                                                          | Description                                                                                |   Unnamed: 2 |   Unnamed: 3 |
|-------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|--------------|--------------|
| How to fine-tune a model on image classification (Torchvision)    | Show how to preprocess the data using Torchvision and fine-tune any pretrained Vision model on Image Classification    |          nan |          nan |
| How to fine-tune a model on ima


PyTorch Examples

ge classification (Albumentations) | Show how to preprocess the data using Albumentations and fine-tune any pretrained Vision model on Image Classification |          nan |          nan |
| How to fine-tune a model on image classification (Kornia)         | Show how to preprocess the data using Kornia and fine-tune any pretrained Vision model on Image Classification         |          nan |          nan |
| How to perform zero-shot object detection with OWL-ViT            | Show how to perform zero-shot object detection on images with text queries                                             |          nan |          nan |
| How to fine-tune an image captioning model                        | Show how to fine-tune BLIP for image captioning on a custom dataset                                                    |          nan |          nan |
| How to build an image similarity system with Transformer


PyTorch Examples

s         | Show how to build an image similarity system                                                                           |          nan |          nan |
| How to fine-tune a SegFormer model on semantic segmentation       | Show how to preprocess the data and fine-tune a pretrained SegFormer model on Semantic Segmentation                    |          nan |          nan |
| How to fine-tune a VideoMAE model on video classification         | Show how to preprocess the data and fine-tune a pretrained VideoMAE model on Video Classification                      |          nan |          nan |
 Audio
| Notebook                                                    | Description                                                                                |   Unnamed: 2 |   Unnamed: 3 |
|-------------------------------------------------------------|----------------------------------


PyTorch Examples

-----------------------------------------------------------------------|--------------|--------------|
| How to fine-tune a speech recognition model in English      | Show how to preprocess the data and fine-tune a pretrained Speech model on TIMIT                        |          nan |          nan |
| How to fine-tune a speech recognition model in any language | Show how to preprocess the data and fine-tune a multi-lingually pretrained speech model on Common Voice |          nan |          nan |
| How to fine-tune a model on audio classification            | Show how to preprocess the data and fine-tune a pretrained Speech model on Keyword Spotting             |          nan |          nan |
 Biological Sequences
| Notebook                                           | Description                                                                             |   Unnamed: 2 |   Unnamed: 3 |
|--------


PyTorch Examples

--------------------------------------------|-----------------------------------------------------------------------------------------|--------------|--------------|
| How to fine-tune a pre-trained protein model       | See how to tokenize proteins and fine-tune a large pre-trained protein “language” model |          nan |          nan |
| How to generate protein folds                      | See how to go from protein sequence to a full protein model and PDB file                |          nan |          nan |
| How to fine-tune a Nucleotide Transformer model    | See how to tokenize DNA and fine-tune a large pre-trained DNA “language” model          |          nan |          nan |
| Fine-tune a Nucleotide Transformer model with LoRA | Train even larger DNA models in a memory-efficient way                                  |          nan |          nan |
 Other modalities
| Notebook


PyTorch Examples

               | Description                                                  |   Unnamed: 2 |   Unnamed: 3 |
|---------------------------------------|--------------------------------------------------------------|--------------|--------------|
| Probabilistic Time Series Forecasting | See how to train Time Series Transformer on a custom dataset |          nan |          nan |
 Utility notebooks
| Notebook                    | Description                                                      |   Unnamed: 2 |   Unnamed: 3 |
|-----------------------------|------------------------------------------------------------------|--------------|--------------|
| How to export model to ONNX | Highlight how to export and run inference workloads through ONNX |          nan |          nan |
| How to use Benchmarks       | How to benchmark models with transformers                        |          nan |


TensorFlow Examples

Natural Language Processing
| Notebook                                         | Description                                                                                |   Unnamed: 2 |   Unnamed: 3 |
|--------------------------------------------------|-------------------------------------------------------------------------------------------------------------|--------------|--------------|
| Train your tokenizer                             | How to train and use your very own tokenizer                                                                |          nan |          nan |
| Train your language model                        | How to easily start using transformers                                                                      |          nan |          nan |
| How to fine-tune a model on text classification  | Show how to preprocess the data and fine-tune a pretrained model on any GLUE task.                          |


TensorFlow Examples

  nan |          nan |
| How to fine-tune a model on language modeling    | Show how to preprocess the data and fine-tune a pretrained model on a causal or masked LM task.             |          nan |          nan |
| How to fine-tune a model on token classification | Show how to preprocess the data and fine-tune a pretrained model on a token classification task (NER, PoS). |          nan |          nan |
| How to fine-tune a model on question answering   | Show how to preprocess the data and fine-tune a pretrained model on SQUAD.                                  |          nan |          nan |
| How to fine-tune a model on multiple choice      | Show how to preprocess the data and fine-tune a pretrained model on SWAG.                                   |          nan |          nan |
| How to fine-tune a model on translation          | Show how to preprocess the data and fine-tune a pretrained model on WMT.                                    |          nan


TensorFlow Examples

 |          nan |
| How to fine-tune a model on summarization        | Show how to preprocess the data and fine-tune a pretrained model on XSUM.                                   |          nan |          nan |
 Computer Vision
| Notebook                                                    | Description                                                                                |   Unnamed: 2 |   Unnamed: 3 |
|-------------------------------------------------------------|-----------------------------------------------------------------------------------------------------|--------------|--------------|
| How to fine-tune a model on image classification            | Show how to preprocess the data and fine-tune any pretrained Vision model on Image Classification   |          nan |          nan |
| How to fine-tune a SegFormer model on semantic segmentation | Show how to preprocess the data and fine-tune a pretrained SegFormer model on Semantic Seg


TensorFlow Examples

mentation |          nan |          nan |
 Biological Sequences
| Notebook                                     | Description                                                                             |   Unnamed: 2 |   Unnamed: 3 |
|----------------------------------------------|-----------------------------------------------------------------------------------------|--------------|--------------|
| How to fine-tune a pre-trained protein model | See how to tokenize proteins and fine-tune a large pre-trained protein “language” model |          nan |          nan |
 Utility notebooks
| Notebook                            | Description                                             |   Unnamed: 2 |   Unnamed: 3 |
|-------------------------------------|---------------------------------------------------------|--------------|--------------|
| How to train TF/Keras models on TPU | See how to train at high speed on Google’s TPU hardware |          nan |          na


Optimum notebooks

🤗  Optimum is an extension of 🤗 Transformers, providing a set of performance optimization tools enabling maximum efficiency to train and run models on targeted hardwares.
| Notebook                                                                     | Description                                                                                |   Unnamed: 2 |   Unnamed: 3 |
|------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|--------------|--------------|
| How to quantize a model with ONNX Runtime for text classification            | Show how to apply static and dynamic quantization on a model using  ONNX Runtime


Optimum notebooks

  for any GLUE task.                                  |          nan |          nan |
| How to quantize a model with Intel Neural Compressor for text classification | Show how to apply static, dynamic and aware training quantization on a model using  Intel Neural Compressor (INC)  for any GLUE task. |          nan |          nan |
| How to fine-tune a model on text classification with ONNX Runtime            | Show how to preprocess the data and fine-tune a model on any GLUE task using  ONNX Runtime  .                                         |          nan |          nan |
| How to fine-tune a model on summarization with ONNX Runtime                  | Show how to preprocess the data and fine-tune a model on XSUM using  ONNX Runtime  .                                                  |          nan |          nan |


Generation with LLMs

LLMs, or Large Language Models, are the key component behind text generation. In a nutshell, they consist of large pretrained transformer models trained to predict the next word (or, more precisely, token) given some input text. Since they predict one token at a time, you need to do something more elaborate to generate new sentences other than just calling the model — you need to do autoregressive generation.
Autoregressive generation is the inference-time procedure of iteratively calling a model with its own generated outputs, given a few initial inputs. In 🤗 Transformers, this is handled by the generate() method, which is available to all models with generative capabilities.
This tutorial will show you how to:
Generate text with an LLM Avoid common pitfalls Next steps to help you get the most out of your LLM
Before you begin, make sure you have all the necessary libraries installed:
  Copied pip install transformers bitsandbytes>=0.39.0 -q


Generate text

A language model trained for causal language modeling takes a sequence of text tokens as input and returns the probability distribution for the next token.
 "Forward pass of an LLM"
A critical aspect of autoregressive generation with LLMs is how to select the next token from this probability distribution. Anything goes in this step as long as you end up with a token for the next iteration. This means it can be as simple as selecting the most likely token from the probability distribution or as complex as applying a dozen transformations before sampling from the resulting distribution.
 "Autoregressive generation iteratively selects the next token from a probability distribution to generate text"
The process depicted above is repeated iteratively until some stopping condition is


Generate text

 reached. Ideally, the stopping condition is dictated by the model, which should learn when to output an end-of-sequence (EOS) token. If this is not the case, generation stops when some predefined maximum length is reached.
Properly setting up the token selection step and the stopping condition is essential to make your model behave as you’d expect on your task. That is why we have a GenerationConfig file associated with each model, which contains a good default generative parameterization and is loaded alongside your model.
Let’s talk code!
If you’re interested in basic LLM usage, our high-level Pipeline interface is a great starting point. However, LLMs often require advanced features like quantization and fine control of the token selection step, which is best done through g


Generate text

enerate(). Autoregressive generation with LLMs is also resource-intensive and should be executed on a GPU for adequate throughput.
First, you need to load the model.
  Copied >>> from transformers import AutoModelForCausalLM
>>> model = AutoModelForCausalLM.from_pretrained(
...     "openlm-research/open_llama_7b", device_map="auto", load_in_4bit=True
... )
You’ll notice two flags in the from_pretrained call:
device_map ensures the model is moved to your GPU(s) load_in_4bit applies 4-bit dynamic quantization to massively reduce the resource requirements
There are other ways to initialize a model, but this is a good baseline to begin with an LLM.
Next, you need to preprocess your text input with a tokenizer.
  Copied >>> from transformers import AutoTokenizer
>>> tokenizer = Auto


Generate text

Tokenizer.from_pretrained("openlm-research/open_llama_7b")
>>> model_inputs = tokenizer(["A list of colors: red, blue"], return_tensors="pt").to("cuda")
The model_inputs variable holds the tokenized text input, as well as the attention mask. While generate() does its best effort to infer the attention mask when it is not passed, we recommend passing it whenever possible for optimal results.
Finally, call the generate() method to returns the generated tokens, which should be converted to text before printing.
  Copied >>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'A list of colors: red, blue, green, yellow, black, white, and brown'
And that’s it! In a few lines of code, you can harness the power of an LL


Common pitfalls

There are many generation strategies, and sometimes the default values may not be appropriate for your use case. If your outputs aren’t aligned with what you’re expecting, we’ve created a list of the most common pitfalls and how to avoid them.
  Copied >>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("openlm-research/open_llama_7b")
>>> tokenizer.pad_token = tokenizer.eos_token  # Llama has no pad token by default
>>> model = AutoModelForCausalLM.from_pretrained(
...     "openlm-research/open_llama_7b", device_map="auto", load_in_4bit=True
... )


Generated output is too short/long

If not specified in the GenerationConfig file, generate returns up to 20 tokens by default. We highly recommend manually setting max_new_tokens in your generate call to control the maximum number of new tokens it can return. Keep in mind LLMs (more precisely, decoder-only models) also return the input prompt as part of the output.
  Copied >>> model_inputs = tokenizer(["A sequence of numbers: 1, 2"], return_tensors="pt").to("cuda")
>>> # By default, the output will contain up to 20 tokens
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'A sequence of numbers: 1, 2, 3, 4, 5'
>>> # Setting `max_new_tokens` allows you to control the maximum length
>>> generated_ids = model.generate(**model_inputs, max_new_tokens=50)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'A sequence of numbers: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,'


Incorrect generation mode

By default, and unless specified in the GenerationConfig file, generate selects the most likely token at each iteration (greedy decoding). Depending on your task, this may be undesirable; creative tasks like chatbots or writing an essay benefit from sampling. On the other hand, input-grounded tasks like audio transcription or translation benefit from greedy decoding. Enable sampling with do_sample=True, and you can learn more about this topic in this blog post.
  Copied >>> # Set seed or reproducibility -- you don't need this unless you want full reproducibility
>>> from transformer


Incorrect generation mode

s import set_seed
>>> set_seed(0)
>>> model_inputs = tokenizer(["I am a cat."], return_tensors="pt").to("cuda")
>>> # LLM + greedy decoding = repetitive, boring output
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'I am a cat. I am a cat. I am a cat. I am a cat'
>>> # With sampling, the output becomes more creative!
>>> generated_ids = model.generate(**model_inputs, do_sample=True)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'I am a cat.\nI just need to be. I am always.\nEvery time


Wrong padding side

LLMs are decoder-only architectures, meaning they continue to iterate on your input prompt. If your inputs do not have the same length, they need to be padded. Since LLMs are not trained to continue from pad tokens, your input needs to be left-padded. Make sure you also don’t forget to pass the attention mask to generate!
  Copied >>> # The tokenizer initialized above has right-padding active by default: the 1st sequence,
>>> # which is shorter, has padding on the right side. Generation fails.
>>> model_inputs = tokenizer(
...     ["1, 2, 3", "A, B, C, D, E"], padding=True, return_tensors="pt"
... ).to("cuda")


Wrong padding side

>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids[0], skip_special_tokens=True)[0]
''
>>> # With left-padding, it works as expected!
>>> tokenizer = AutoTokenizer.from_pretrained("openlm-research/open_llama_7b", padding_side="left")
>>> tokenizer.pad_token = tokenizer.eos_token  # Llama has no pad token by default
>>> model_inputs = tokenizer(
...     ["1, 2, 3", "A, B, C, D, E"], padding=True, return_tensors="pt"
... ).to("cuda")
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'1, 2, 3, 4, 5, 6,'


Further resources

While the autoregressive generation process is relatively straightforward, making the most out of your LLM can be a challenging endeavor because there are many moving parts. For your next steps to help you dive deeper into LLM usage and understanding:


Advanced generate usage

Guide on how to control different generation methods, how to set up the generation configuration file, and how to stream the output; API reference on GenerationConfig, generate(), and generate-related classes.


LLM leaderboards

Open LLM Leaderboard, which focuses on the quality of the open-source models; Open LLM-Perf Leaderboard, which focuses on LLM throughput.


Related libraries

text-generation-inference, a production-ready server for LLMs; optimum, an extension of 🤗 Transformers that optimizes for specific hardware devices.


Load pretrained instances with an AutoClass

With so many different Transformer architectures, it can be challenging to create one for your checkpoint. As a part of 🤗 Transformers core philosophy to make the library easy, simple and flexible to use, an AutoClass automatically infers and loads the correct architecture from a given checkpoint. The from_pretrained() method lets you quickly load a pretrained model for any architecture so you don’t have to devote time and resources to train a model from scratch. Producing this type of checkpoint-agnostic code means if your code works for one checkpoint, it will


 work with another checkpoint - as long as it was trained for a similar task - even if the architecture is different.
Remember, architecture refers to the skeleton of the model and checkpoints are the weights for a given architecture. For example, BERT is an architecture, while bert-base-uncased is a checkpoint. Model is a general term that can mean either architecture or checkpoint.
In this tutorial, learn to:
Load a pretrained tokenizer. Load a pretrained image processor Load a pretrained feature extractor. Load a pretrained processor. Load a pretrained model.


AutoTokenizer

Nearly every NLP task begins with a tokenizer. A tokenizer converts your input into a format that can be processed by the model.
Load a tokenizer with AutoTokenizer.from_pretrained():
  Copied >>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
Then tokenize your input as shown below:
  Copied >>> sequence = "In a hole in the ground there lived a hobbit."
>>> print(tokenizer(sequence))
{'input_ids': [101, 1999, 1037, 4920, 1999, 1996, 2598, 2045, 2973, 1037, 7570, 10322, 4183, 1012, 102],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}


AutoImageProcessor

For vision tasks, an image processor processes the image into the correct input format.
  Copied >>> from transformers import AutoImageProcessor
>>> image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224")


AutoFeatureExtractor

For audio tasks, a feature extractor processes the audio signal the correct input format.
Load a feature extractor with AutoFeatureExtractor.from_pretrained():
  Copied >>> from transformers import AutoFeatureExtractor
>>> feature_extractor = AutoFeatureExtractor.from_pretrained(
...     "ehcalabres/wav2vec2-lg-xlsr-en-speech-emotion-recognition"
... )


AutoProcessor

Multimodal tasks require a processor that combines two types of preprocessing tools. For example, the LayoutLMV2 model requires an image processor to handle images and a tokenizer to handle text; a processor combines both of them.
Load a processor with AutoProcessor.from_pretrained():
  Copied >>> from transformers import AutoProcessor
>>> processor = AutoProcessor.from_pretrained("microsoft/layoutlmv2-base-uncased")


AutoModel

Pytorch  Hide Pytorch content  Finally, the AutoModelFor classes let you load a pretrained model for a given task (see here for a complete list of available tasks). For example, load a model for sequence classification with AutoModelForSequenceClassification.from_pretrained():   Copied >>> from transformers import AutoModelForSequenceClassification
>>> model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased") Easily reuse the same checkpoint to load an architecture for a different task:   Copied >>> from transformers import AutoModelForTokenClassification
>>> model = AutoModelForTokenClassification.from_pretrained("distilbert-base-uncased") For PyTorch models, the from_pretrained() method uses torch.load() which internally uses pickle and is known to be insecure. In general, never load a model that could have come from an untrusted source, or that could have been ta


AutoModel

mpered with. This security risk is partially mitigated for public models hosted on the Hugging Face Hub, which are scanned for malware at each commit. See the Hub documentation for best practices like signed commit verification with GPG. TensorFlow and Flax checkpoints are not affected, and can be loaded within PyTorch architectures using the from_tf and from_flax kwargs for the from_pretrained method to circumvent this issue. Generally, we recommend using the AutoTokenizer class and the AutoModelFor class to load pretrained instances of models. This will ensure you load the correct architecture every time. In the next tutorial, learn how to use your newly loaded tokenizer, image processor, feature extractor and processor to preprocess a dataset for fine-tuning.  TensorFlow  Hide TensorFlow content  Finally, the TFAutoModelFor classes let you load a pretrained model for a given task (see here


AutoModel

for a complete list of available tasks). For example, load a model for sequence classification with TFAutoModelForSequenceClassification.from_pretrained():   Copied >>> from transformers import TFAutoModelForSequenceClassification
>>> model = TFAutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased") Easily reuse the same checkpoint to load an architecture for a different task:   Copied >>> from transformers import TFAutoModelForTokenClassification
>>> model = TFAutoModelForTokenClassification.from_pretrained("distilbert-base-uncased") Generally, we recommend using the AutoTokenizer class and the TFAutoModelFor class to load pretrained instances of models. This will ensure you load the correct architecture every time. In the next tutorial, learn how to use your newly loaded tokenizer, image processor, feature extractor and processor to preprocess a dataset for fine-tuning.


Attention mechanisms

Most transformer models use full attention in the sense that the attention matrix is square. It can be a big
computational bottleneck when you have long texts. Longformer and reformer are models that try to be more efficient and
use a sparse version of the attention matrix to speed up training.


LSH attention

Reformer uses LSH attention. In the softmax(QK^t), only the biggest elements (in the softmax
dimension) of the matrix QK^t are going to give useful contributions. So for each query q in Q, we can consider only
the keys k in K that are close to q. A hash function is used to determine if q and k are close. The attention mask is
modified to mask the current token (except at the first position), because it will give a query and a key equal (so
very similar to each other). Since the hash can be a bit random, several hash functions are used in practice
(determined by a n_rounds parameter) and then are averaged together.


Local attention

Longformer uses local attention: often, the local context (e.g., what are the two tokens to the
left and right?) is enough to take action for a given token. Also, by stacking attention layers that have a small
window, the last layer will have a receptive field of more than just the tokens in the window, allowing them to build a
representation of the whole sentence.
Some preselected input tokens are also given global attention: for those few tokens, the attention matrix can access
all tokens and this process is symmetric: all other tokens have access to those specific tokens (on top of the ones in
their local window). This is shown in Figure 2d of the paper, see below for a sample attention mask:
Using those attention matrices with less parameters then allows the model to have inputs having a bigger sequence
length.


Axial positional encodings

Reformer uses axial positional encodings: in traditional transformer models, the positional encoding
E is a matrix of size lll by ddd, lll being the sequence length and ddd the dimension of the
hidden state. If you have very long texts, this matrix can be huge and take way too much space on the GPU. To alleviate
that, axial positional encodings consist of factorizing that big matrix E in two smaller matrices E1 and E2, with
dimensions l1×d1l_{1} \times d_{1}l1×d1 and l2×d2l_{2} \times d_{2}l2×d2, such that l1×l2=ll_{1} \times l_{2} = ll1×l2=l and
d1+d2=dd_{1} + d_{2} = dd1+d2=d (with the product for the lengths, this ends up being way smaller). The embedding for time
step jjj in E is obtained by concatenating the embeddings for timestep j%l1j \% l1j%l1 in E1 and j//l1j // l1j//l1
in E2.


Checks on a Pull Request

When you open a pull request on 🤗 Transformers, a fair number of checks will be run to make sure the patch you are adding is not breaking anything existing. Those checks are of four types:
regular tests documentation build code and documentation style general repository consistency
In this document, we will take a stab at explaining what those various checks are and the reason behind them, as well as how to debug them locally if one of them fails on your PR.
Note that, ideally, they require you to have a


Checks on a Pull Request

dev install:
  Copied pip install transformers[dev]
or for an editable install:
  Copied pip install -e .[dev]
inside the Transformers repo. Since the number of optional dependencies of Transformers has grown a lot, it’s possible you don’t manage to get all of them. If the dev install fails, make sure to install the Deep Learning framework you are working with (PyTorch, TensorFlow and/or Flax) then do
  Copied pip install transformers[quality]
or for an editable install:
  Copied pip install -e .[quality]


Tests

All the jobs that begin with ci/circleci: run_tests_ run parts of the Transformers testing suite. Each of those jobs focuses on a part of the library in a certain environment: for instance ci/circleci: run_tests_pipelines_tf runs the pipelines test in an environment where TensorFlow only is installed.
Note that to avoid running tests when there is no real change in the modules they are testing, only part of the test suite is run each time: a utility is run to determine the differences in the library between before and after the PR (what GitHub shows you in the “Files changes” tab) and picks the tests impacted by that diff. That utility can be run locally with:
  Copied python utils/tests_fetcher.py
from the root of the Transformers repo. It will:
Check for each file in the diff if the changes are in the code or only in comments or docstrings. Only the files with real code changes are kept. Build an internal map that g


Tests

ives for each file of the source code of the library all the files it recursively impacts. Module A is said to impact module B if module B imports module A. For the recursive impact, we need a chain of modules going from module A to module B in which each module imports the previous one. Apply this map on the files gathered in step 1, which  gives us the list of model files impacted by the PR. Map each of those files to their corresponding test file(s) and get the list of tests to run.
When executing the script locally, you should get the results of step 1, 3 and 4 printed and thus know which tests are run. The script will also create a file named test_list.txt which contains the list of tests to run, and you can run them locally with the following command:
  Copied python -m pytest -n 8 --dist=loadfile -rA -s $(cat test_list.txt)
Just in case anything slipped through the cracks, the full test suite is also run daily.


Documentation build

The build_pr_documentation job builds and generates a preview of the documentation to make sure everything looks okay once your PR is merged. A bot will add a link to preview the documentation in your PR. Any changes you make to the PR are automatically updated in the preview. If the documentation fails to build, click on Details next to the failed job to see where things went wrong. Often, the error is as simple as a missing file in the toctree.
If you’re interested in building or previewing the documentation locally, take a look at the README.md in the docs folder.


Code and documentation style

Code formatting is applied to all the source files, the examples and the tests using black and ruff. We also have a custom tool taking care of the formatting of docstrings and rst files (utils/style_doc.py), as well as the order of the lazy imports performed in the Transformers __init__.py files (utils/custom_init_isort.py). All of this can be launched by executing
  Copied make style
The CI checks those have been applied inside the ci/circleci: check_code_quality check. It also runs ruff, that will have a basic look at your code and will complain if it finds an undefined variable, or one that is not used. To run that check locally, use
  Copied make quality
This can take a lot of time, so to run the same thing on only the files you modified in the current branch, run
  Copied make fixup
This last command will also run all the additional checks for the repository consistency. Let’s have a look at them.


Repository consistency

This regroups all the tests to make sure your PR leaves the repository in a good state, and is performed by the ci/circleci: check_repository_consistency check. You can locally run that check by executing the following:
  Copied make repo-consistency
This checks that:
All objects added to the init are documented (performed by utils/check_repo.py) All __init__.py files have the same content in their two sections (performed by utils/check_inits.py) All code identified as a copy from another module is consistent with the original (performed by utils/check_copies.py) All configuration classes have at least one valid checkpoint mentioned in their docstrings (performed by utils/check_config_docstrings.py) All configuration classes only contain attributes that are used in corresponding modeling files (performed by utils/check_config_attributes.py)


Repository consistency

 The translations of the READMEs and the index of the doc have the same model list as the main README (performed by utils/check_copies.py) The auto-generated tables in the documentation are up to date (performed by utils/check_table.py) The library has all objects available even if not all optional dependencies are installed (performed by utils/check_dummies.py) All docstrings properly document the arguments in the signature of the object (performed by utils/check_docstrings.py)
Should this check fail, the first two items require manual fixing, the last four can be fixed automatically for you by running the command
  Copied make fix-copies
Additional checks concern PRs that add new models, mainly that:
All models added are in an Auto-mapping (performed by utils/check_repo.py) All models are properly tested (performed by utils/check_repo.py)


Check copies

Since the Transformers library is very opinionated with respect to model code, and each model should fully be implemented in a single file without relying on other models, we have added a mechanism that checks whether a copy of the code of a layer of a given model stays consistent with the original. This way, when there is a bug fix, we can see all other impacted models and choose to trickle down the modification or break the copy.
If a file is a full copy of another file, you should register it in the constant FULL_COPIES of utils/check_copies.py.
This mechanism relies on comments of the form # Copied from xxx. The xxx should contain the whole path to the class of function which is being copied below. For instance, RobertaSelfOutput is a direct copy of the BertSelfOutput class, so you can see here it has


Check copies

a comment:
  Copied # Copied from transformers.models.bert.modeling_bert.BertSelfOutput
Note that instead of applying this to a whole class, you can apply it to the relevant methods that are copied from. For instance here you can see how RobertaPreTrainedModel._init_weights is copied from the same method in BertPreTrainedModel with the comment:
  Copied # Copied from transformers.models.bert.modeling_bert.BertPreTrainedModel._init_weights
Sometimes the copy is exactly the same except for names: for instance in RobertaAttention, we use RobertaSelfAttention insted of BertSelfAttention but other than that, the code is exactly the same. This is why # Copied from supports simple string replacements with the follwoing syntax: Copied from xxx with foo->bar. This means the code is copied with all instances of foo


Check copies

being replaced by bar. You can see how it used here in RobertaAttention with the comment:
  Copied # Copied from transformers.models.bert.modeling_bert.BertAttention with Bert->Roberta
Note that there shouldn’t be any spaces around the arrow (unless that space is part of the pattern to replace of course).
You can add several patterns separated by a comma. For instance here CamemberForMaskedLM is a direct copy of RobertaForMaskedLM with two replacements: Roberta to Camembert and ROBERTA to CAMEMBERT. You can see here this is done with the comment:
  Copied # Copied from transformers.models.roberta.modeling_roberta.RobertaForMaskedLM with Roberta->Camembert, ROBERTA->CAMEMBERT
If the order matters (because one of the replacements might conflict with a previous one), the replacements are executed from left to


Check copies

 right.
If the replacements change the formatting (if you replace a short name by a very long name for instance), the copy is checked after applying the auto-formatter.
Another way when the patterns are just different casings of the same replacement (with an uppercased and a lowercased variants) is just to add the option all-casing. Here is an example in MobileBertForSequenceClassification with the comment:
  Copied # Copied from transformers.models.bert.modeling_bert.BertForSequenceClassification with Bert->MobileBert all-casing
In this case, the code is copied from BertForSequenceClassification by replacing:
Bert by MobileBert (for instance when using MobileBertModel in the init) bert by mobilebert (for instance when defining self.mobilebert) BERT by MOBILEBERT (in the constant MOBILEBERT_INPUTS_DOCSTRIN


Train with a script

Along with the 🤗 Transformers notebooks, there are also example scripts demonstrating how to train a model for a task with PyTorch, TensorFlow, or JAX/Flax.
You will also find scripts we’ve used in our research projects and legacy examples which are mostly community contributed. These scripts are not actively maintained and require a specific version of 🤗 Transformers that will most likely be incompatible with the latest version of the library.
The example scripts are not expected to work out-of-the-box on every problem, and you may need to adapt the script to the problem you’re trying to sol


Train with a script

ve. To help you with this, most of the scripts fully expose how data is preprocessed, allowing you to edit it as necessary for your use case.
For any feature you’d like to implement in an example script, please discuss it on the forum or in an issue before submitting a Pull Request. While we welcome bug fixes, it is unlikely we will merge a Pull Request that adds more functionality at the cost of readability.
This guide will show you how to run an example summarization training script in PyTorch and TensorFlow. All examples are expected to work with both frameworks unless otherwise specified.


Setup

To successfully run the latest version of the example scripts, you have to install 🤗 Transformers from source in a new virtual environment:
  Copied git clone https://github.com/huggingface/transformers
cd transformers
pip install .
For older versions of the example scripts, click on the toggle below:
Examples for older versions of 🤗 Transformers v4.5.1 v4.4.2 v4.3.3 v4.2.2 v4.1.1 v4.0.1 v3.5.1 v3.4.0 v3.3.1 v3.2.0 v3.1.0 v3.0.2 v2.11.0 v2.10.0 v2.9.1 v2.8.0 v2.7.0 v2.6.0 v2.5.1 v2.4.0 v2.3.0 v2.2.0 v2.1.1 v2.0.0 v1.2.0 v1.1.0 v1.0.0
Then switch your current clone of 🤗 Transformers to a specific version, like v3.5.1 for example:
  Copied git checkout tags/v3.5.1
After you’ve setup the correct library version, navigate to the example folder of your choice and install the example specific requirements:
  Copied pip install -r requirements.txt


Run a script

Pytorch  Hide Pytorch content  The example script downloads and preprocesses a dataset from the 🤗 Datasets library. Then the script fine-tunes a dataset with the Trainer on an architecture that supports summarization. The following example shows how to fine-tune T5-small on the CNN/DailyMail dataset. The T5 model requires an additional source_prefix argument due to how it was trained. This prompt lets T5 know this is a summarization task.   Copied python examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir


Run a script

 \
    --predict_with_generate  TensorFlow  Hide TensorFlow content  The example script downloads and preprocesses a dataset from the 🤗 Datasets library. Then the script fine-tunes a dataset using Keras on an architecture that supports summarization. The following example shows how to fine-tune T5-small on the CNN/DailyMail dataset. The T5 model requires an additional source_prefix argument due to how it was trained. This prompt lets T5 know this is a summarization task.   Copied python examples/tensorflow/summarization/run_summarization.py  \
    --model_name_or_path t5-small \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --output_dir /tmp/tst-summarization  \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 16 \
    --num_train_epochs 3 \
    --do_train \
    --do_eva


Distributed training and mixed precision

The Trainer supports distributed training and mixed precision, which means you can also use it in a script. To enable both of these features:
Add the fp16 argument to enable mixed precision. Set the number of GPUs to use with the nproc_per_node argument.
  Copied python -m torch.distributed.launch \
    --nproc_per_node 8 pytorch/summarization/run_summarization.py \
    --fp16 \
    --model_name_or_path t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
TensorFlow scripts utilize a MirroredStrategy for distributed training, and you don’t need to add any additional arguments to the training script. The TensorFlow script will use multiple GPUs by default if they are available.


Run a script on a TPU

Pytorch  Hide Pytorch content  Tensor Processing Units (TPUs) are specifically designed to accelerate performance. PyTorch supports TPUs with the XLA deep learning compiler (see here for more details). To use a TPU, launch the xla_spawn.py script and use the num_cores argument to set the number of TPU cores you want to use.   Copied python xla_spawn.py --num_cores 8 \
    summarization/run_summarization.py \
    --model_name_or_path t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_e


Run a script on a TPU

val_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate  TensorFlow  Hide TensorFlow content  Tensor Processing Units (TPUs) are specifically designed to accelerate performance. TensorFlow scripts utilize a TPUStrategy for training on TPUs. To use a TPU, pass the name of the TPU resource to the tpu argument.   Copied python run_summarization.py  \
    --tpu name_of_tpu_resource \
    --model_name_or_path t5-small \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --output_dir /tmp/tst-summarization  \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 16 \
    --num_train_epochs 3 \
    --do_train \
    --do_eval


Run a script with 🤗 Accelerate

🤗 Accelerate is a PyTorch-only library that offers a unified method for training a model on several types of setups (CPU-only, multiple GPUs, TPUs) while maintaining complete visibility into the PyTorch training loop. Make sure you have 🤗 Accelerate installed if you don’t already have it:
Note: As Accelerate is rapidly developing, the git version of accelerate must be installed to run the scripts   Copied pip install git+https://github.com/huggingface/accelerate
Instead of the run_summarization.py script, you need to use the run_summarization_no_trainer.p


Run a script with 🤗 Accelerate

y script. 🤗 Accelerate supported scripts will have a task_no_trainer.py file in the folder. Begin by running the following command to create and save a configuration file:
  Copied accelerate config
Test your setup to make sure it is configured correctly:
  Copied accelerate test
Now you are ready to launch the training:
  Copied accelerate launch run_summarization_no_trainer.py \
    --model_name_or_path t5-small \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir ~/tmp/tst-summarizatio


Use a custom dataset

The summarization script supports custom datasets as long as they are a CSV or JSON Line file. When you use your own dataset, you need to specify several additional arguments:
train_file and validation_file specify the path to your training and validation files. text_column is the input text to summarize. summary_column is the target text to output.
A summarization script using a custom dataset would look like this:
  Copied python examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path t5-small \
    --do_train \
    --do_eval \
    --train_file path_to_csv_or_jsonlines_file \
    --validation_file path_to_csv_or_jsonlines_file \
    --text_column text_column_name \
    --summary_column summary_column_name \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --overwrite_output_dir \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --predict_with_generate


Test a script

It is often a good idea to run your script on a smaller number of dataset examples to ensure everything works as expected before committing to an entire dataset which may take hours to complete. Use the following arguments to truncate the dataset to a maximum number of samples:
max_train_samples max_eval_samples max_predict_samples
  Copied python examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path t5-small \
    --max_train_samples 50 \
    --max_eval_samples 50 \
    --max_predict_samples 50 \


Test a script

   --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate
Not all example scripts support the max_predict_samples argument. If you aren’t sure whether your script supports this argument, add the -h argument to check:
  Copied examples/pytorch/summarization/run_summarization.py -


Resume training from checkpoint

Another helpful option to enable is resuming training from a previous checkpoint. This will ensure you can pick up where you left off without starting over if your training gets interrupted. There are two methods to resume training from a checkpoint.
The first method uses the output_dir previous_output_dir argument to resume training from the latest checkpoint stored in output_dir. In this case, you should remove overwrite_output_dir:
  Copied python examples/pytorch/summarization/run_summarization.py
    --model_name_or_path t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_


Resume training from checkpoint

train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --output_dir previous_output_dir \
    --predict_with_generate
The second method uses the resume_from_checkpoint path_to_specific_checkpoint argument to resume training from a specific checkpoint folder.
  Copied python examples/pytorch/summarization/run_summarization.py
    --model_name_or_path t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --resume_from_checkpoint path_to_specific_checkpoint \
    --predict_with_generate


Share your model

All scripts can upload your final model to the Model Hub. Make sure you are logged into Hugging Face before you begin:
  Copied huggingface-cli login
Then add the push_to_hub argument to the script. This argument will create a repository with your Hugging Face username and the folder name specified in output_dir.
To give your repository a specific name, use the push_to_hub_model_id argument to add it. The repository will be automatically listed under your namespace.
The following example shows how to upload a model


Share your model

with a specific repository name:
  Copied python examples/pytorch/summarization/run_summarization.py
    --model_name_or_path t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --push_to_hub \
    --push_to_hub_model_id finetuned-t5-cnn_dailymail \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate


Create a custom architecture

An AutoClass automatically infers the model architecture and downloads pretrained configuration and weights. Generally, we recommend using an AutoClass to produce checkpoint-agnostic code. But users who want more control over specific model parameters can create a custom 🤗 Transformers model from just a few base classes. This could be particularly useful for anyone who is interested in studying, training or experimenting with a 🤗 Transformers model. In this guide, dive deeper into creating a custom model without an AutoClass. Learn how to:
Load and customize a model configuration. Create a model architecture. Create a slow and fast tokenizer for text. Create an image processor for vision tasks. Create a feature extractor for audio tasks. Create a processor for multimodal tasks.


Configuration

A configuration refers to a model’s specific attributes. Each model configuration has different attributes; for instance, all NLP models have the hidden_size, num_attention_heads, num_hidden_layers and vocab_size attributes in common. These attributes specify the number of attention heads or hidden layers to construct a model with.
Get a closer look at DistilBERT by accessing DistilBertConfig to inspect it’s attributes:
  Copied >>> from transformers import DistilBertConfig
>>> config = DistilBertConfig()
>>> print(config)
DistilBertConfig {
  "activation": "gelu",
  "attention_dropout": 0.1,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "initializer_range": 0.02,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "transfor


Configuration

mers_version": "4.16.2",
  "vocab_size": 30522
}
DistilBertConfig displays all the default attributes used to build a base DistilBertModel. All attributes are customizable, creating space for experimentation. For example, you can customize a default model to:
Try a different activation function with the activation parameter. Use a higher dropout ratio for the attention probabilities with the attention_dropout parameter.
  Copied >>> my_config = DistilBertConfig(activation="relu", attention_dropout=0.4)
>>> print(my_config)
DistilBertConfig {
  "activation": "relu",
  "attention_dropout": 0.4,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "initializer_range": 0.02,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "transfor


Configuration

mers_version": "4.16.2",
  "vocab_size": 30522
}
Pretrained model attributes can be modified in the from_pretrained() function:
  Copied >>> my_config = DistilBertConfig.from_pretrained("distilbert-base-uncased", activation="relu", attention_dropout=0.4)
Once you are satisfied with your model configuration, you can save it with save_pretrained(). Your configuration file is stored as a JSON file in the specified save directory:
  Copied >>> my_config.save_pretrained(save_directory="./your_model_save_path")
To reuse the configuration file, load it with from_pretrained():
  Copied >>> my_config = DistilBertConfig.from_pretrained("./your_model_save_path/config.json")
You can also save your configuration file as a dictionary or even just the difference between your custom configuration attributes and the default configuration attributes! See the configuration documentation for more details


Model

The next step is to create a model. The model - also loosely referred to as the architecture - defines what each layer is doing and what operations are happening. Attributes like num_hidden_layers from the configuration are used to define the architecture. Every model shares the base class PreTrainedModel and a few common methods like resizing input embeddings and pruning self-attention heads. In addition, all models are also either a torch.nn.Module, tf.keras.Model or flax.linen.Module subclass. This means models are compatible with each of their respective framework’s usage.
  Pytorch  Hide Pytorch content  Load your custom configuration attributes into the model:   Copied >>> from transformers import DistilBertModel
>>> my_config = DistilBertConfig.from_pretrained("./your_model_save_path/config.json")
>>> model = DistilBertModel(my_config) This creates a model with random values instead of pretrained weights. You won’t be


Model

able to use this model for anything useful yet until you train it. Training is a costly and time-consuming process. It is generally better to use a pretrained model to obtain better results faster, while using only a fraction of the resources required for training. Create a pretrained model with from_pretrained():   Copied >>> model = DistilBertModel.from_pretrained("distilbert-base-uncased") When you load pretrained weights, the default model configuration is automatically loaded if the model is provided by 🤗 Transformers. However, you can still replace - some or all of - the default model configuration attributes with your own if you’d like:   Copied >>> model = DistilBertModel.from_pretrained("distilbert-base-uncased", config=my_config)  TensorFlow  Hide TensorFlow content  Load your custom configuration attributes into the model:   Copied >>> from transformers import TFDistilBertModel
>>> my_config = DistilBertConfig.from


Model

_pretrained("./your_model_save_path/my_config.json")
>>> tf_model = TFDistilBertModel(my_config) This creates a model with random values instead of pretrained weights. You won’t be able to use this model for anything useful yet until you train it. Training is a costly and time-consuming process. It is generally better to use a pretrained model to obtain better results faster, while using only a fraction of the resources required for training. Create a pretrained model with from_pretrained():   Copied >>> tf_model = TFDistilBertModel.from_pretrained("distilbert-base-uncased") When you load pretrained weights, the default model configuration is automatically loaded if the model is provided by 🤗 Transformers. However, you can still replace - some or all of - the default model configuration attributes with your own if you’d like:   Copied >>> tf_model = TFDistilBertModel.from_pretrained("distilbert-base-uncased", config=my_config


Model heads

At this point, you have a base DistilBERT model which outputs the hidden states. The hidden states are passed as inputs to a model head to produce the final output. 🤗 Transformers provides a different model head for each task as long as a model supports the task (i.e., you can’t use DistilBERT for a sequence-to-sequence task like translation).
  Pytorch  Hide Pytorch content  For example, DistilBertForSequenceClassification is a base DistilBERT model with a sequence classification head. The sequence classification head is a linear layer on top of the pooled outputs.   Copied >>> from transformers import DistilBertForSequenceClassification
>>> model = DistilBertForSequenceClassification.fro


Model heads

m_pretrained("distilbert-base-uncased") Easily reuse this checkpoint for another task by switching to a different model head. For a question answering task, you would use the DistilBertForQuestionAnswering model head. The question answering head is similar to the sequence classification head except it is a linear layer on top of the hidden states output.   Copied >>> from transformers import DistilBertForQuestionAnswering
>>> model = DistilBertForQuestionAnswering.from_pretrained("distilbert-base-uncased")  TensorFlow  Hide TensorFlow content  For example, TFDistilBertForSequenceClassification is a base DistilBERT model with a sequence classification head. The sequence classification head


Model heads

is a linear layer on top of the pooled outputs.   Copied >>> from transformers import TFDistilBertForSequenceClassification
>>> tf_model = TFDistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased") Easily reuse this checkpoint for another task by switching to a different model head. For a question answering task, you would use the TFDistilBertForQuestionAnswering model head. The question answering head is similar to the sequence classification head except it is a linear layer on top of the hidden states output.   Copied >>> from transformers import TFDistilBertForQuestionAnswering
>>> tf_model = TFDistilBertForQuestionAnswering.from_pretrained("distilbert-base-uncased


Tokenizer

The last base class you need before using a model for textual data is a tokenizer to convert raw text to tensors. There are two types of tokenizers you can use with 🤗 Transformers:
PreTrainedTokenizer: a Python implementation of a tokenizer. PreTrainedTokenizerFast: a tokenizer from our Rust-based 🤗 Tokenizer library. This tokenizer type is significantly faster - especially during batch tokenization - due to its Rust implementation. The fast tokenizer also offers additional methods like offset mapping which maps tokens to their original words or characters.
Both tokenizers support common methods such as encoding and decoding, adding new tokens, and managing special tokens.
Not every model supports a fast tokenizer. Take a look at this table to check if a model has fast tokenizer support.
If you trained your own tokenizer, you can create one from your vocabulary file:
  Copied >>> from transformers import DistilBertTokenizer
>>> my_tokenizer


Tokenizer

 = DistilBertTokenizer(vocab_file="my_vocab_file.txt", do_lower_case=False, padding_side="left")
It is important to remember the vocabulary from a custom tokenizer will be different from the vocabulary generated by a pretrained model’s tokenizer. You need to use a pretrained model’s vocabulary if you are using a pretrained model, otherwise the inputs won’t make sense. Create a tokenizer with a pretrained model’s vocabulary with the DistilBertTokenizer class:
  Copied >>> from transformers import DistilBertTokenizer
>>> slow_tokenizer = DistilBertTokenizer.from_pretrained("distilbert-base-uncased")
Create a fast tokenizer with the DistilBertTokenizerFast class:
  Copied >>> from transformers import DistilBertTokenizerFast
>>> fast_tokenizer = DistilBertTokenizerFast.from_pretrained("distilbert-base-uncased")
By default, AutoTokenizer will try to load a fast tokenizer. You can disable this behavior by setting use_fast=False in from_pretrained


Image Processor

An image processor processes vision inputs. It inherits from the base ImageProcessingMixin class.
To use, create an image processor associated with the model you’re using. For example, create a default ViTImageProcessor if you are using ViT for image classification:
  Copied >>> from transformers import ViTImageProcessor
>>> vit_extractor = ViTImageProcessor()
>>> print(vit_extractor)
ViTImageProcessor {
  "do_normalize": true,
  "do_resize": true,
  "image_processor_type": "ViTImageProcessor",
  "image_mean": [
    0.5,
    0.5,
    0.5
  ],
  "image_std": [
    0.5,
    0.5,
    0.5
  ],
  "resample": 2,
  "size": 224
}
If you aren’t looking f


Image Processor

or any customization, just use the from_pretrained method to load a model’s default image processor parameters.
Modify any of the ViTImageProcessor parameters to create your custom image processor:
  Copied >>> from transformers import ViTImageProcessor
>>> my_vit_extractor = ViTImageProcessor(resample="PIL.Image.BOX", do_normalize=False, image_mean=[0.3, 0.3, 0.3])
>>> print(my_vit_extractor)
ViTImageProcessor {
  "do_normalize": false,
  "do_resize": true,
  "image_processor_type": "ViTImageProcessor",
  "image_mean": [
    0.3,
    0.3,
    0.3
  ],
  "image_std": [
    0.5,
    0.5,
    0.5
  ],
  "resample": "PIL.Image.BOX",
  "size": 224
}


Feature Extractor

A feature extractor processes audio inputs. It inherits from the base FeatureExtractionMixin class, and may also inherit from the SequenceFeatureExtractor class for processing audio inputs.
To use, create a feature extractor associated with the model you’re using. For example, create a default Wav2Vec2FeatureExtractor if you are using Wav2Vec2 for audio classification:
  Copied >>> from transformers import Wav2Vec2FeatureExtractor
>>> w2v2_extractor = Wav2Vec2FeatureExtractor()
>>> print(w2v2_extractor)
Wav2Vec2FeatureExtractor {
  "do_normalize": true,
  "feature_extractor_type": "Wav2Vec2FeatureExtractor",
  "feature_size": 1,
  "padding_side": "right",
  "padding_value": 0.0,
  "return


Feature Extractor

_attention_mask": false,
  "sampling_rate": 16000
}
If you aren’t looking for any customization, just use the from_pretrained method to load a model’s default feature extractor parameters.
Modify any of the Wav2Vec2FeatureExtractor parameters to create your custom feature extractor:
  Copied >>> from transformers import Wav2Vec2FeatureExtractor
>>> w2v2_extractor = Wav2Vec2FeatureExtractor(sampling_rate=8000, do_normalize=False)
>>> print(w2v2_extractor)
Wav2Vec2FeatureExtractor {
  "do_normalize": false,
  "feature_extractor_type": "Wav2Vec2FeatureExtractor",
  "feature_size": 1,
  "padding_side": "right",
  "padding_value": 0.0,
  "return_attention_mask": false,
  "sampling_rate": 8000


Processor

For models that support multimodal tasks, 🤗 Transformers offers a processor class that conveniently wraps processing classes such as a feature extractor and a tokenizer into a single object. For example, let’s use the Wav2Vec2Processor for an automatic speech recognition task (ASR). ASR transcribes audio to text, so you will need a feature extractor and a tokenizer.
Create a feature extractor to handle the audio inputs:
  Copied >>> from transformers import Wav2Vec2FeatureExtractor
>>> feature_extractor = Wav2Vec2FeatureExtractor(padding_value=1.0, do_normalize=True)
Create a tokenizer to handle the text inputs:
  Copied >>> from transformers import Wav2Vec2CTCTokenizer
>>> t


Processor

okenizer = Wav2Vec2CTCTokenizer(vocab_file="my_vocab_file.txt")
Combine the feature extractor and tokenizer in Wav2Vec2Processor:
  Copied >>> from transformers import Wav2Vec2Processor
>>> processor = Wav2Vec2Processor(feature_extractor=feature_extractor, tokenizer=tokenizer)
With two basic classes - configuration and model - and an additional preprocessing class (tokenizer, image processor, feature extractor, or processor), you can create any of the models supported by 🤗 Transformers. Each of these base classes are configurable, allowing you to use the specific attributes you want. You can easily setup a model for training or modify an existing pretrained model to fine-tune


Perplexity of fixed-length models

Perplexity (PPL) is one of the most common metrics for evaluating language models. Before diving in, we should note
that the metric applies specifically to classical language models (sometimes called autoregressive or causal language
models) and is not well defined for masked language models like BERT (see summary of the models).
Perplexity is defined as the exponentiated average negative log-likelihood of a sequence. If we have a tokenized
sequence X=(x0,x1,…,xt)X = (x_0, x_1, \dots, x_t)X=(x0,x1,…,xt), then the perplexity of XXX is,
PPL(X)=exp{−1t∑itlogpθ(xi∣x<i)}\text{PPL}(X) = \exp \left\{ {-\frac{1}{t}\sum_i^t \log p_\theta (x_i|x_{<i}) } \right\}PPL(X)=exp{−t1i∑tlogpθ(xi∣x<i)}
where logpθ(xi∣x<i)\log p_\the


Perplexity of fixed-length models

ta (x_i|x_{<i})logpθ(xi∣x<i) is the log-likelihood of the ith token conditioned on the preceding tokens x<ix_{<i}x<i according to our model. Intuitively, it can be thought of as an evaluation of the model’s ability to predict uniformly among the set of specified tokens in a corpus. Importantly, this means that the tokenization procedure has a direct impact on a model’s perplexity which should always be taken into consideration when comparing different models.
This is also equivalent to the exponentiation of the cross-entropy between the data and model predictions. For more
intuition about perplexity and its relationship to Bits Per Character (BPC) and data compression, check out this
fantastic blog post on The Gradient.


Calculating PPL with fixed-length models

If we weren’t limited by a model’s context size, we would evaluate the model’s perplexity by autoregressively
factorizing a sequence and conditioning on the entire preceding subsequence at each step, as shown below.
When working with approximate models, however, we typically have a constraint on the number of tokens the model can
process. The largest version of GPT-2, for example, has a fixed length of 1024 tokens, so we
cannot calculate pθ(xt∣x<t)p_\theta(x_t|x_{<t})pθ(xt∣x<t) directly when ttt is greater than 1024.
Instead, the sequence is typically broken into subsequences equal to the model’s maximum input size. If a model’s max
input size is kkk, we th


en approximate the likelihood of a token xtx_txt by conditioning only on the
k−1k-1k−1 tokens that precede it rather than the entire context. When evaluating the model’s perplexity of a
sequence, a tempting but suboptimal approach is to break the sequence into disjoint chunks and add up the decomposed
log-likelihoods of each segment independently.
This is quick to compute since the perplexity of each segment can be computed in one forward pass, but serves as a poor
approximation of the fully-factorized perplexity and will typically yield a higher (worse) PPL because the model will
have less context at most of the prediction steps.
Instead, the PPL of fixed-le