Exporting a 🤗 Transformers model to ONNX with optimum.onnxruntime

Alternative to CLI, you can export a 🤗 Transformers model to ONNX programmatically like so:
  Copied >>> from optimum.onnxruntime import ORTModelForSequenceClassification
>>> from transformers import AutoTokenizer
>>> model_checkpoint = "distilbert_base_uncased_squad"
>>> save_directory = "onnx/"
>>> # Load a model from transformers and export it to ONNX
>>> ort_model = ORTModelForSequenceClassification.from_pretrained(model_checkpoint, export=True)
>>> tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
>>> # Save the onnx model and tokenizer
>>> ort_model.save_pretrained(save_directory)
>>> tokenizer.save_pretrained(save_directory)


Exporting a model for an unsupported architecture

If you wish to contribute by adding support for a model that cannot be currently exported, you should first check if it is
supported in optimum.exporters.onnx,
and if it is not, contribute to 🤗 Optimum
directly.


Exporting a model with transformers.onnx

tranformers.onnx is no longer maintained, please export models with 🤗 Optimum as described above. This section will be removed in the future versions.
To export a 🤗 Transformers model to ONNX with tranformers.onnx, install extra dependencies:
  Copied pip install transformers[onnx]
Use transformers.onnx package as a Python module to export a checkpoint using a ready-made configuration:
  Copied python -m transformers.onnx --model=distilbert-base-uncased onnx/
This exports an ONNX graph of the checkpoint defined by the --model argument. Pass any checkpoint on the 🤗 Hub or one that’s stored locally.
The resulting model.onnx file can then be run on one of the many accelerators that support the ONNX standard.


For example,
load and run the model with ONNX Runtime as follows:
  Copied >>> from transformers import AutoTokenizer
>>> from onnxruntime import InferenceSession
>>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
>>> session = InferenceSession("onnx/model.onnx")
>>> # ONNX Runtime expects NumPy arrays as input
>>> inputs = tokenizer("Using DistilBERT with ONNX Runtime!", return_tensors="np")
>>> outputs = session.run(output_names=["last_hidden_state"], input_feed=dict(inputs))
The required output names (like ["last_hidden_state"]) can be obtained by taking a look at the ONNX configuration of
each model. For example, for DistilBERT we have:
  Copied >>> from transformers.models.distil


bert import DistilBertConfig, DistilBertOnnxConfig
>>> config = DistilBertConfig()
>>> onnx_config = DistilBertOnnxConfig(config)
>>> print(list(onnx_config.outputs.keys()))
["last_hidden_state"]
The process is identical for TensorFlow checkpoints on the Hub. For example, export a pure TensorFlow checkpoint like so:
  Copied python -m transformers.onnx --model=keras-io/transformers-qa onnx/
To export a model that’s stored locally, save the model’s weights and tokenizer files in the same directory (e.g. local-pt-checkpoint),
then export it to ONNX by pointing the --model argument of the transformers.onnx package to the desired directory:
  Copied python -m transformers.onnx --model=local-pt-checkpoint onnx/


Padding and truncation

Batched inputs are often different lengths, so they can’t be converted to fixed-size tensors. Padding and truncation are strategies for dealing with this problem, to create rectangular tensors from batches of varying lengths. Padding adds a special padding token to ensure shorter sequences will have the same length as either the longest sequence in a batch or the maximum length accepted by the model. Truncation works in the other direction by truncating long sequences.
In most cases, padding your batch to the length of the longest sequence and truncating to the maximum length a model can accept works pretty well. However, the API supports more strategies if you need them. The three arguments you need to are: padding, truncation and max_length.
The padding argument controls padding. It can be a boolean or a string:
True or 'longest': pad to the longest sequence in the batch (no padding is applied if you only provide
a single seque


Padding and truncation

nce). 'max_length': pad to a length specified by the max_length argument or the maximum length accepted
by the model if no max_length is provided (max_length=None). Padding will still be applied if you only provide a single sequence. False or 'do_not_pad': no padding is applied. This is the default behavior.
The truncation argument controls truncation. It can be a boolean or a string:
True or 'longest_first': truncate to a maximum length specified by the max_length argument or
the maximum length accepted by the model if no max_length is provided (max_length=None). This will
truncate token by token, removing a token from the longest sequence in the pair until the proper length is
reached. 'only_second': truncate to a maximum length specified by the max_length argument or the maximum
length accepted by the model if no max_length is provided (max_length=None). This will only truncate
the second sentence of a pair if a pair of sequen


Padding and truncation

ces (or a batch of pairs of sequences) is provided. 'only_first': truncate to a maximum length specified by the max_length argument or the maximum
length accepted by the model if no max_length is provided (max_length=None). This will only truncate
the first sentence of a pair if a pair of sequences (or a batch of pairs of sequences) is provided. False or 'do_not_truncate': no truncation is applied. This is the default behavior.
The max_length argument controls the length of the padding and truncation. It can be an integer or None, in which case it will default to the maximum length the model can accept. If the model has no specific maximum input length, truncation or padding to max_length is deactivated.
The following table summarizes the recommended way to setup padding and truncation. If you use pairs of input sequences in any of the following examples, you can replace truncation=True by a STRATEGY selected in
['only_first', 'o


Padding and truncation

nly_second', 'longest_first'], i.e. truncation='only_second' or truncation='longest_first' to control how both sequences in the pair are truncated as detailed before.
| Truncation                           | Padding                           | Instruction                                                                          |
|--------------------------------------|-----------------------------------|--------------------------------------------------------------------------------------|
| no truncation                        | no padding                        | tokenizer(batch_sentences)                                                           |
| nan                                  | padding to max sequence in batch  | tokenizer(batch_sentences, padding=True)  or                                         |
| nan                                  | nan                               | tokenizer(batch_sentences, padding='longest


Padding and truncation

')                                        |
| nan                                  | padding to max model input length | tokenizer(batch_sentences, padding='max_length')                                     |
| nan                                  | padding to specific length        | tokenizer(batch_sentences, padding='max_length', max_length=42)                      |
| nan                                  | padding to a multiple of a value  | `tokenizer(batch_sentences, padding=True, pad_to_multiple_of=8)                      |
| truncation to max model input length | no padding                        | tokenizer(batch_sentences, truncation=True)  or                                      |
| nan                                  | nan                               | tokenizer(batch_sentences, truncation=STRATEGY)                                      |
| nan                                  | padding to max sequence in batch  | to


Padding and truncation

kenizer(batch_sentences, padding=True, truncation=True)  or                        |
| nan                                  | nan                               | tokenizer(batch_sentences, padding=True, truncation=STRATEGY)                        |
| nan                                  | padding to max model input length | tokenizer(batch_sentences, padding='max_length', truncation=True)  or                |
| nan                                  | nan                               | tokenizer(batch_sentences, padding='max_length', truncation=STRATEGY)                |
| nan                                  | padding to specific length        | Not possible                                                                         |
| truncation to specific length        | no padding                        | tokenizer(batch_sentences, truncation=True, max_length=42)  or                       |
| nan


Padding and truncation

 | nan                               | tokenizer(batch_sentences, truncation=STRATEGY, max_length=42)                       |
| nan                                  | padding to max sequence in batch  | tokenizer(batch_sentences, padding=True, truncation=True, max_length=42)  or         |
| nan                                  | nan                               | tokenizer(batch_sentences, padding=True, truncation=STRATEGY, max_length=42)         |
| nan                                  | padding to max model input length | Not possible                                                                         |
| nan                                  | padding to specific length        | tokenizer(batch_sentences, padding='max_length', truncation=True, max_length=42)  or |
| nan                                  | nan                               | tokenizer(batch_sentences, padding='max_length', truncation=STRATEGY, max_length=42)


Preprocess

Before you can train a model on a dataset, it needs to be preprocessed into the expected model input format. Whether your data is text, images, or audio, they need to be converted and assembled into batches of tensors. 🤗 Transformers provides a set of preprocessing classes to help prepare your data for the model. In this tutorial, you’ll learn that for:
Text, use a Tokenizer to convert text into a sequence of tokens, create a numerical representation of the tokens, and assemble them into tensors. Speech and audio, use a Feature extractor to


Preprocess

 extract sequential features from audio waveforms and convert them into tensors. Image inputs use a ImageProcessor to convert images into tensors. Multimodal inputs, use a Processor to combine a tokenizer and a feature extractor or image processor.
AutoProcessor always works and automatically chooses the correct class for the model you’re using, whether you’re using a tokenizer, image processor, feature extractor or processor.
Before you begin, install 🤗 Datasets so you can load some datasets to experiment with:
  Copied pip install dataset


Natural Language Processing

The main tool for preprocessing textual data is a tokenizer. A tokenizer splits text into tokens according to a set of rules. The tokens are converted into numbers and then tensors, which become the model inputs. Any additional inputs required by the model are added by the tokenizer.
If you plan on using a pretrained model, it’s important to use the associated pretrained tokenizer. This ensures the text is split the same way as the pretraining corpus, and uses the same corresponding tokens-to-index (usually referred to as the vocab) during pretraining.
Get started by loading a pretrained tokenizer with the AutoTokenizer.from_pretrained() method. This downloads the vocab a model was pretrained with:
  Copied >>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
Then pass your text to the tokenizer:
  Copied >>> encoded_input = tokenizer("Do not meddle in the affairs of wizards, for they are subtle and quic


Natural Language Processing

k to anger.")
>>> print(encoded_input)
{'input_ids': [101, 2079, 2025, 19960, 10362, 1999, 1996, 3821, 1997, 16657, 1010, 2005, 2027, 2024, 11259, 1998, 4248, 2000, 4963, 1012, 102],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
The tokenizer returns a dictionary with three important items:
input_ids are the indices corresponding to each token in the sentence. attention_mask indicates whether a token should be attended to or not. token_type_ids identifies which sequence a token belongs to when there is more than one sequence.
Return your input by decoding the input_ids:
  Copied >>> tokenizer.decode(encoded_input["input_ids"])
'[CLS] Do not meddle in the affairs of wizards, for they are subtle and quick to anger. [SEP]'
As you can see, the tokenizer added two special tokens - CLS and SEP (classifier and separator) - to the sentence. Not a


Natural Language Processing

ll models need
special tokens, but if they do, the tokenizer automatically adds them for you.
If there are several sentences you want to preprocess, pass them as a list to the tokenizer:
  Copied >>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?",
... ]
>>> encoded_inputs = tokenizer(batch_sentences)
>>> print(encoded_inputs)
{'input_ids': [[101, 1252, 1184, 1164, 1248, 6462, 136, 102],
               [101, 1790, 112, 189, 1341, 1119, 3520, 1164, 1248, 6462, 117, 21902, 1643, 119, 102],
               [101, 1327, 1164, 5450, 23434, 136, 102]],
 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0],
                    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                    [0, 0, 0, 0, 0, 0, 0]],
 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1],
                    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
                    [1, 1, 1, 1, 1, 1, 1]


Pad

Sentences aren’t always the same length which can be an issue because tensors, the model inputs, need to have a uniform shape. Padding is a strategy for ensuring tensors are rectangular by adding a special padding token to shorter sentences.
Set the padding parameter to True to pad the shorter sequences in the batch to match the longest sequence:
  Copied >>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?",
... ]
>>> encoded_input = tokenizer(batch_sentences, padding=True)
>>> print(encoded_input)
{'input_ids': [[101, 1252, 1184, 1164, 1248, 6462, 136, 102, 0, 0, 0


Pad

, 0, 0, 0, 0],
               [101, 1790, 112, 189, 1341, 1119, 3520, 1164, 1248, 6462, 117, 21902, 1643, 119, 102],
               [101, 1327, 1164, 5450, 23434, 136, 102, 0, 0, 0, 0, 0, 0, 0, 0]],
 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],
 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0],
                    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
                    [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]]}
The first and third sentences are now padded with 0’s because they are shorter


Truncation

On the other end of the spectrum, sometimes a sequence may be too long for a model to handle. In this case, you’ll need to truncate the sequence to a shorter length.
Set the truncation parameter to True to truncate a sequence to the maximum length accepted by the model:
  Copied >>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?",
... ]
>>> encoded_input = tokenizer(batch_sentences, padding=True, truncation=True)
>>> print(encoded_input)
{'input_ids': [[101, 1252, 1184, 1164, 1248, 6462, 136, 102, 0, 0, 0, 0, 0, 0, 0],
               [101, 1790, 112


Truncation

, 189, 1341, 1119, 3520, 1164, 1248, 6462, 117, 21902, 1643, 119, 102],
               [101, 1327, 1164, 5450, 23434, 136, 102, 0, 0, 0, 0, 0, 0, 0, 0]],
 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],
 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0],
                    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
                    [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]]}
Check out the Padding and truncation concept guide to learn more different padding and truncation arguments


Build tensors

Finally, you want the tokenizer to return the actual tensors that get fed to the model.
Set the return_tensors parameter to either pt for PyTorch, or tf for TensorFlow:
  Pytorch  Hide Pytorch content    Copied >>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?",
... ]
>>> encoded_input = tokenizer(batch_sentences, padding=True, truncation=True, return_tensors="pt")
>>> print(encoded_input)
{'input_ids': tensor([[101, 1252, 1184, 1164, 1248, 6462, 136, 102, 0, 0, 0, 0, 0, 0, 0],
                      [101, 1790, 112, 189, 1341, 1119, 3520, 1164, 1248, 6462, 117, 21902, 1643, 119, 102],
                      [101, 1327, 1164, 5450, 23434, 136, 102, 0, 0, 0, 0, 0, 0, 0, 0]]),
 't


Build tensors

oken_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                           [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                           [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]),
 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0],
                           [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
                           [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]])}  TensorFlow  Hide TensorFlow content    Copied >>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?",
... ]
>>> encoded_input = tokenizer(batch_sentences, padding=True, truncation=True, return_tensors="tf")
>>> print(encoded_input)
{'input_ids


Build tensors

': <tf.Tensor: shape=(2, 9), dtype=int32, numpy=
array([[101, 1252, 1184, 1164, 1248, 6462, 136, 102, 0, 0, 0, 0, 0, 0, 0],
       [101, 1790, 112, 189, 1341, 1119, 3520, 1164, 1248, 6462, 117, 21902, 1643, 119, 102],
       [101, 1327, 1164, 5450, 23434, 136, 102, 0, 0, 0, 0, 0, 0, 0, 0]],
      dtype=int32)>,
 'token_type_ids': <tf.Tensor: shape=(2, 9), dtype=int32, numpy=
array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=int32)>,
 'attention_mask': <tf.Tensor: shape=(2, 9), dtype=int32, numpy=
array([[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0],
       [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
       [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=int32)


Audio

For audio tasks, you’ll need a feature extractor to prepare your dataset for the model. The feature extractor is designed to extract features from raw audio data, and convert them into tensors.
Load the MInDS-14 dataset (see the 🤗 Datasets tutorial for more details on how to load a dataset) to see how you can use a feature extractor with audio datasets:
  Copied >>> from datasets import load_dataset, Audio
>>> dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")
Access the first element of the audio column to take a look at the input. Calling the audio column automatically loads and resamples the audio file:
  Copied >>> dataset[0]["audio"]
{'array': array([ 0.        ,  0.00024414, -0.00024414, ..., -0.00024414,
         0.        ,  0.        ], dtype=float32),
 'path': '/root/.cache/huggingface/datasets/downloads/extracted


Audio

/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~JOINT_ACCOUNT/602ba55abb1e6d0fbce92065.wav',
 'sampling_rate': 8000}
This returns three items:
array is the speech signal loaded - and potentially resampled - as a 1D array. path points to the location of the audio file. sampling_rate refers to how many data points in the speech signal are measured per second.
For this tutorial, you’ll use the Wav2Vec2 model. Take a look at the model card, and you’ll learn Wav2Vec2 is pretrained on 16kHz sampled speech audio. It is important your audio data’s sampling rate matches the sampling rate of the dataset used to pretrain the model. If your data’s sampling rate isn’t the same, then you need to resample your data.
Use 🤗 Datasets’ cast_column method to upsample the sampling rate to 16kHz:
  Copied >>> dataset = dataset.cast_column("


Audio

audio", Audio(sampling_rate=16_000))
Call the audio column again to resample the audio file:
  Copied >>> dataset[0]["audio"]
{'array': array([ 2.3443763e-05,  2.1729663e-04,  2.2145823e-04, ...,
         3.8356509e-05, -7.3497440e-06, -2.1754686e-05], dtype=float32),
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~JOINT_ACCOUNT/602ba55abb1e6d0fbce92065.wav',
 'sampling_rate': 16000}
Next, load a feature extractor to normalize and pad the input. When padding textual data, a 0 is added for shorter sequences. The same idea applies to audio data. The feature extractor adds a 0 - interpreted as silence - to array.
Load the feature extractor with AutoFeatureExtractor.from_pretrained():
  Copied >>> from transformers import AutoFeatureExtractor
>>> feature_extrac


Audio

tor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base")
Pass the audio array to the feature extractor. We also recommend adding the sampling_rate argument in the feature extractor in order to better debug any silent errors that may occur.
  Copied >>> audio_input = [dataset[0]["audio"]["array"]]
>>> feature_extractor(audio_input, sampling_rate=16000)
{'input_values': [array([ 3.8106556e-04,  2.7506407e-03,  2.8015103e-03, ...,
        5.6335266e-04,  4.6588284e-06, -1.7142107e-04], dtype=float32)]}
Just like the tokenizer, you can apply padding or truncation to handle variable sequences in a batch. Take a look at the sequence length of these two audio samples:
  Copied >>> dataset[0]["audio"]["array"].shape
(173398,)
>>> dataset[1]["audio"]["array"].shape
(106496,)
Create a function to preprocess the dataset so the audio samples


Audio

are the same lengths. Specify a maximum sample length, and the feature extractor will either pad or truncate the sequences to match it:
  Copied >>> def preprocess_function(examples):
...     audio_arrays = [x["array"] for x in examples["audio"]]
...     inputs = feature_extractor(
...         audio_arrays,
...         sampling_rate=16000,
...         padding=True,
...         max_length=100000,
...         truncation=True,
...     )
...     return inputs
Apply the preprocess_function to the the first few examples in the dataset:
  Copied >>> processed_dataset = preprocess_function(dataset[:5])
The sample lengths are now the same and match the specified maximum length. You can pass your processed dataset to the model now!
  Copied >>> processed_dataset["input_values"][0].shape
(100000,)
>>> processed_dataset["input_values"][1].shape
(100000,)


Computer vision

For computer vision tasks, you’ll need an image processor to prepare your dataset for the model.
Image preprocessing consists of several steps that convert images into the input expected by the model. These steps
include but are not limited to resizing, normalizing, color channel correction, and converting images to tensors.
Image preprocessing often follows some form of image augmentation. Both image preprocessing and image augmentation
transform image data, but they serve different purposes: Image augmentation alters images in a way that can help prevent overfitting and increase the robustness of the model. You can get creative in how you augment your data - adjust brightness and colors, crop, rotate, resize, zoom, etc. However, be mindful not to change the meaning of the images with your augmentations. Image preprocessing guarantees that the images match the model’s expected input format. When fine-tuning a co


Computer vision

mputer vision model, images must be preprocessed exactly as when the model was initially trained. You can use any library you like for image augmentation. For image preprocessing, use the ImageProcessor associated with the model.
Load the food101 dataset (see the 🤗 Datasets tutorial for more details on how to load a dataset) to see how you can use an image processor with computer vision datasets:
Use 🤗 Datasets split parameter to only load a small sample from the training split since the dataset is quite large!
  Copied >>> from datasets import load_dataset
>>> dataset = load_dataset("food101", split="train[:100]")
Next, take a look at the image with 🤗 Datasets Image feature:
  Copied >>> dataset[0]["image"]
Load the image processor with AutoImageProcessor.from_pretrained():
  Copied >>> from transformers import AutoImageProcessor
>>> image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-2


Computer vision

24")
First, let’s add some image augmentation. You can use any library you prefer, but in this tutorial, we’ll use torchvision’s transforms module. If you’re interested in using another data augmentation library, learn how in the Albumentations or Kornia notebooks.
Here we use Compose to chain together a couple of
transforms - RandomResizedCrop and ColorJitter.
Note that for resizing, we can get the image size requirements from the image_processor. For some models, an exact height and
width are expected, for others only the shortest_edge is defined.
  Copied >>> from torchvision.transforms import RandomResizedCrop, ColorJitter, Compose
>>> size = (
...     image_processor.size["shortest_edge"]
...     if "shortest_edge" in image_processor.size
...     else (image_processor.size["height"], image_processor.size["width"])
... )
>>> _transforms = Compose([RandomResizedCrop(size), ColorJitter(brightness=0.5, hue=0.5)]


Computer vision

)
The model accepts pixel_values
as its input. ImageProcessor can take care of normalizing the images, and generating appropriate tensors.
Create a function that combines image augmentation and image preprocessing for a batch of images and generates pixel_values:
  Copied >>> def transforms(examples):
...     images = [_transforms(img.convert("RGB")) for img in examples["image"]]
...     examples["pixel_values"] = image_processor(images, do_resize=False, return_tensors="pt")["pixel_values"]
...     return examples
In the example above we set do_resize=False because we have already resized the images in the image augmentation transformation,
and leveraged the size attribute from the appropriate image_processor. If you do not resize images during image augmentation,
leave this parameter out. By default, ImageProcessor will handle the resizing. If you wish to normalize images as a part of the augmentation transforma


Computer vision

tion, use the image_processor.image_mean,
and image_processor.image_std values.
Then use 🤗 Datasets set_transform to apply the transforms on the fly:
  Copied >>> dataset.set_transform(transforms)
Now when you access the image, you’ll notice the image processor has added pixel_values. You can pass your processed dataset to the model now!
  Copied >>> dataset[0].keys()
Here is what the image looks like after the transforms are applied. The image has been randomly cropped and it’s color properties are different.
  Copied >>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> img = dataset[0]["pixel_values"]
>>> plt.imshow(img.permute(1, 2, 0))
For tasks like object detection, semantic segmentation, instance segmentation, and panoptic segmentation, ImageProcessor
offers post processing methods. These methods convert model’s raw outputs into meaningful predictions such as bounding boxes,
or segmentation maps.


Pad

In some cases, for instance, when fine-tuning DETR, the model applies scale augmentation at training
time. This may cause images to be different sizes in a batch. You can use DetrImageProcessor.pad()
from DetrImageProcessor and define a custom collate_fn to batch images together.
  Copied >>> def collate_fn(batch):
...     pixel_values = [item["pixel_values"] for item in batch]
...     encoding = image_processor.pad(pixel_values, return_tensors="pt")
...     labels = [item["labels"] for item in batch]
...     batch = {}
...     batch["pixel_values"] = encoding["pixel_values"]
...     batch["pixel_mask"] = encoding["pixel_mask"]
...     batch["labels"] = labels
...     return batch


Multimodal

For tasks involving multimodal inputs, you’ll need a processor to prepare your dataset for the model. A processor couples together two processing objects such as as tokenizer and feature extractor.
Load the LJ Speech dataset (see the 🤗 Datasets tutorial for more details on how to load a dataset) to see how you can use a processor for automatic speech recognition (ASR):
  Copied >>> from datasets import load_dataset
>>> lj_speech = load_dataset("lj_speech", split="train")
For ASR, you’re mainly focused on audio and text so you can remove the other columns:
  Copied >>> lj_speech = lj_speech.map(remove_columns=["file", "id", "normalized_text"])
Now take a look at the audio and text columns:
  Copied >>> lj_speech[0]["audio"]
{'array': array([-7.


Multimodal

3242188e-04, -7.6293945e-04, -6.4086914e-04, ...,
         7.3242188e-04,  2.1362305e-04,  6.1035156e-05], dtype=float32),
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/917ece08c95cf0c4115e45294e3cd0dee724a1165b7fc11798369308a465bd26/LJSpeech-1.1/wavs/LJ001-0001.wav',
 'sampling_rate': 22050}
>>> lj_speech[0]["text"]
'Printing, in the only sense with which we are at present concerned, differs from most if not from all the arts and crafts represented in the Exhibition'
Remember you should always resample your audio dataset’s sampling rate to match the sampling rate of the dataset used to pretrain a model!
  Copied >>> lj_speech = lj_speech.cast_column("audio", Audio(sampling_rate=16_000))
Load a processor with AutoProcessor.fr


Multimodal

om_pretrained():
  Copied >>> from transformers import AutoProcessor
>>> processor = AutoProcessor.from_pretrained("facebook/wav2vec2-base-960h")
Create a function to process the audio data contained in array to input_values, and tokenize text to labels. These are the inputs to the model:
  Copied >>> def prepare_dataset(example):
...     audio = example["audio"]
...     example.update(processor(audio=audio["array"], text=example["text"], sampling_rate=16000))
...     return example
Apply the prepare_dataset function to a sample:
  Copied >>> prepare_dataset(lj_speech[0])
The processor has now added input_values and labels, and the sampling rate has also been correctly downsampled to 16kHz. You can pass your processed dataset to the model now!


Community resources:

| Resource                                      | Description                                                                                | Author           |
|-----------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------


Community resources:

------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|
| Hugging Face Transformers Glossary Flashcards | A set of flashcards based on the  Transformers Docs Glossary  that has been put into a form which can be easily learned/revised using  Anki  an open source, cross platform app specifically designed for long term knowledge retention. See this  Introductory video on how to use the flashcards  . | Darigov Research |


Community notebooks:

| Notebook                                                                 | Description                                                                                | Author                |   Unnamed: 3 |
|--------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------|--------------|
| Fine-tune a pre-trained Transformer to generate lyrics                   | How to generate lyrics in the style of your favorite artist by fine-tuning a GPT-2 model                                                                                | Aleksey Korshuk       |          nan |
| Train T5 in Tensorflow 2                                                 | How to train T5


Community notebooks:

 for any task using Tensorflow 2. This notebook demonstrates a Question & Answer task implemented in Tensorflow 2 using SQUAD                                      | Muhammad Harris       |          nan |
| Train T5 on TPU                                                          | How to train T5 on SQUAD with Transformers and Nlp                                                                                | Suraj Patil           |          nan |
| Fine-tune T5 for Classification and Multiple Choice                      | How to fine-tune T5 for classification and multiple choice tasks using a text-to-text format with PyTorch Lightning                                                               | Suraj Patil           |          nan |
| Fine-tune DialoGPT on New Datasets and Languages                         | How to fine-tune the DialoGPT model on a new dataset for open-dialog conversational chatbots


Community notebooks:

                                                                       | Nathan Cooper         |          nan |
| Long Sequence Modeling with Reformer                                     | How to train on sequences as long as 500,000 tokens with Reformer                                                                                | Patrick von Platen    |          nan |
| Fine-tune BART for Summarization                                         | How to fine-tune BART for summarization with fastai using blurr                                                                                | Wayde Gilliam         |          nan |
| Fine-tune a pre-trained Transformer on anyone’s tweets                   | How to generate tweets in the style of your favorite Twitter account by fine-tuning a GPT-2 model                                                                                | Boris Dayma


Community notebooks:

   |          nan |
| Optimize 🤗 Hugging Face models with Weights & Biases                     | A complete tutorial showcasing W&B integration with Hugging Face                                                                                | Boris Dayma           |          nan |
| Pretrain Longformer                                                      | How to build a “long” version of existing pretrained models                                                                                | Iz Beltagy            |          nan |
| Fine-tune Longformer for QA                                              | How to fine-tune longformer model for QA task                                                                                | Suraj Patil           |          nan |
| Evaluate Model with 🤗nlp


Community notebooks:

   | How to evaluate longformer on TriviaQA with  nlp                                                                                | Patrick von Platen    |          nan |
| Fine-tune T5 for Sentiment Span Extraction                               | How to fine-tune T5 for sentiment span extraction using a text-to-text format with PyTorch Lightning                                                                              | Lorenzo Ampil         |          nan |
| Fine-tune DistilBert for Multiclass Classification                       | How to fine-tune DistilBert for multiclass classification with PyTorch                                                                                | Abhishek Kumar Mishra |          nan |
| Fine-tune BERT for Multi-label Classification                            | How to fine-tune BERT for multi-label classification using PyTorch


Community notebooks:

                                                                                | Abhishek Kumar Mishra |          nan |
| Fine-tune T5 for Summarization                                           | How to fine-tune T5 for summarization in PyTorch and track experiments with WandB                                                                                | Abhishek Kumar Mishra |          nan |
| Speed up Fine-Tuning in Transformers with Dynamic Padding / Bucketing    | How to speed up fine-tuning by a factor of 2 using dynamic padding / bucketing                                                                                | Michael Benesty       |          nan |
| Pretrain Reformer for Masked Language Modeling                           | How to train a Reformer model with bi-directional self-attention layers                                                                                |


Community notebooks:

 Patrick von Platen    |          nan |
| Expand and Fine Tune Sci-BERT                                            | How to increase vocabulary of a pretrained SciBERT model from AllenAI on the CORD dataset and pipeline it.                                                                        | Tanmay Thakur         |          nan |
| Fine Tune BlenderBotSmall for Summarization using the Trainer API        | How to fine-tune BlenderBotSmall for summarization on a custom dataset, using the Trainer API.                                                                                | Tanmay Thakur         |          nan |
| Fine-tune Electra and interpret with Integrated Gradients                | How to fine-tune Electra for sentiment analysis and interpret predictions with Captum Integrated Gradients                                                                        | Eliza Szczechla       |          nan |
| fine-tune a non-English GPT-2 Model with Trainer c


Community notebooks:

lass                   | How to fine-tune a non-English GPT-2 Model with Trainer class                                                                                | Philipp Schmid        |          nan |
| Fine-tune a DistilBERT Model for Multi Label Classification task         | How to fine-tune a DistilBERT Model for Multi Label Classification task                                                                                | Dhaval Taunk          |          nan |
| Fine-tune ALBERT for sentence-pair classification                        | How to fine-tune an ALBERT model or another BERT-based model for the sentence-pair classification task                                                                            | Nadir El Manouzi      |          nan |
| Fine-tune Roberta for sentiment analysis                                 | How to fine-tune a Roberta model for sentiment analysis


Community notebooks:

                                                                                | Dhaval Taunk          |          nan |
| Evaluating Question Generation Models                                    | How accurate are the answers to questions generated by your seq2seq transformer model?                                                                                | Pascal Zoleko         |          nan |
| Classify text with DistilBERT and Tensorflow                             | How to fine-tune DistilBERT for text classification in TensorFlow                                                                                | Peter Bayerle         |          nan |
| Leverage BERT for Encoder-Decoder Summarization on CNN/Dailymail         | How to warm-start a  EncoderDecoderModel  with a  bert-base-uncased  checkpoint for summarization on CNN/Dailymail


Community notebooks:

                   | Patrick von Platen    |          nan |
| Leverage RoBERTa for Encoder-Decoder Summarization on BBC XSum           | How to warm-start a shared  EncoderDecoderModel  with a  roberta-base  checkpoint for summarization on BBC/XSum                                                                   | Patrick von Platen    |          nan |
| Fine-tune TAPAS on Sequential Question Answering (SQA)                   | How to fine-tune  TapasForQuestionAnswering  with a  tapas-base  checkpoint on the Sequential Question Answering (SQA) dataset                                                    | Niels Rogge           |          nan |
| Evaluate TAPAS on Table Fact Checking (TabFact)                          | How to evaluate a fine-tuned  TapasForSequenceClassification  with a  tapas-base-finetuned-tabfact  checkpoint using a combination of the 🤗 datasets and 🤗 transformers libraries | Niels Rogge           |          nan |
| Fine-tuning mBART for translat


Community notebooks:

ion                                        | How to fine-tune mBART using Seq2SeqTrainer for Hindi to English translation                                                                                | Vasudev Gupta         |          nan |
| Fine-tune LayoutLM on FUNSD (a form understanding dataset)               | How to fine-tune  LayoutLMForTokenClassification  on the FUNSD dataset for information extraction from scanned documents                                                          | Niels Rogge           |          nan |
| Fine-Tune DistilGPT2 and Generate Text                                   | How to fine-tune DistilGPT2 and generate text                                                                                | Aakash Tripathi       |          nan |
| Fine-Tune LED on up to 8K tokens                                         | How to fine-tune LED on pubmed for long-range s


Community notebooks:

ummarization                                                                                | Patrick von Platen    |          nan |
| Evaluate LED on Arxiv                                                    | How to effectively evaluate LED on long-range summarization                                                                                | Patrick von Platen    |          nan |
| Fine-tune LayoutLM on RVL-CDIP (a document image classification dataset) | How to fine-tune  LayoutLMForSequenceClassification  on the RVL-CDIP dataset for scanned document classification                                                                  | Niels Rogge           |          nan |
| Wav2Vec2 CTC decoding with GPT2 adjustment                               | How to decode CTC sequence with language model adjustment


Community notebooks:

                                       | Eric Lam              |          nan |
| Fine-tune BART for summarization in two languages with Trainer class     | How to fine-tune BART for summarization in two languages with Trainer class                                                                                | Eliza Szczechla       |          nan |
| Evaluate Big Bird on Trivia QA                                           | How to evaluate BigBird on long document question answering on Trivia QA                                                                                | Patrick von Platen    |          nan |
| Create video captions using Wav2Vec2                                     | How to create YouTube captions from any video by transcribing the audio with Wav2Vec                                                                                | Niklas Muennighoff    |          nan |
| Fine-tune


Community notebooks:

the Vision Transformer on CIFAR-10 using PyTorch Lightning     | How to fine-tune the Vision Transformer (ViT) on CIFAR-10 using HuggingFace Transformers, Datasets and PyTorch Lightning                                                          | Niels Rogge           |          nan |
| Fine-tune the Vision Transformer on CIFAR-10 using the 🤗 Trainer         | How to fine-tune the Vision Transformer (ViT) on CIFAR-10 using HuggingFace Transformers, Datasets and the 🤗 Trainer                                                              | Niels Rogge           |          nan |
| Evaluate LUKE on Open Entity, an entity typing dataset                   | How to evaluate  LukeForEntityClassification  on the Open Entity dataset                                                                                | Ikuya Yamada          |          nan |
| Evaluate LUKE on TACRED, a relation extraction dataset                   | How to evaluate  LukeForEnt


Community notebooks:

ityPairClassification  on the TACRED dataset                                                                                | Ikuya Yamada          |          nan |
| Evaluate LUKE on CoNLL-2003, an important NER benchmark                  | How to evaluate  LukeForEntitySpanClassification  on the CoNLL-2003 dataset                                                                                | Ikuya Yamada          |          nan |
| Evaluate BigBird-Pegasus on PubMed dataset                               | How to evaluate  BigBirdPegasusForConditionalGeneration  on PubMed dataset                                                                                | Vasudev Gupta         |          nan |
| Speech Emotion Classification with Wav2Vec2                              | How to leverage a pretrained Wav2Vec2 model for Emotion Classification on the MEGA dataset


Community notebooks:

                                                           | Mehrdad Farahani      |          nan |
| Detect objects in an image with DETR                                     | How to use a trained  DetrForObjectDetection  model to detect objects in an image and visualize attention                                                                         | Niels Rogge           |          nan |
| Fine-tune DETR on a custom object detection dataset                      | How to fine-tune  DetrForObjectDetection  on a custom object detection dataset                                                                                | Niels Rogge           |          nan |
| Finetune T5 for Named Entity Recognition                                 | How to fine-tune  T5  on a Named Entity Recognition Task                                                                                | Ogundepo Odunayo      |


Pipelines for inference

The pipeline() makes it simple to use any model from the Hub for inference on any language, computer vision, speech, and multimodal tasks. Even if you don’t have experience with a specific modality or aren’t familiar with the underlying code behind the models, you can still use them for inference with the pipeline()! This tutorial will teach you to:
Use a pipeline() for inference. Use a specific tokenizer or model. Use a pipeline() for audio, vision, and multimodal tasks.
Take a look at the pipeline() documentation for a complete list of supported tasks and available parameters.


Pipeline usage

While each task has an associated pipeline(), it is simpler to use the general pipeline() abstraction which contains
all the task-specific pipelines. The pipeline() automatically loads a default model and a preprocessing class capable
of inference for your task. Let’s take the example of using the pipeline() for automatic speech recognition (ASR), or
speech-to-text.
Start by creating a pipeline() and specify the inference task:
  Copied >>> from transformers import pipeline
>>> transcriber = pipeline(task="automatic-speech-recognition")
Pass your input to the pipeline(). In the case of speech recognition, this is an audio input file:
  Copied >>> transcriber("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac")
{'text': 'I HAVE A DREAM BUT ONE DAY THIS NATION WILL RISE UP LIVE UP THE TRUE MEANING OF ITS TREES'}
Not the result you had in mind? Check out some of the most downloaded au


Pipeline usage

tomatic speech recognition models
on the Hub to see if you can get a better transcription.
Let’s try the Whisper large-v2 model from OpenAI. Whisper was released
2 years later than Wav2Vec2, and was trained on close to 10x more data. As such, it beats Wav2Vec2 on most downstream
benchmarks. It also has the added benefit of predicting punctuation and casing, neither of which are possible with
Wav2Vec2.
Let’s give it a try here to see how it performs:
  Copied >>> transcriber = pipeline(model="openai/whisper-large-v2")
>>> transcriber("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac")
{'text': ' I have a dream that one day this nation will rise up and live out the true meaning of its creed.'}
Now this result looks more accurate! For a deep-dive comparison on Wav2Vec2 vs Whisper, refer to the Audio Transformers Course.
We really encourage you to check out the Hub for models in diffe


Pipeline usage

rent languages, models specialized in your field, and more.
You can check out and compare model results directly from your browser on the Hub to see if it fits or
handles corner cases better than other ones.
And if you don’t find a model for your use case, you can always start training your own!
If you have several inputs, you can pass your input as a list:
  Copied transcriber(
    [
        "https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac",
        "https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac",
    ]
)
Pipelines are great for experimentation as switching from one model to another is trivial; however, there are some ways to optimize them for larger workloads than experimentation. See the following guides that dive into iterating over whole datasets or using pipelines in a webserver:
of the docs:
Using pipelines on a dataset Using pipelines for a webserve


Parameters

pipeline() supports many parameters; some are task specific, and some are general to all pipelines.
In general, you can specify parameters anywhere you want:
  Copied transcriber = pipeline(model="openai/whisper-large-v2", my_parameter=1)
out = transcriber(...)  # This will use `my_parameter=1`.
out = transcriber(..., my_parameter=2)  # This will override and use `my_parameter=2`.
out = transcriber(...)  # This will go back to using `my_parameter=1`.
Let’s check out 3 important ones:


Device

If you use device=n, the pipeline automatically puts the model on the specified device.
This will work regardless of whether you are using PyTorch or Tensorflow.
  Copied transcriber = pipeline(model="openai/whisper-large-v2", device=0)
If the model is too large for a single GPU and you are using PyTorch, you can set device_map="auto" to automatically
determine how to load and store the model weights. Using the device_map argument requires the 🤗 Accelerate
package:
  Copied pip install --upgrade accelerate
The following code automatically loads and stores model weights across devices:
  Copied transcriber = pipeline(model="openai/whisper-large-v2", device_map="auto")
Note that if  device_map="auto" is passed, there is no need to add the argument device=device when instantiating your pipeline as you may encounter some unexpected behavior!


Batch size

By default, pipelines will not batch inference for reasons explained in detail here. The reason is that batching is not necessarily faster, and can actually be quite slower in some cases.
But if it works in your use case, you can use:
  Copied transcriber = pipeline(model="openai/whisper-large-v2", device=0, batch_size=2)
audio_filenames = [f"https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/{i}.flac" for i in range(1, 5)]
texts = transcriber(audio_filenames)
This runs the pipeline on the 4 provided audio files, but it will p


Batch size

ass them in batches of 2
to the model (which is on a GPU, where batching is more likely to help) without requiring any further code from you.
The output should always match what you would have received without batching. It is only meant as a way to help you get more speed out of a pipeline.
Pipelines can also alleviate some of the complexities of batching because, for some pipelines, a single item (like a long audio file) needs to be chunked into multiple parts to be processed by a model. The pipeline performs this chunk batching for you


Task specific parameters

All tasks provide task specific parameters which allow for additional flexibility and options to help you get your job done.
For instance, the transformers.AutomaticSpeechRecognitionPipeline.call() method has a return_timestamps parameter which sounds promising for subtitling videos:
  Copied >>> transcriber = pipeline(model="openai/whisper-large-v2", return_timestamps=True)
>>> transcriber("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac")
{'text': ' I have a dream that one day this nation will rise up and live out the true meaning of its creed.', 'chunks': [{'timestamp': (0.0, 11.88), 'text': ' I have a dream that one day this nation will rise up and live out the true meaning of its'}, {'timestamp': (11.88, 12.38), 'text': ' creed.'}]}
As you can see, the model inferred the text and also outputted when the various sentences were pronounced.
There are many parameters available f


Task specific parameters

or each task, so check out each task’s API reference to see what you can tinker with!
For instance, the AutomaticSpeechRecognitionPipeline has a chunk_length_s parameter which is helpful
for working on really long audio files (for example, subtitling entire movies or hour-long videos) that a model typically
cannot handle on its own:
  Copied >>> transcriber = pipeline(model="openai/whisper-large-v2", chunk_length_s=30, return_timestamps=True)
>>> transcriber("https://huggingface.co/datasets/sanchit-gandhi/librispeech_long/resolve/main/audio.wav")
{'text': " Chapter 16. I might have told you of the beginning of this liaison in a few lines, but I wanted you to see every step by which we came.  I, too, agree to whatever Marguerite wished, Marguerite to be unable to live apart from me. It was the day after the evening...
If you can’t find a parameter that would really help you out, feel free to request it!


Using pipelines on a dataset

The pipeline can also run inference on a large dataset. The easiest way we recommend doing this is by using an iterator:
  Copied def data():
    for i in range(1000):
        yield f"My example {i}"
pipe = pipeline(model="gpt2", device=0)
generated_characters = 0
for out in pipe(data()):
    generated_characters += len(out[0]["generated_text"])
The iterator data() yields each result, and the pipeline automatically
recognizes the input is iterable and will start fetching the data while
it continues to process it on the GPU (this uses DataLoader under the hood).
This is important because you don’t have to allocate memory for the whole da


Using pipelines on a dataset

taset
and you can feed the GPU as fast as possible.
Since batching could speed things up, it may be useful to try tuning the batch_size parameter here.
The simplest way to iterate over a dataset is to just load one from 🤗 Datasets:
  Copied # KeyDataset is a util that will just output the item we're interested in.
from transformers.pipelines.pt_utils import KeyDataset
from datasets import load_dataset
pipe = pipeline(model="hf-internal-testing/tiny-random-wav2vec2", device=0)
dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation[:10]")
for out in pipe(KeyDataset(dataset, "audio")):
    print(out)


Vision pipeline

Using a pipeline() for vision tasks is practically identical.
Specify your task and pass your image to the classifier. The image can be a link, a local path or a base64-encoded image. For example, what species of cat is shown below?
  Copied >>> from transformers import pipeline
>>> vision_classifier = pipeline(model="google/vit-base-patch16-224")
>>> preds = vision_classifier(
...     images="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> preds
[{'score': 0.4335, 'label': 'lynx, catamount'}, {'score': 0.0348, 'label': 'cougar, puma, catamount, mountain lion, painter, panther, Felis concolor'}, {'score': 0.0324, 'label': 'snow leopard, ounce, Panthera uncia'}, {'score': 0.0239, 'label': 'Egyptian cat'}, {'score': 0.0229, 'label': 'tiger cat'}]


Text pipeline

Using a pipeline() for NLP tasks is practically identical.
  Copied >>> from transformers import pipeline
>>> # This model is a `zero-shot-classification` model.
>>> # It will classify text, except you are free to choose any label you might imagine
>>> classifier = pipeline(model="facebook/bart-large-mnli")
>>> classifier(
...     "I have a problem with my iphone that needs to be resolved asap!!",
...     candidate_labels=["urgent", "not urgent", "phone", "tablet", "computer"],
... )
{'sequence': 'I have a problem with my iphone that needs to be resolved asap!!', 'labels': ['urgent', 'phone', 'computer', 'not urgent', 'tablet'], 'scores': [0.504, 0.479, 0.013, 0.003, 0.002]}


Multimodal pipeline

The pipeline() supports more than one modality. For example, a visual question answering (VQA) task combines text and image. Feel free to use any image link you like and a question you want to ask about the image. The image can be a URL or a local path to the image.
For example, if you use this invoice image:
  Copied >>> from transformers import pipeline
>>> vqa = pipeline(model="impira/layoutlm-document-qa")
>>> vqa(
...     image="https://huggingface.co/spaces/impira/docquery/resolve/2359223c1837a7587402bda0f2643382a6eefeab/invoice.png",
...     question="What is the invoice number?",
... )
[{'score': 0.42515, 'answer': 'us-001', 'start': 16, 'end': 16}]
To run the example above you need to have pytesseract installed in addition to 🤗 Transformers:   Copied sudo apt install -y tesseract-ocr
pip install pytesseract


Using pipeline on large models with 🤗 accelerate :

You can easily run pipeline on large models using 🤗 accelerate! First make sure you have installed accelerate with pip install accelerate.
First load your model using device_map="auto"! We will use facebook/opt-1.3b for our example.
  Copied # pip install accelerate
import torch
from transformers import pipeline
pipe = pipeline(model="facebook/opt-1.3b", torch_dtype=torch.bfloat16, device_map="auto")
output = pipe("This is a cool example!", do_sample=True, top_p=0.95)
You can also pass 8-bit loaded models if you install bitsandbytes and add the argument load_in_8bit=True
  Copied # pip install accelerate bitsandbytes
import torch
from transformers import pipeline
pipe = pipeline(model="facebook/opt-1.3b", device_map="auto", model_kwargs={"load_in_8bit": True})
output = pipe("This is a cool example!", do_sample=True, top_p=0.95)
Note that you can replace the checkpoint with any of the Hugging Face model that supports large model loading such as BLOOM!


Introduction

An increasingly common use case for LLMs is chat. In a chat context, rather than continuing a single string
of text (as is the case with a standard language model), the model instead continues a conversation that consists
of one or more messages, each of which includes a role as well as message text.
Most commonly, these roles are “user” for messages sent by the user, and “assistant” for messages sent by the model.
Some models also support a “system” role. System messages are usually sent at the beginning of the conversation
and include directives about how the model should behave in the subsequent chat.
All language models, including models fine-tuned for chat, operate on linear sequences of tokens and do not intrinsically
have special handling for roles. This means that role information is usually injected by adding control tokens
between messages, to indicate both the message boundary and the relevant role


Introduction

s.
Unfortunately, there isn’t (yet!) a standard for which tokens to use, and so different models have been trained
with wildly different formatting and control tokens for chat. This can be a real problem for users - if you use the
wrong format, then the model will be confused by your input, and your performance will be a lot worse than it should be.
This is the problem that chat templates aim to resolve.
Chat conversations are typically represented as a list of dictionaries, where each dictionary contains role
and content keys, and represents a single chat message. Chat templates are strings containing a Jinja template that
specifies how to format a conversation for a given model into a single tokenizable sequence. By storing this information
with the tokenizer, we can ensure that models get input data in the format they expect.
Let’s make this concrete with a quick example using the BlenderBot model. Blender


Introduction

Bot has an extremely simple default
template, which mostly just adds whitespace between rounds of dialogue:
  Copied >>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/blenderbot-400M-distill")
>>> chat = [
...   {"role": "user", "content": "Hello, how are you?"},
...   {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
...   {"role": "user", "content": "I'd like to show off how chat templating works!"},
... ]
>>> tokenizer.apply_chat_template(chat, tokenize=False)
" Hello, how are you?  I'm doing great. How can I help you today?   I'd like to show off how chat templating works!</s>"
Notice how the entire chat is condensed into a single string. If we use tokenize=True, which is the default setting,
that string will also be tokenized for us. To see a more complex template in action, though, let’s use the
meta-llama/Llama-2-7b-chat-hf


Introduction

model. Note that this model has gated access, so you will have to
request access on the repo if you want to run this code yourself:
  Copied >> from transformers import AutoTokenizer
>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
>> chat = [
...   {"role": "user", "content": "Hello, how are you?"},
...   {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
...   {"role": "user", "content": "I'd like to show off how chat templating works!"},
... ]
>> tokenizer.use_default_system_prompt = False
>> tokenizer.apply_chat_template(chat, tokenize=False)
"<s>[INST] Hello, how are you? [/INST] I'm doing great. How can I help you today? </s><s>[INST] I'd like to show off how chat templating works! [/INST]"
Note that this time, the tokenizer has added the control tokens [INST] and [/INST] to indicate the start and end of
user messages (but not assistant messages!


How do chat templates work?

The chat template for a model is stored on the tokenizer.chat_template attribute. If no chat template is set, the
default template for that model class is used instead. Let’s take a look at the template for BlenderBot:
  Copied
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/blenderbot-400M-distill")
>>> tokenizer.default_chat_template
"{% for message in messages %}{% if message['role'] == 'user' %}{{ ' ' }}{% endif %}{{ message['content'] }}{% if not loop.last %}{{ '  ' }}{% endif %}{% endfor %}{{ eos_token }}"
That’s kind of intimidating. Let’s add some newlines and indentation to make it more readable. Note that the first
newline after each block as well as any preceding whitespace before a block are ignored by default, using the
Jinja trim_blocks and lstrip_blocks flags. H


How do chat templates work?

owever, be cautious - although leading whitespace on each
line is stripped, spaces between blocks on the same line are not. We strongly recommend checking that your template
isn’t printing extra spaces where it shouldn’t be!
  Copied {% for message in messages %}
    {% if message['role'] == 'user' %}
        {{ ' ' }}
    {% endif %}
    {{ message['content'] }}
    {% if not loop.last %}
        {{ '  ' }}
    {% endif %}
{% endfor %}
{{ eos_token }}
If you’ve never seen one of these before, this is a Jinja template.
Jinja is a templating language that allows you to write simple code that generates text. In many ways, the code and
syntax resembles Python. In pure Python, this template would look something like this:
  Copied for idx, message in enumerate(messages):
    if message['role'] == 'user':
        print(' ')
    prin


How do chat templates work?

t(message['content'])
    if not idx == len(messages) - 1:  # Check for the last message in the conversation
        print('  ')
print(eos_token)
Effectively, the template does three things:
For each message, if the message is a user message, add a blank space before it, otherwise print nothing. Add the message content If the message is not the last message, add two spaces after it. After the final message, print the EOS token.
This is a pretty simple template - it doesn’t add any control tokens, and it doesn’t support “system” messages, which
are a common way to give the model directives about how it should behave in the subsequent conversation.
But Jinja gives you a lot of flexibility to do those things! Let’s see a Jinja template that can format inputs
similarly to the way LLaMA formats them (note that the real LLaMA templat


How do chat templates work?

e includes handling for default system
messages and slightly different system message handling in general - don’t use this one in your actual code!)
  Copied {% for message in messages %}
    {% if message['role'] == 'user' %}
        {{ bos_token + '[INST] ' + message['content'] + ' [/INST]' }}
    {% elif message['role'] == 'system' %}
        {{ '<<SYS>>\\n' + message['content'] + '\\n<</SYS>>\\n\\n' }}
    {% elif message['role'] == 'assistant' %}
        {{ ' '  + message['content'] + ' ' + eos_token }}
    {% endif %}
{% endfor %}
Hopefully if you stare at this for a little bit you can see what this template is doing - it adds specific tokens based
on the “role” of each message, which represents who sent it. User, assistant and system messages are clearly
distinguishable to the model because of the tokens they’re wrapped


How do I create a chat template?

Simple, just write a jinja template and set tokenizer.chat_template. You may find it easier to start with an
existing template from another model and simply edit it for your needs! For example, we could take the LLaMA template
above and add ”[ASST]” and ”[/ASST]” to assistant messages:
  Copied {% for message in messages %}
    {% if message['role'] == 'user' %}
        {{ bos_token + '[INST] ' + message['content'].strip() + ' [/INST]' }}
    {% elif message['role'] == 'system' %}
        {{ '<<SYS>>\\n' + message['content'].strip() + '\\n<</SYS>>\\n\\n' }}
    {% elif message['role'] == 'assistant' %}
        {{ '[ASST] '  + message['content'] + ' [/ASST]' + eos_token }}
    {% endif %}
{% endfor %}
Now, simply set the tokenizer.chat_template att


How do I create a chat template?

ribute. Next time you use apply_chat_template(), it will
use your new template! This attribute will be saved in the tokenizer_config.json file, so you can use
push_to_hub() to upload your new template to the Hub and make sure everyone’s using the right
template for your model!
  Copied template = tokenizer.chat_template
template = template.replace("SYS", "SYSTEM")  # Change the system token
tokenizer.chat_template = template  # Set the new template
tokenizer.push_to_hub("model_name")  # Upload your new template to the Hub!
The method apply_chat_template() which uses your chat template is called by the ConversationalPipeline class, so
once you set the correct chat template, your model will automatically become compatible with ConversationalPipeline


What are “default” templates?

Before the introduction of chat templates, chat handling was hardcoded at the model class level. For backwards
compatibility, we have retained this class-specific handling as default templates, also set at the class level. If a
model does not have a chat template set, but there is a default template for its model class, the ConversationalPipeline
class and methods like apply_chat_template will use the class template instead. You can find out what the default
template for your tokenizer is by checking the tokenizer.default_chat_template attribute.
This is something we do purely for backward compatibility reasons, to avoid breaking any existing workflows. Even when
the class template is appropriate for your model, we strongly recommend overriding the default template by
setting the chat_template attribute explicitly to make it clear to users that your model has been correctly configured
for chat, and to future-proof in case the default templates are ever altered or deprecated.


What template should I use?

When setting the template for a model that’s already been trained for chat, you should ensure that the template
exactly matches the message formatting that the model saw during training, or else you will probably experience
performance degradation. This is true even if you’re training the model further - you will probably get the best
performance if you keep the chat tokens constant. This is very analogous to tokenization - you generally get the
best performance for inference or fine-tuning when you precisely match the tokenization used during training.
If you’re training a model from scratch, or fine-tuning a base language model for chat, on the other hand,
you have a lot of freedom to choose an appropriate template! LLMs are smart enough to learn to handle lots of different
input


What template should I use?

 formats. Our default template for models that don’t have a class-specific template follows the
ChatML format, and this is a good, flexible choice for many use-cases. It looks like this:
  Copied {% for message in messages %}
    {{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}
{% endfor %}
If you like this one, here it is in one-liner form, ready to copy into your code. The one-liner also includes
handy support for “generation prompts” - see the next section for more!
  Copied tokenizer.chat_template = "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_p


What template should I use?

rompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"
This template wraps each message in <|im_start|> and <|im_end|> tokens, and simply writes the role as a string, which
allows for flexibility in the roles you train with. The output looks like this:
  Copied <|im_start|>system
You are a helpful chatbot that will do its best not to say anything so stupid that people tweet about it.<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
I'm doing great!<|im_end|>
The “user”, “system” and “assistant” roles are the standard for chat, and we recommend using them when it makes sense,
particularly if you want your model to operate well with ConversationalPipeline. However, you are not limited
to these roles - templating is extremely flexible, and any string can be a role.


What are “generation prompts”?

You may notice that the apply_chat_template method has an add_generation_prompt argument. This argument tells
the template to add tokens that indicate the start of a bot response. For example, consider the following chat:
  Copied messages = [
    {"role": "user", "content": "Hi there!"},
    {"role": "assistant", "content": "Nice to meet you!"},
    {"role": "user", "content": "Can I ask a question?"}
]
Here’s what this will look like without a generation prompt, using the ChatML template we described above:
  Copied >> tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
"""<|im_start|>user
Hi there!<|im_end|>
<|im_start|>assistant
Nice to meet you!<|im_end|>
<|im_start|>user
Can I ask a question?<|im_end|>
"""
And here’s what it looks like with a generation prompt:
  Copied >> tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
"""<|im_start|>use


What are “generation prompts”?

r
Hi there!<|im_end|>
<|im_start|>assistant
Nice to meet you!<|im_end|>
<|im_start|>user
Can I ask a question?<|im_end|>
<|im_start|>assistant
"""
Note that this time, we’ve added the tokens that indicate the start of a bot response. This ensures that when the model
generates text it will write a bot response instead of doing something unexpected, like continuing the user’s
message. Remember, chat models are still just language models - they’re trained to continue text, and chat is just a
special kind of text to them! You need to guide them with the appropriate control tokens so they know what they’re
supposed to be doing.
Not all models require generation prompts. Some models, like BlenderBot and LLaMA, don’t have any
special tokens before bot responses. In these cases, the add_generation_prompt argument will have no effect. The exact
effect that add_generation_prompt has will depend on the template being used.


I want to use chat templates! How should I get started?

If you have any chat models, you should set their tokenizer.chat_template attribute and test it using
apply_chat_template(). This applies even if you’re not the model owner - if you’re using a model
with an empty chat template, or one that’s still using the default class template, please open a pull request to
the model repository so that this attribute can be set properly!
Once the attribute is set, that’s it, you’re done! tokenizer.apply_chat_template will now work correctly for that
model, which means it is also automatically supported in places like ConversationalPipeline!
By ensuring that models have this attribute, we can make sure that the whole community gets to use the full power of
open-source models. Formatting mismatches have been haunting the field and silently harming performance for too long -
it’s time to put an end to them!


Template writing tips

If you’re unfamiliar with Jinja, we generally find that the easiest way to write a chat template is to first
write a short Python script that formats messages the way you want, and then convert that script into a template.
Remember that the template handler will receive the conversation history as a variable called messages. Each
message is a dictionary with two keys, role and content. You will be able to access messages in your template
just like you can in Python, which means you can loop over it with {% for message in messages %} or access
individual messages with, for example, {{ messages[0] }}.
You can also use the following tips to convert your code to Jinja:


For loops

For loops in Jinja look like this:
  Copied {% for message in messages %}
{{ message['content'] }}
{% endfor %}
Note that whatever’s inside the {{ expression block }} will be printed to the output. You can use operators like
+ to combine strings inside expression blocks.


If statements

If statements in Jinja look like this:
  Copied {% if message['role'] == 'user' %}
{{ message['content'] }}
{% endif %}
Note how where Python uses whitespace to mark the beginnings and ends of for and if blocks, Jinja requires you
to explicitly end them with {% endfor %} and {% endif %}.


Special variables

Inside your template, you will have access to the list of messages, but you can also access several other special
variables. These include special tokens like bos_token and eos_token, as well as the add_generation_prompt
variable that we discussed above. You can also use the loop variable to access information about the current loop
iteration, for example  using {% if loop.last %} to check if the current message is the last message in the
conversation. Here’s an example that puts these ideas together to add a generation prompt at the end of the
conversation if add_generation_prompt is True:
  Copied {% if loop.last and add_generation_prompt %}
{{ bos_token + 'Assistant:\n' }}
{% endif %}


Notes on whitespace

As much as possible, we’ve tried to get Jinja to ignore whitespace outside of {{ expressions }}. However, be aware
that Jinja is a general-purpose templating engine, and it may treat whitespace between blocks on the same line
as significant and print it to the output. We strongly recommend checking that your template isn’t printing extra
spaces where it shouldn’t be before you upload it!


🤗 Transformers

State-of-the-art Machine Learning for PyTorch, TensorFlow, and JAX.
🤗 Transformers provides APIs and tools to easily download and train state-of-the-art pretrained models. Using pretrained models can reduce your compute costs, carbon footprint, and save you the time and resources required to train a model from scratch. These models support common tasks in different modalities, such as:
📝 Natural Language Processing: text classification, named entity recognition, question answering, language modeling, summarization, translation, multiple choice, and text generation.
🖼 Computer Vision: image classification, object detection, and segmentation.
🗣 Audio: autom


🤗 Transformers

atic speech recognition and audio classification.
🐙 Multimodal: table question answering, optical character recognition, information extraction from scanned documents, video classification, and visual question answering.
🤗 Transformers support framework interoperability between PyTorch, TensorFlow, and JAX. This provides the flexibility to use a different framework at each stage of a model’s life; train a model in three lines of code in one framework, and load it for inference in another. Models can also be exported to a format like ONNX and TorchScript for deployment in production environments.
Join the growing community on the Hub, forum, or Discord today


Contents

The documentation is organized into five sections:
GET STARTED provides a quick tour of the library and installation instructions to get up and running. TUTORIALS are a great place to start if you’re a beginner. This section will help you gain the basic skills you need to start using the library. HOW-TO GUIDES show you how to achieve a specific goal, like finetuning a pretrained model for language modeling or how to write and share a custom model. CONCEPTUAL GUIDES offers more discussion and explanation of the underlying concepts and ideas behind models, tasks, and the design philosophy of 🤗 Transformers. API describes all classes and functions: MAIN CLASSES details the most important classes like configuration, model, tokenizer, and pipeline. MODELS details the classes and functions related to each model implemented in the library. INTERNAL HELPERS details utility classes and functions used internally.


Supported models and frameworks

The table below represents the current support in the library for each of those models, whether they have a Python
tokenizer (called “slow”). A “fast” tokenizer backed by the 🤗 Tokenizers library, whether they have support in Jax (via
Flax), PyTorch, and/or TensorFlow.
| Model                         | PyTorch support   | TensorFlow support   | Flax Support   |
|-------------------------------|-------------------|----------------------|----------------|
| ALBERT                        | ✅                 | ✅                    | ✅              |
| ALIGN                         | ✅                 | ❌                    | ❌              |
| AltCLIP                       | ✅                 | ❌                    | ❌              |
| Audio Spectrogram Transformer | ✅                 | ❌                    | ❌              |
| Autoformer                    | ✅                 | ❌                    | ❌              |
| Bark                          | ✅                 | ❌


Supported models and frameworks

                   | ❌              |
| BART                          | ✅                 | ✅                    | ✅              |
| BARThez                       | ✅                 | ✅                    | ✅              |
| BARTpho                       | ✅                 | ✅                    | ✅              |
| BEiT                          | ✅                 | ❌                    | ✅              |
| BERT                          | ✅                 | ✅                    | ✅              |
| Bert Generation               | ✅                 | ❌                    | ❌              |
| BertJapanese                  | ✅                 | ✅                    | ✅              |
| BERTweet                      | ✅                 | ✅                    | ✅              |
| BigBird                       | ✅                 | ❌                    | ✅              |
| BigBird-Pegasus               | ✅                 | ❌                    | ❌              |
| BioG


Supported models and frameworks

pt                        | ✅                 | ❌                    | ❌              |
| BiT                           | ✅                 | ❌                    | ❌              |
| Blenderbot                    | ✅                 | ✅                    | ✅              |
| BlenderbotSmall               | ✅                 | ✅                    | ✅              |
| BLIP                          | ✅                 | ✅                    | ❌              |
| BLIP-2                        | ✅                 | ❌                    | ❌              |
| BLOOM                         | ✅                 | ❌                    | ✅              |
| BORT                          | ✅                 | ✅                    | ✅              |
| BridgeTower                   | ✅                 | ❌                    | ❌              |
| BROS                          | ✅                 | ❌                    | ❌              |
| ByT5                          | ✅


Supported models and frameworks

  | ✅                    | ✅              |
| CamemBERT                     | ✅                 | ✅                    | ❌              |
| CANINE                        | ✅                 | ❌                    | ❌              |
| Chinese-CLIP                  | ✅                 | ❌                    | ❌              |
| CLAP                          | ✅                 | ❌                    | ❌              |
| CLIP                          | ✅                 | ✅                    | ✅              |
| CLIPSeg                       | ✅                 | ❌                    | ❌              |
| CodeGen                       | ✅                 | ❌                    | ❌              |
| CodeLlama                     | ✅                 | ❌                    | ❌              |
| Conditional DETR              | ✅                 | ❌                    | ❌              |
| ConvBERT                      | ✅                 | ✅                    | ❌              |


Supported models and frameworks

| ConvNeXT                      | ✅                 | ✅                    | ❌              |
| ConvNeXTV2                    | ✅                 | ❌                    | ❌              |
| CPM                           | ✅                 | ✅                    | ✅              |
| CPM-Ant                       | ✅                 | ❌                    | ❌              |
| CTRL                          | ✅                 | ✅                    | ❌              |
| CvT                           | ✅                 | ✅                    | ❌              |
| Data2VecAudio                 | ✅                 | ❌                    | ❌              |
| Data2VecText                  | ✅                 | ❌                    | ❌              |
| Data2VecVision                | ✅                 | ✅                    | ❌              |
| DeBERTa                       | ✅                 | ✅                    | ❌              |
| DeBERTa-v2                    | ✅


Supported models and frameworks

        | ✅                    | ❌              |
| Decision Transformer          | ✅                 | ❌                    | ❌              |
| Deformable DETR               | ✅                 | ❌                    | ❌              |
| DeiT                          | ✅                 | ✅                    | ❌              |
| DePlot                        | ✅                 | ❌                    | ❌              |
| DETA                          | ✅                 | ❌                    | ❌              |
| DETR                          | ✅                 | ❌                    | ❌              |
| DialoGPT                      | ✅                 | ✅                    | ✅              |
| DiNAT                         | ✅                 | ❌                    | ❌              |
| DINOv2                        | ✅                 | ❌                    | ❌              |
| DistilBERT                    | ✅                 | ✅                    | ✅


Supported models and frameworks

    |
| DiT                           | ✅                 | ❌                    | ✅              |
| DonutSwin                     | ✅                 | ❌                    | ❌              |
| DPR                           | ✅                 | ✅                    | ❌              |
| DPT                           | ✅                 | ❌                    | ❌              |
| EfficientFormer               | ✅                 | ✅                    | ❌              |
| EfficientNet                  | ✅                 | ❌                    | ❌              |
| ELECTRA                       | ✅                 | ✅                    | ✅              |
| EnCodec                       | ✅                 | ❌                    | ❌              |
| Encoder decoder               | ✅                 | ✅                    | ✅              |
| ERNIE                         | ✅                 | ❌                    | ❌              |
| ErnieM                        | ✅


Supported models and frameworks

              | ❌                    | ❌              |
| ESM                           | ✅                 | ✅                    | ❌              |
| FairSeq Machine-Translation   | ✅                 | ❌                    | ❌              |
| Falcon                        | ✅                 | ❌                    | ❌              |
| FLAN-T5                       | ✅                 | ✅                    | ✅              |
| FLAN-UL2                      | ✅                 | ✅                    | ✅              |
| FlauBERT                      | ✅                 | ✅                    | ❌              |
| FLAVA                         | ✅                 | ❌                    | ❌              |
| FNet                          | ✅                 | ❌                    | ❌              |
| FocalNet                      | ✅                 | ❌                    | ❌              |
| Funnel Transformer            | ✅                 | ✅                    | ❌


Supported models and frameworks

          |
| GIT                           | ✅                 | ❌                    | ❌              |
| GLPN                          | ✅                 | ❌                    | ❌              |
| GPT Neo                       | ✅                 | ❌                    | ✅              |
| GPT NeoX                      | ✅                 | ❌                    | ❌              |
| GPT NeoX Japanese             | ✅                 | ❌                    | ❌              |
| GPT-J                         | ✅                 | ✅                    | ✅              |
| GPT-Sw3                       | ✅                 | ✅                    | ✅              |
| GPTBigCode                    | ✅                 | ❌                    | ❌              |
| GPTSAN-japanese               | ✅                 | ❌                    | ❌              |
| Graphormer                    | ✅                 | ❌                    | ❌              |
| GroupViT


Supported models and frameworks

| ✅                 | ✅                    | ❌              |
| HerBERT                       | ✅                 | ✅                    | ✅              |
| Hubert                        | ✅                 | ✅                    | ❌              |
| I-BERT                        | ✅                 | ❌                    | ❌              |
| IDEFICS                       | ✅                 | ❌                    | ❌              |
| ImageGPT                      | ✅                 | ❌                    | ❌              |
| Informer                      | ✅                 | ❌                    | ❌              |
| InstructBLIP                  | ✅                 | ❌                    | ❌              |
| Jukebox                       | ✅                 | ❌                    | ❌              |
| LayoutLM                      | ✅                 | ✅                    | ❌              |
| LayoutLMv2                    | ✅                 | ❌                    |


Supported models and frameworks

 ❌              |
| LayoutLMv3                    | ✅                 | ✅                    | ❌              |
| LayoutXLM                     | ✅                 | ❌                    | ❌              |
| LED                           | ✅                 | ✅                    | ❌              |
| LeViT                         | ✅                 | ❌                    | ❌              |
| LiLT                          | ✅                 | ❌                    | ❌              |
| LLaMA                         | ✅                 | ❌                    | ❌              |
| Llama2                        | ✅                 | ❌                    | ❌              |
| Longformer                    | ✅                 | ✅                    | ❌              |
| LongT5                        | ✅                 | ❌                    | ✅              |
| LUKE                          | ✅                 | ❌                    | ❌              |
| LXMERT


Supported models and frameworks

      | ✅                 | ✅                    | ❌              |
| M-CTC-T                       | ✅                 | ❌                    | ❌              |
| M2M100                        | ✅                 | ❌                    | ❌              |
| Marian                        | ✅                 | ✅                    | ✅              |
| MarkupLM                      | ✅                 | ❌                    | ❌              |
| Mask2Former                   | ✅                 | ❌                    | ❌              |
| MaskFormer                    | ✅                 | ❌                    | ❌              |
| MatCha                        | ✅                 | ❌                    | ❌              |
| mBART                         | ✅                 | ✅                    | ✅              |
| mBART-50                      | ✅                 | ✅                    | ✅              |
| MEGA                          | ✅                 | ❌


Supported models and frameworks

     | ❌              |
| Megatron-BERT                 | ✅                 | ❌                    | ❌              |
| Megatron-GPT2                 | ✅                 | ✅                    | ✅              |
| MGP-STR                       | ✅                 | ❌                    | ❌              |
| Mistral                       | ✅                 | ❌                    | ❌              |
| mLUKE                         | ✅                 | ❌                    | ❌              |
| MMS                           | ✅                 | ✅                    | ✅              |
| MobileBERT                    | ✅                 | ✅                    | ❌              |
| MobileNetV1                   | ✅                 | ❌                    | ❌              |
| MobileNetV2                   | ✅                 | ❌                    | ❌              |
| MobileViT                     | ✅                 | ✅                    | ❌              |
| MobileViTV2


Supported models and frameworks

            | ✅                 | ❌                    | ❌              |
| MPNet                         | ✅                 | ✅                    | ❌              |
| MPT                           | ✅                 | ❌                    | ❌              |
| MRA                           | ✅                 | ❌                    | ❌              |
| MT5                           | ✅                 | ✅                    | ✅              |
| MusicGen                      | ✅                 | ❌                    | ❌              |
| MVP                           | ✅                 | ❌                    | ❌              |
| NAT                           | ✅                 | ❌                    | ❌              |
| Nezha                         | ✅                 | ❌                    | ❌              |
| NLLB                          | ✅                 | ❌                    | ❌              |
| NLLB-MOE                      | ✅                 | ❌


Supported models and frameworks

           | ❌              |
| Nougat                        | ✅                 | ✅                    | ✅              |
| Nyströmformer                 | ✅                 | ❌                    | ❌              |
| OneFormer                     | ✅                 | ❌                    | ❌              |
| OpenAI GPT                    | ✅                 | ✅                    | ❌              |
| OpenAI GPT-2                  | ✅                 | ✅                    | ✅              |
| OpenLlama                     | ✅                 | ❌                    | ❌              |
| OPT                           | ✅                 | ✅                    | ✅              |
| OWL-ViT                       | ✅                 | ❌                    | ❌              |
| Pegasus                       | ✅                 | ✅                    | ✅              |
| PEGASUS-X                     | ✅                 | ❌                    | ❌              |
| Perceiver


Supported models and frameworks

                  | ✅                 | ❌                    | ❌              |
| Persimmon                     | ✅                 | ❌                    | ❌              |
| PhoBERT                       | ✅                 | ✅                    | ✅              |
| Pix2Struct                    | ✅                 | ❌                    | ❌              |
| PLBart                        | ✅                 | ❌                    | ❌              |
| PoolFormer                    | ✅                 | ❌                    | ❌              |
| Pop2Piano                     | ✅                 | ❌                    | ❌              |
| ProphetNet                    | ✅                 | ❌                    | ❌              |
| PVT                           | ✅                 | ❌                    | ❌              |
| QDQBert                       | ✅                 | ❌                    | ❌              |
| RAG                           | ✅                 | ✅


Supported models and frameworks

                 | ❌              |
| REALM                         | ✅                 | ❌                    | ❌              |
| Reformer                      | ✅                 | ❌                    | ❌              |
| RegNet                        | ✅                 | ✅                    | ✅              |
| RemBERT                       | ✅                 | ✅                    | ❌              |
| ResNet                        | ✅                 | ✅                    | ✅              |
| RetriBERT                     | ✅                 | ❌                    | ❌              |
| RoBERTa                       | ✅                 | ✅                    | ✅              |
| RoBERTa-PreLayerNorm          | ✅                 | ✅                    | ✅              |
| RoCBert                       | ✅                 | ❌                    | ❌              |
| RoFormer                      | ✅                 | ✅                    | ✅              |
| RWKV


Supported models and frameworks

                        | ✅                 | ❌                    | ❌              |
| SAM                           | ✅                 | ✅                    | ❌              |
| SegFormer                     | ✅                 | ✅                    | ❌              |
| SEW                           | ✅                 | ❌                    | ❌              |
| SEW-D                         | ✅                 | ❌                    | ❌              |
| Speech Encoder decoder        | ✅                 | ❌                    | ✅              |
| Speech2Text                   | ✅                 | ✅                    | ❌              |
| SpeechT5                      | ✅                 | ❌                    | ❌              |
| Splinter                      | ✅                 | ❌                    | ❌              |
| SqueezeBERT                   | ✅                 | ❌                    | ❌              |
| SwiftFormer                   | ✅


Supported models and frameworks

| ❌                    | ❌              |
| Swin Transformer              | ✅                 | ✅                    | ❌              |
| Swin Transformer V2           | ✅                 | ❌                    | ❌              |
| Swin2SR                       | ✅                 | ❌                    | ❌              |
| SwitchTransformers            | ✅                 | ❌                    | ❌              |
| T5                            | ✅                 | ✅                    | ✅              |
| T5v1.1                        | ✅                 | ✅                    | ✅              |
| Table Transformer             | ✅                 | ❌                    | ❌              |
| TAPAS                         | ✅                 | ✅                    | ❌              |
| TAPEX                         | ✅                 | ✅                    | ✅              |
| Time Series Transformer       | ✅                 | ❌                    | ❌              |
|


Supported models and frameworks

TimeSformer                   | ✅                 | ❌                    | ❌              |
| Trajectory Transformer        | ✅                 | ❌                    | ❌              |
| Transformer-XL                | ✅                 | ✅                    | ❌              |
| TrOCR                         | ✅                 | ❌                    | ❌              |
| TVLT                          | ✅                 | ❌                    | ❌              |
| UL2                           | ✅                 | ✅                    | ✅              |
| UMT5                          | ✅                 | ❌                    | ❌              |
| UniSpeech                     | ✅                 | ❌                    | ❌              |
| UniSpeechSat                  | ✅                 | ❌                    | ❌              |
| UPerNet                       | ✅                 | ❌                    | ❌              |
| VAN                           | ✅