Diverse beam search decoding

ermaculture is a design system that encompasses a wide variety of disciplines, such "
...     "as ecology, landscape design, environmental science and energy conservation, and the "
...     "Permaculture design principles are drawn from these various disciplines. Each individual "
...     "design principle itself embodies a complete conceptual framework based on sound "
...     "scientific principles. When we bring all these separate  principles together, we can "
...     "create a design system that both looks at whole systems, the parts that these systems "
...     "consist of, and how those parts interact with each other to create a complex, dynamic, "
...     "living system. Each design principle serves as a tool that allows us to integrate all "
...     "the separate parts of a design, referred to as elements, into a functional, synergistic, "
...     "whole system, where the e


Diverse beam search decoding

lements harmoniously interact and work together in the most "
...     "efficient way possible."
... )
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")
>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)
>>> outputs = model.generate(**inputs, num_beams=5, num_beam_groups=5, max_new_tokens=30, diversity_penalty=1.0)
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'The Design Principles are a set of universal design principles that can be applied to any location, climate and
culture, and they allow us to design the'
This guide illustrates the main parameters that enable various decoding strategies. More advanced parameters exist for the
generate method, which gives you even further control over the generate method’s behavior.
For the complete list of the available parameters, refer to the API documentatio


Assisted Decoding

Assisted decoding is a modification of the decoding strategies above that uses an assistant model with the same
tokenizer (ideally a much smaller model) to greedily generate a few candidate tokens. The main model then validates
the candidate tokens in a single forward pass, which speeds up the decoding process. Currently, only greedy search
and sampling are supported with assisted decoding, and doesn’t support batched inputs. To learn more about assisted
decoding, check this blog post.
To enable assisted decoding, set the assistant_model argument with a model.
  Copied >>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> prompt = "Alice and Bob"
>>> checkpoint = "EleutherAI/pythia-1.4b-deduped"
>>> assistant_checkpoint = "E


Assisted Decoding

leutherAI/pythia-160m-deduped"
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")
>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)
>>> assistant_model = AutoModelForCausalLM.from_pretrained(assistant_checkpoint)
>>> outputs = model.generate(**inputs, assistant_model=assistant_model)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Alice and Bob are sitting in a bar. Alice is drinking a beer and Bob is drinking a']
When using assisted decoding with sampling methods, you can use the temperature argument to control the randomness
just like in multinomial sampling. However, in assisted decoding, reducing the temperature will help improving latency.
  Copied >


Assisted Decoding

>> from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed
>>> set_seed(42)  # For reproducibility
>>> prompt = "Alice and Bob"
>>> checkpoint = "EleutherAI/pythia-1.4b-deduped"
>>> assistant_checkpoint = "EleutherAI/pythia-160m-deduped"
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")
>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)
>>> assistant_model = AutoModelForCausalLM.from_pretrained(assistant_checkpoint)
>>> outputs = model.generate(**inputs, assistant_model=assistant_model, do_sample=True, temperature=0.5)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Alice and Bob are going to the same party. It is a small party, in a small'


How to create a custom pipeline?

In this guide, we will see how to create a custom pipeline and share it on the Hub or add it to the
🤗 Transformers library.
First and foremost, you need to decide the raw entries the pipeline will be able to take. It can be strings, raw bytes,
dictionaries or whatever seems to be the most likely desired input. Try to keep these inputs as pure Python as possible
as it makes compatibility easier (even through other languages via JSON). Those will be the inputs of the
pipeline (preprocess).
Then define the outputs. Same policy as the inputs. The simpler, the better. Those will be the outputs of
postprocess method.
Start by inheriting the base class Pipeline with the 4 methods needed to implement preprocess,
_forward, postprocess, and _sanitize_parameters.
  Copied from transformers import Pipeline
class MyPipeline(Pipeline):
    de


How to create a custom pipeline?

f _sanitize_parameters(self, **kwargs):
        preprocess_kwargs = {}
        if "maybe_arg" in kwargs:
            preprocess_kwargs["maybe_arg"] = kwargs["maybe_arg"]
        return preprocess_kwargs, {}, {}
    def preprocess(self, inputs, maybe_arg=2):
        model_input = Tensor(inputs["input_ids"])
        return {"model_input": model_input}
    def _forward(self, model_inputs):
        # model_inputs == {"model_input": model_input}
        outputs = self.model(**model_inputs)
        # Maybe {"logits": Tensor(...)}
        return outputs
    def postprocess(self, model_outputs):
        best_class = model_outputs["logits"].softmax(-1)
        return best_class
The structure of this breakdown is to support relatively seamless support for CPU/GPU, while supporting doing
pre/postprocessing on the CPU on different threads
p


How to create a custom pipeline?

reprocess will take the originally defined inputs, and turn them into something feedable to the model. It might
contain more information and is usually a Dict.
_forward is the implementation detail and is not meant to be called directly. forward is the preferred
called method as it contains safeguards to make sure everything is working on the expected device. If anything is
linked to a real model it belongs in the _forward method, anything else is in the preprocess/postprocess.
postprocess methods will take the output of _forward and turn it into the final output that was decided
earlier.
_sanitize_parameters exists to allow users to pass any parameters whenever they wish, be it at initialization
time pipeline(...., maybe_arg=4) or at call time pipe = pipeline(...); output = pipe(...., maybe_arg=4).
The returns of _sanitize_para


How to create a custom pipeline?

meters are the 3 dicts of kwargs that will be passed directly to preprocess,
_forward, and postprocess. Don’t fill anything if the caller didn’t call with any extra parameter. That
allows to keep the default arguments in the function definition which is always more “natural”.
A classic example would be a top_k argument in the post processing in classification tasks.
  Copied >>> pipe = pipeline("my-new-task")
>>> pipe("This is a test")
[{"label": "1-star", "score": 0.8}, {"label": "2-star", "score": 0.1}, {"label": "3-star", "score": 0.05}
{"label": "4-star", "score": 0.025}, {"label": "5-star", "score": 0.025}]
>>> pipe("This is a test", top_k=2)
[{"label": "1-star", "score": 0.8}, {"label": "2-star", "score": 0.1}]
In order to achieve that, we’ll update our postprocess method with a default parameter to 5. and edit
_sanitize_p


How to create a custom pipeline?

arameters to allow this new parameter.
  Copied def postprocess(self, model_outputs, top_k=5):
    best_class = model_outputs["logits"].softmax(-1)
    # Add logic to handle top_k
    return best_class
def _sanitize_parameters(self, **kwargs):
    preprocess_kwargs = {}
    if "maybe_arg" in kwargs:
        preprocess_kwargs["maybe_arg"] = kwargs["maybe_arg"]
    postprocess_kwargs = {}
    if "top_k" in kwargs:
        postprocess_kwargs["top_k"] = kwargs["top_k"]
    return preprocess_kwargs, {}, postprocess_kwargs
Try to keep the inputs/outputs very simple and ideally JSON-serializable as it makes the pipeline usage very easy
without requiring users to understand new kinds of objects. It’s also relatively common to support many different types
of arguments for ease of use (audio files, which can be filenames, URLs or pure byt


Adding it to the list of supported tasks

To register your new-task to the list of supported tasks, you have to add it to the PIPELINE_REGISTRY:
  Copied from transformers.pipelines import PIPELINE_REGISTRY
PIPELINE_REGISTRY.register_pipeline(
    "new-task",
    pipeline_class=MyPipeline,
    pt_model=AutoModelForSequenceClassification,
)
You can specify a default model if you want, in which case it should come with a specific revision (which can be the name of a branch or a commit hash, here we took "abcdef") as well as the type:
  Copied PIPELINE_REGISTRY.register_pipeline(
    "new-task",
    pipeline_class=MyPipeline,
    pt_model=AutoModelForSequenceClassification,
    default={"pt": ("user/awesome_model", "abcdef")},
    type="text",  # current support type: text, audio, image, multimodal
)


Share your pipeline on the Hub

To share your custom pipeline on the Hub, you just have to save the custom code of your Pipeline subclass in a
python file. For instance, let’s say we want to use a custom pipeline for sentence pair classification like this:
  Copied import numpy as np
from transformers import Pipeline
def softmax(outputs):
    maxes = np.max(outputs, axis=-1, keepdims=True)
    shifted_exp = np.exp(outputs - maxes)
    return shifted_exp / shifted_exp.sum(axis=-1, keepdims=True)
class PairClassificationPipeline(Pipeline):
    def _sanitize_parameters(self, **kwargs):
        preprocess_kwargs = {}
        if "second_text" in kwargs:
            preprocess_kwargs["second_text"] = kwargs["second_text"]
        return preprocess_kwargs, {}, {}
    def preprocess(self, text, se


Share your pipeline on the Hub

cond_text=None):
        return self.tokenizer(text, text_pair=second_text, return_tensors=self.framework)
    def _forward(self, model_inputs):
        return self.model(**model_inputs)
    def postprocess(self, model_outputs):
        logits = model_outputs.logits[0].numpy()
        probabilities = softmax(logits)
        best_class = np.argmax(probabilities)
        label = self.model.config.id2label[best_class]
        score = probabilities[best_class].item()
        logits = logits.tolist()
        return {"label": label, "score": score, "logits": logits}
The implementation is framework agnostic, and will work for PyTorch and TensorFlow models. If we have saved this in
a file named pair_classification.py, we can then import it and register it like this:


Share your pipeline on the Hub

  Copied from pair_classification import PairClassificationPipeline
from transformers.pipelines import PIPELINE_REGISTRY
from transformers import AutoModelForSequenceClassification, TFAutoModelForSequenceClassification
PIPELINE_REGISTRY.register_pipeline(
    "pair-classification",
    pipeline_class=PairClassificationPipeline,
    pt_model=AutoModelForSequenceClassification,
    tf_model=TFAutoModelForSequenceClassification,
)
Once this is done, we can use it with a pretrained model. For instance sgugger/finetuned-bert-mrpc has been
fine-tuned on the MRPC dataset, which classifies pairs of sentences as paraphrases or not.
  Copied from transformers import pipeline
classifier = pipeline("pair-classification", model="sgugger/finetuned-bert-mrpc")
Then we can


Share your pipeline on the Hub

 share it on the Hub by using the save_pretrained method in a Repository:
  Copied from huggingface_hub import Repository
repo = Repository("test-dynamic-pipeline", clone_from="{your_username}/test-dynamic-pipeline")
classifier.save_pretrained("test-dynamic-pipeline")
repo.push_to_hub()
This will copy the file where you defined PairClassificationPipeline inside the folder "test-dynamic-pipeline",
along with saving the model and tokenizer of the pipeline, before pushing everything into the repository
{your_username}/test-dynamic-pipeline. After that, anyone can use it as long as they provide the option
trust_remote_code=True:
  Copied from transformers import pipeline
classifier = pipeline(model="{your_username}/test-dynamic-pipeline", trust_remote_code=True)


Add the pipeline to 🤗 Transformers

If you want to contribute your pipeline to 🤗 Transformers, you will need to add a new module in the pipelines submodule
with the code of your pipeline, then add it to the list of tasks defined in pipelines/__init__.py.
Then you will need to add tests. Create a new file tests/test_pipelines_MY_PIPELINE.py with examples of the other tests.
The run_pipeline_test function will be very generic and run on small random models on every possible
architecture as defined by model_mapping and tf_model_mapping.
This is very important to test future compatibility, meaning if someone adds a new model for
XXXForQuestionAnswering then the pipeline test will attempt to run on it. Because the models are random it’s
impossible to check for actual values, that’s why there is a helper ANY that will simply attempt to match the
output of the pipeline TYPE.
You also need to implement 2 (ideally 4) tests.
test_small_mo


Add the pipeline to 🤗 Transformers

del_pt : Define 1 small model for this pipeline (doesn’t matter if the results don’t make sense)
and test the pipeline outputs. The results should be the same as test_small_model_tf. test_small_model_tf : Define 1 small model for this pipeline (doesn’t matter if the results don’t make sense)
and test the pipeline outputs. The results should be the same as test_small_model_pt. test_large_model_pt (optional): Tests the pipeline on a real pipeline where the results are supposed to
make sense. These tests are slow and should be marked as such. Here the goal is to showcase the pipeline and to make
sure there is no drift in future releases. test_large_model_tf (optional): Tests the pipeline on a real pipeline where the results are supposed to
make sense. These tests are slow and should be marked as such. Here the goal is to showcase the pipeline and to make
sure there is no drift in future releases.


Training on Specialized Hardware

Note: Most of the strategies introduced in the single GPU section (such as mixed precision training or gradient accumulation) and multi-GPU section are generic and apply to training models in general so make sure to have a look at it before diving into this section.
This document will be completed soon with information on how to train on specialized hardware.


Optimize inference using torch.compile()

This guide aims to provide a benchmark on the inference speed-ups introduced with torch.compile() for computer vision models in 🤗 Transformers.


Benefits of torch.compile

Depending on the model and the GPU, torch.compile() yields up to 30% speed-up during inference. To use torch.compile(), simply install any version of torch above 2.0.
Compiling a model takes time, so it’s useful if you are compiling the model only once instead of every time you infer.
To compile any computer vision model of your choice, call torch.compile() on the model as shown below:
  Copied from transformers import AutoModelForImageClassification
model = AutoModelForImageClassification.from_pretrained(MODEL_ID).to("cuda")
+ model =


Benefits of torch.compile

 torch.compile(model)
compile() comes with multiple modes for compiling, which essentially differ in compilation time and inference overhead. max-autotune takes longer than reduce-overhead but results in faster inference. Default mode is fastest for compilation but is not as efficient compared to reduce-overhead for inference time. In this guide, we used the default mode. You can learn more about it here.
We benchmarked torch.compile with different computer vision models, tasks, types of hardware, and batch sizes on torch version 2.0.1


Benchmarking code

Below you can find the benchmarking code for each task. We warm up the GPU before inference and take the mean time of 300 inferences, using the same image each time.


Image Classification with ViT

Copied import torch
from PIL import Image
import requests
import numpy as np
from transformers import AutoImageProcessor, AutoModelForImageClassification
url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)
processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224")
model = AutoModelForImageClassification.from_pretrained("google/vit-base-patch16-224").to("cuda")
model = torch.compile(model)
processed_input = processor(image, return_tensors='pt').to(device="cuda")
with torch.no_grad():
    _ = model(**processed_input)
 Object Detection with DETR
  Copied from transformers import AutoImageProcessor, AutoModelForObjectDetection
processor = AutoImageProcessor.from_pretrained("facebook/detr-resnet-50")
mo


Image Classification with ViT

del = AutoModelForObjectDetection.from_pretrained("facebook/detr-resnet-50").to("cuda")
model = torch.compile(model)
texts = ["a photo of a cat", "a photo of a dog"]
inputs = processor(text=texts, images=image, return_tensors="pt").to("cuda")
with torch.no_grad():
    _ = model(**inputs)
 Image Segmentation with Segformer
  Copied from transformers import SegformerImageProcessor, SegformerForSemanticSegmentation
processor = SegformerImageProcessor.from_pretrained("nvidia/segformer-b0-finetuned-ade-512-512")
model = SegformerForSemanticSegmentation.from_pretrained("nvidia/segformer-b0-finetuned-ade-512-512").to("cuda")
model = torch.compile(model)
seg_inputs = processor(images=image, return_tensors="pt").to("cuda")
with torch.no_grad():
    _ = model(**seg_inputs)
Below you can find


Image Classification with ViT

 the list of the models we benchmarked.
Image Classification
google/vit-base-patch16-224 microsoft/beit-base-patch16-224-pt22k-ft22k facebook/convnext-large-224 microsoft/resnet-50
Image Segmentation
nvidia/segformer-b0-finetuned-ade-512-512 facebook/mask2former-swin-tiny-coco-panoptic facebook/maskformer-swin-base-ade google/deeplabv3_mobilenet_v2_1.0_513
Object Detection
google/owlvit-base-patch32 facebook/detr-resnet-101 microsoft/conditional-detr-resnet-50
Below you can find visualization of inference durations with and without torch.compile() and percentage improvements for each model in different hardware and batch sizes.

Below you can find inference durations in milliseconds for each model with and without compile(). Note that OwlViT results in OOM in larger batch sizes


A100 (batch size: 1)

| Task/Model                        |   torch 2.0 -  no compile |   torch 2.0 -  compile |
|-----------------------------------|---------------------------|------------------------|
| Image Classification/ViT          |                     9.325 |                  7.584 |
| Image Segmentation/Segformer      |                    11.759 |                 10.5   |
| Object Detection/OwlViT           |                    24.978 |                 18.42  |
| Image Classification/BeiT         |                    11.282 |                  8.448 |
| Object Detection/DETR             |                    34.619 |                 19.04  |


A100 (batch size: 1)

| Image Classification/ConvNeXT     |                    10.41  |                 10.208 |
| Image Classification/ResNet       |                     6.531 |                  4.124 |
| Image Segmentation/Mask2former    |                    60.188 |                 49.117 |
| Image Segmentation/Maskformer     |                    75.764 |                 59.487 |
| Image Segmentation/MobileNet      |                     8.583 |                  3.974 |
| Object Detection/Resnet-101       |                    36.276 |                 18.197 |
| Object Detection/Conditional-DETR |                    31.219 |                 17.993


A100 (batch size: 4)

| Task/Model                        |   torch 2.0 -  no compile |   torch 2.0 -  compile |
|-----------------------------------|---------------------------|------------------------|
| Image Classification/ViT          |                    14.832 |                 14.499 |
| Image Segmentation/Segformer      |                    18.838 |                 16.476 |
| Image Classification/BeiT         |                    13.205 |                 13.048 |
| Object Detection/DETR             |                    48.657 |                 32.418 |
| Image Classification/ConvNeXT     |


A100 (batch size: 4)

            22.94  |                 21.631 |
| Image Classification/ResNet       |                     6.657 |                  4.268 |
| Image Segmentation/Mask2former    |                    74.277 |                 61.781 |
| Image Segmentation/Maskformer     |                   180.7   |                159.116 |
| Image Segmentation/MobileNet      |                    14.174 |                  8.515 |
| Object Detection/Resnet-101       |                    68.101 |                 44.998 |
| Object Detection/Conditional-DETR |                    56.47  |                 35.552 |


A100 (batch size: 16)

| Task/Model                        |   torch 2.0 -  no compile |   torch 2.0 -  compile |
|-----------------------------------|---------------------------|------------------------|
| Image Classification/ViT          |                    40.944 |                 40.01  |
| Image Segmentation/Segformer      |                    37.005 |                 31.144 |
| Image Classification/BeiT         |                    41.854 |                 41.048 |
| Object Detection/DETR             |                   164.382 |                161.902 |
| Image Classification/ConvNeXT     |


A100 (batch size: 16)

            82.258 |                 75.561 |
| Image Classification/ResNet       |                     7.018 |                  5.024 |
| Image Segmentation/Mask2former    |                   178.945 |                154.814 |
| Image Segmentation/Maskformer     |                   638.57  |                579.826 |
| Image Segmentation/MobileNet      |                    51.693 |                 30.31  |
| Object Detection/Resnet-101       |                   232.887 |                155.021 |
| Object Detection/Conditional-DETR |                   180.491 |                124.032 |


V100 (batch size: 1)

| Task/Model                        |   torch 2.0 -  no compile |   torch 2.0 -  compile |
|-----------------------------------|---------------------------|------------------------|
| Image Classification/ViT          |                    10.495 |                  6     |
| Image Segmentation/Segformer      |                    13.321 |                  5.862 |
| Object Detection/OwlViT           |                    25.769 |                 22.395 |
| Image Classification/BeiT         |                    11.347 |                  7.234 |
| Object Detection/DETR             |                    33.951 |                 19.388 |


V100 (batch size: 1)

| Image Classification/ConvNeXT     |                    11.623 |                 10.412 |
| Image Classification/ResNet       |                     6.484 |                  3.82  |
| Image Segmentation/Mask2former    |                    64.64  |                 49.873 |
| Image Segmentation/Maskformer     |                    95.532 |                 72.207 |
| Image Segmentation/MobileNet      |                     9.217 |                  4.753 |
| Object Detection/Resnet-101       |                    52.818 |                 28.367 |
| Object Detection/Conditional-DETR |                    39.512 |                 20.816


V100 (batch size: 4)

| Task/Model                        |   torch 2.0 -  no compile |   torch 2.0 -  compile |
|-----------------------------------|---------------------------|------------------------|
| Image Classification/ViT          |                    15.181 |                 14.501 |
| Image Segmentation/Segformer      |                    16.787 |                 16.188 |
| Image Classification/BeiT         |                    15.171 |                 14.753 |
| Object Detection/DETR             |                    88.529 |                 64.195 |
| Image Classification/ConvNeXT     |


V100 (batch size: 4)

            29.574 |                 27.085 |
| Image Classification/ResNet       |                     6.109 |                  4.731 |
| Image Segmentation/Mask2former    |                    90.402 |                 76.926 |
| Image Segmentation/Maskformer     |                   234.261 |                205.456 |
| Image Segmentation/MobileNet      |                    24.623 |                 14.816 |
| Object Detection/Resnet-101       |                   134.672 |                101.304 |
| Object Detection/Conditional-DETR |                    97.464 |                 69.739 |


V100 (batch size: 16)

| Task/Model                        | torch 2.0 -  no compile   | torch 2.0 -  compile   |
|-----------------------------------|---------------------------|------------------------|
| Image Classification/ViT          | 52.209                    | 51.633                 |
| Image Segmentation/Segformer      | 61.013                    | 55.499                 |
| Image Classification/BeiT         | 53.938                    | 53.581                 |
| Object Detection/DETR             | OOM                       | OOM                    |
| Image Classification/ConvNeXT     | 109.682


V100 (batch size: 16)

                   | 100.771                |
| Image Classification/ResNet       | 14.857                    | 12.089                 |
| Image Segmentation/Mask2former    | 249.605                   | 222.801                |
| Image Segmentation/Maskformer     | 831.142                   | 743.645                |
| Image Segmentation/MobileNet      | 93.129                    | 55.365                 |
| Object Detection/Resnet-101       | 482.425                   | 361.843                |
| Object Detection/Conditional-DETR | 344.661                   | 255.298                |


T4 (batch size: 1)

| Task/Model                        |   torch 2.0 -  no compile |   torch 2.0 -  compile |
|-----------------------------------|---------------------------|------------------------|
| Image Classification/ViT          |                    16.52  |                 15.786 |
| Image Segmentation/Segformer      |                    16.116 |                 14.205 |
| Object Detection/OwlViT           |                    53.634 |                 51.105 |
| Image Classification/BeiT         |                    16.464 |                 15.71  |
| Object Detection/DETR             |                    73.1   |                 53.99  |


T4 (batch size: 1)

| Image Classification/ConvNeXT     |                    32.932 |                 30.845 |
| Image Classification/ResNet       |                     6.031 |                  4.321 |
| Image Segmentation/Mask2former    |                    79.192 |                 66.815 |
| Image Segmentation/Maskformer     |                   200.026 |                188.268 |
| Image Segmentation/MobileNet      |                    18.908 |                 11.997 |
| Object Detection/Resnet-101       |                   106.622 |                 82.566 |
| Object Detection/Conditional-DETR |                    77.594 |                 56.984


T4 (batch size: 4)

| Task/Model                        |   torch 2.0 -  no compile |   torch 2.0 -  compile |
|-----------------------------------|---------------------------|------------------------|
| Image Classification/ViT          |                    43.653 |                 43.626 |
| Image Segmentation/Segformer      |                    45.327 |                 42.445 |
| Image Classification/BeiT         |                    52.007 |                 51.354 |
| Object Detection/DETR             |                   277.85  |                268.003 |
| Image Classification/ConvNeXT     |


T4 (batch size: 4)

           119.259 |                105.58  |
| Image Classification/ResNet       |                    13.039 |                 11.388 |
| Image Segmentation/Mask2former    |                   201.54  |                184.67  |
| Image Segmentation/Maskformer     |                   764.052 |                711.28  |
| Image Segmentation/MobileNet      |                    74.289 |                 48.677 |
| Object Detection/Resnet-101       |                   421.859 |                357.614 |
| Object Detection/Conditional-DETR |                   289.002 |                226.945 |


T4 (batch size: 16)

| Task/Model                        | torch 2.0 -  no compile   | torch 2.0 -  compile   |
|-----------------------------------|---------------------------|------------------------|
| Image Classification/ViT          | 163.914                   | 160.907                |
| Image Segmentation/Segformer      | 192.412                   | 163.620                |
| Image Classification/BeiT         | 188.978                   | 187.976                |
| Object Detection/DETR             | OOM                       | OOM                    |
| Image Classification/ConvNeXT     | 422.886


T4 (batch size: 16)

                   | 388.078                |
| Image Classification/ResNet       | 44.114                    | 37.604                 |
| Image Segmentation/Mask2former    | 756.337                   | 695.291                |
| Image Segmentation/Maskformer     | 2842.940                  | 2656.88                |
| Image Segmentation/MobileNet      | 299.003                   | 201.942                |
| Object Detection/Resnet-101       | 1619.505                  | 1262.758               |
| Object Detection/Conditional-DETR | 1137.513                  | 897.390                |


PyTorch Nightly

We also benchmarked on PyTorch nightly (2.1.0dev, find the wheel here) and observed improvement in latency both for uncompiled and compiled models.


A100

| Task/Model                | Batch Size   |   torch 2.0 - no compile |   torch 2.0 -  compile |
|---------------------------|--------------|--------------------------|------------------------|
| Image Classification/BeiT | Unbatched    |                   12.462 |                  6.954 |
| Image Classification/BeiT | 4            |                   14.109 |                 12.851 |
| Image Classification/BeiT | 16           |                   42.179 |                 42.147 |
| Object Detection/DETR     | Unbatched    |                   30.484 |                 15.221 |
| Object Detection/DETR     | 4            |                   46.816 |                 30.942 |
| Object Detection/DETR     | 16           |                  163.749 |                163.706 |


T4

| Task/Model                | Batch Size   | torch 2.0 -  no compile   | torch 2.0 -  compile   |
|---------------------------|--------------|---------------------------|------------------------|
| Image Classification/BeiT | Unbatched    | 14.408                    | 14.052                 |
| Image Classification/BeiT | 4            | 47.381                    | 46.604                 |
| Image Classification/BeiT | 16           | 42.179                    | 42.147                 |
| Object Detection/DETR     | Unbatched    | 68.382                    | 53.481                 |
| Object Detection/DETR     | 4            | 269.615                   | 204.785                |
| Object Detection/DETR     | 16           | OOM                       | OOM                    |
###


T4

V100
| Task/Model                | Batch Size   | torch 2.0 -  no compile   | torch 2.0 -  compile   |
|---------------------------|--------------|---------------------------|------------------------|
| Image Classification/BeiT | Unbatched    | 13.477                    | 7.926                  |
| Image Classification/BeiT | 4            | 15.103                    | 14.378                 |
| Image Classification/BeiT | 16           | 52.517                    | 51.691                 |
| Object Detection/DETR     | Unbatched    | 28.706                    | 19.077                 |
| Object Detection/DETR     | 4            | 88.402                    | 62.949                 |
| Object Detection/DETR     | 16           | OOM                       | OOM                    |


A100

| Task/Model                        | Batch Size   |   torch 2.0 -  no compile |   torch 2.0 -  compile |
|-----------------------------------|--------------|---------------------------|------------------------|
| Image Classification/ConvNeXT     | Unbatched    |                    11.758 |                  7.335 |
| Image Classification/ConvNeXT     | 4            |                    23.171 |                 21.49  |
| Image Classification/ResNet       | Unbatched    |                     7.435 |                  3.801 |


A100

| Image Classification/ResNet       | 4            |                     7.261 |                  2.187 |
| Object Detection/Conditional-DETR | Unbatched    |                    32.823 |                 11.627 |
| Object Detection/Conditional-DETR | 4            |                    50.622 |                 33.831 |
| Image Segmentation/MobileNet      | Unbatched    |                     9.869 |                  4.244 |
| Image Segmentation/MobileNet      | 4            |                    14.385 |                  7.946


T4

| Task/Model                        | Batch Size   |   torch 2.0 -  no compile |   torch 2.0 -  compile |
|-----------------------------------|--------------|---------------------------|------------------------|
| Image Classification/ConvNeXT     | Unbatched    |                    32.137 |                 31.84  |
| Image Classification/ConvNeXT     | 4            |                   120.944 |                110.209 |
| Image Classification/ResNet       | Unbatched    |                     9.761 |                  7.698 |


T4

| Image Classification/ResNet       | 4            |                    15.215 |                 13.871 |
| Object Detection/Conditional-DETR | Unbatched    |                    72.15  |                 57.66  |
| Object Detection/Conditional-DETR | 4            |                   301.494 |                247.543 |
| Image Segmentation/MobileNet      | Unbatched    |                    22.266 |                 19.339 |
| Image Segmentation/MobileNet      | 4            |                    78.311 |                 50.983


Model training anatomy

To understand performance optimization techniques that one can apply to improve efficiency of model training
speed and memory utilization, it’s helpful to get familiar with how GPU is utilized during training, and how compute
intensity varies depending on an operation performed.
Let’s start by exploring a motivating example of GPU utilization and the training run of a model. For the demonstration,
we’ll need to install a few libraries:
  Copied pip install transformers datasets accelerate nvidia-ml-py3
The nvidia-ml-py3 library allows us to monitor the memory usage of the models from within Python. You might be familiar
with the nvidia-smi command in the terminal - this library allows to access the same information in Python directly.
Then, we create some dummy data: random token IDs between 100 and 30000 and binary labels for a classifier.
In total, we get 512 sequenc


Model training anatomy

es each with length 512 and store them in a Dataset with PyTorch format.
  Copied >>> import numpy as np
>>> from datasets import Dataset
>>> seq_len, dataset_size = 512, 512
>>> dummy_data = {
...     "input_ids": np.random.randint(100, 30000, (dataset_size, seq_len)),
...     "labels": np.random.randint(0, 1, (dataset_size)),
... }
>>> ds = Dataset.from_dict(dummy_data)
>>> ds.set_format("pt")
To print summary statistics for the GPU utilization and the training run with the Trainer we define two helper functions:
  Copied >>> from pynvml import *
>>> def print_gpu_utilization():
...     nvmlInit()
...     handle = nvmlDeviceGetHandleByIndex(0)
...     info = nvmlDeviceGetMemoryInfo(handle)
...     print(f"GPU memory occupied: {info.used//1024**2} MB.")
>>> def print_summary(result):
...     print(f"Time: {result.metrics['train_runtime']:.2f}")
...     print(f"Samples


Model training anatomy

/second: {result.metrics['train_samples_per_second']:.2f}")
...     print_gpu_utilization()
Let’s verify that we start with a free GPU memory:
  Copied >>> print_gpu_utilization()
GPU memory occupied: 0 MB.
That looks good: the GPU memory is not occupied as we would expect before we load any models. If that’s not the case on
your machine make sure to stop all processes that are using GPU memory. However, not all free GPU memory can be used by
the user. When a model is loaded to the GPU the kernels are also loaded, which can take up 1-2GB of memory. To see how
much it is we load a tiny tensor into the GPU which triggers the kernels to be loaded as well.
  Copied >>> import torch
>>> torch.ones((1, 1)).to("cuda")
>>> print_gpu_utilization()
GPU memory occupied: 1343 MB.
We see that the kernels alone take up 1.3GB of GPU memory. Now let’s see how much space the model uses


Load Model

First, we load the bert-large-uncased model. We load the model weights directly to the GPU so that we can check
how much space just the weights use.
  Copied >>> from transformers import AutoModelForSequenceClassification
>>> model = AutoModelForSequenceClassification.from_pretrained("bert-large-uncased").to("cuda")
>>> print_gpu_utilization()
GPU memory occupied: 2631 MB.
We can see that the model weights alone take up 1.3 GB of GPU memory. The exact number depends on the specific
GPU you are using. Note that on newer GPUs a model can sometimes take up more space since the weights are loaded in an
optimized fashion that speeds up the usage of the model. Now we can also quickly check if we get the same result
as with nvidia-smi CLI:
  Copied nvidia-smi
  Copied Tue Jan 11 08:58:05 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.91.03    Driver Version: 460.91.03


Load Model

 CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:04.0 Off |                    0 |
| N/A   37C    P0    39W / 300W |   2631MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        P


Load Model

ID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      3721      C   ...nvs/codeparrot/bin/python     2629MiB |
+-----------------------------------------------------------------------------+
We get the same number as before and you can also see that we are using a V100 GPU with 16GB of memory. So now we can
start training the model and see how the GPU memory consumption changes. First, we set up a few standard training
arguments:
  Copied default_args = {
    "output_dir": "tmp",
    "evaluation_strategy": "steps",
    "num_train_epochs": 1,
    "log_level": "error",
    "report_to": "none",
}
If you plan to run multiple experiments, in order to properly clear the memory between experiments, restart the Python
kernel between experiments.


Memory utilization at vanilla training

Let’s use the Trainer and train the model without using any GPU performance optimization techniques and a batch size of 4:
  Copied >>> from transformers import TrainingArguments, Trainer, logging
>>> logging.set_verbosity_error()
>>> training_args = TrainingArguments(per_device_train_batch_size=4, **default_args)
>>> trainer = Trainer(model=model, args=training_args, train_dataset=ds)
>>> result = trainer.train()
>>> print_summary(result)
  Copied Time: 57.82
Samples/second: 8.86
GPU memory occupied: 14949 MB.
We see that already a relatively small batch size almost fills up our GPU’s entire memory. However, a larger batch size
can often result in faster model convergence or better end performance. So ideally we want to tune the batch size to our
model’s needs and not to the GPU limitations. What’s interesting is that we use much more memory than the size of the model.
To understand a bit better why this is the case let’s have a look at a model’s operations and memory needs.


Anatomy of Model’s Operations

Transformers architecture includes 3 main groups of operations grouped below by compute-intensity.
Tensor Contractions Linear layers and components of Multi-Head Attention all do batched matrix-matrix multiplications. These operations are the most compute-intensive part of training a transformer. Statistical Normalizations Softmax and layer normalization are less compute-intensive than tensor contractions, and involve one or more reduction operations, the result of which is then applied via a map. Element-wise Operators These are the remaining operators: biases, dropout, activations, and residual connections. These are the least compute-intensive operations.
This knowledge can be helpful to know when analyzing performance bottlenecks.
This summary is derived from Data Movement Is All You Need: A Case Study on Optimizing Transformers 2020


Anatomy of Model’s Memory

We’ve seen that training the model uses much more memory than just putting the model on the GPU. This is because there
are many components during training that use GPU memory. The components on GPU memory are the following:
model weights optimizer states gradients forward activations saved for gradient computation temporary buffers functionality-specific memory
A typical model trained in mixed precision with AdamW requires 18 bytes per model parameter plus activation memory. For
inference there are no optimizer states and gradients, so we can subtract those. And thus we end up with 6 bytes per
model parameter for mixed precision inference, plus activation memory.
Let’s look at the details.
Model Weights:
4 bytes * number of parameters for fp32 training 6 bytes * number of parameters for mixed precision training (maintains a model in fp32 and one in fp16 in memory)
Optimizer States:
8 bytes * number of parameters for normal AdamW (maintains 2 states) 2 bytes * number of parame


Anatomy of Model’s Memory

ters for 8-bit AdamW optimizers like bitsandbytes 4 bytes * number of parameters for optimizers like SGD with momentum (maintains only 1 state)
Gradients
4 bytes * number of parameters for either fp32 or mixed precision training (gradients are always kept in fp32)
Forward Activations
size depends on many factors, the key ones being sequence length, hidden size and batch size.
There are the input and output that are being passed and returned by the forward and the backward functions and the
forward activations saved for gradient computation.
Temporary Memory
Additionally, there are all kinds of temporary variables which get released once the calculation is done, but in the
moment these could require additional memory and could push to OOM. Therefore, when coding it’s crucial to think
strategically about such temporary variables and sometimes to explicitly free those as soon as they are no longer needed.
Functionality-specific memory
Then, your software could have special memor


Anatomy of Model’s Memory

y needs. For example, when generating text using beam search, the software
needs to maintain multiple copies of inputs and outputs.
forward vs backward Execution Speed
For convolutions and linear layers there are 2x flops in the backward compared to the forward, which generally translates
into ~2x slower (sometimes more, because sizes in the backward tend to be more awkward). Activations are usually
bandwidth-limited, and it’s typical for an activation to have to read more data in the backward than in the forward
(e.g. activation forward reads once, writes once, activation backward reads twice, gradOutput and output of the forward,
and writes once, gradInput).
As you can see, there are potentially a few places where we could save GPU memory or speed up operations.
Now that you understand what affects GPU utilization and computation speed, refer to
the Methods and tools for efficient training on a single GPU documentation page to learn about
performance optimization techniques


How 🤗 Transformers solve tasks

In What 🤗 Transformers can do, you learned about natural language processing (NLP), speech and audio, computer vision tasks, and some important applications of them. This page will look closely at how models solve these tasks and explain what’s happening under the hood. There are many ways to solve a given task, some models may implement certain techniques or even approach the task from a new angle, but for Transformer models, the general idea is the same. Owing to its flexible architecture, most models are a variant of an encoder, decoder, or encoder-decoder structure. In addition to Transformer models, our library also has several convolutional neural networks (CNNs), which are still used today for computer vision tasks. We’ll also explain how a modern CNN works.
To explain how tasks are solved, we’ll walk through what goes on


How 🤗 Transformers solve tasks

 inside the model to output useful predictions.
Wav2Vec2 for audio classification and automatic speech recognition (ASR) Vision Transformer (ViT) and ConvNeXT for image classification DETR for object detection Mask2Former for image segmentation GLPN for depth estimation BERT for NLP tasks like text classification, token classification and question answering that use an encoder GPT2 for NLP tasks like text generation that use a decoder BART for NLP tasks like summarization and translation that use an encoder-decoder
Before you go further, it is good to have some basic knowledge of the original Transformer architecture. Knowing how encoders, decoders, and attention work will aid you in understanding how different Transformer models work. If you’re just getting started or need a refresher, check out our course for more information!


Speech and audio

Wav2Vec2 is a self-supervised model pretrained on unlabeled speech data and finetuned on labeled data for audio classification and automatic speech recognition.
This model has four main components:
A feature encoder takes the raw audio waveform, normalizes it to zero mean and unit variance, and converts it into a sequence of feature vectors that are each 20ms long. Waveforms are continuous by nature, so they can’t be divided into separate units like a sequence of text can be split into words. That’s why the feature vectors are passed to a quantization module, which aims to learn discrete speech units. The speech unit is chosen from a collection of codewords, known as a codebook (you can think of this as the vocabulary). From the


Speech and audio

codebook, the vector or speech unit, that best represents the continuous audio input is chosen and forwarded through the model. About half of the feature vectors are randomly masked, and the masked feature vector is fed to a context network, which is a Transformer encoder that also adds relative positional embeddings. The pretraining objective of the context network is a contrastive task. The model has to predict the true quantized speech representation of the masked prediction from a set of false ones, encouraging the model to find the most similar context vector and quantized speech unit (the target label).
Now that wav2vec2 is pretrained, you can finetune it on your data for audio classification or automatic speech recognition


Audio classification

To use the pretrained model for audio classification, add a sequence classification head on top of the base Wav2Vec2 model. The classification head is a linear layer that accepts the encoder’s hidden states. The hidden states represent the learned features from each audio frame which can have varying lengths. To create one vector of fixed-length, the hidden states are pooled first and then transformed into logits over the class labels. The cross-entropy loss is calculated between the logits and target to find the most likely class.
Ready to try your hand at audio classification? Check out our complete audio classification guide to learn how to finetune Wav2Vec2 and use it for inference!


Automatic speech recognition

To use the pretrained model for automatic speech recognition, add a language modeling head on top of the base Wav2Vec2 model for connectionist temporal classification (CTC). The language modeling head is a linear layer that accepts the encoder’s hidden states and transforms them into logits. Each logit represents a token class (the number of tokens comes from the task vocabulary). The CTC loss is calculated between the logits and targets to find the most likely sequence of tokens, which are then decoded into a transcription.
Ready to try your hand at automatic speech recognition? Check out our complete automatic speech recognition guide to learn how to finetune Wav2Vec2 and use it for inference!


Computer vision

There are two ways to approach computer vision tasks:
Split an image into a sequence of patches and process them in parallel with a Transformer. Use a modern CNN, like ConvNeXT, which relies on convolutional layers but adopts modern network designs.
A third approach mixes Transformers with convolutions (for example, Convolutional Vision Transformer or LeViT). We won’t discuss those because they just combine the two approaches we examine here.
ViT and ConvNeXT are commonly used for image classification, but for other vision tasks like object detection, segmentation, and depth estimation, we’ll look at DETR, Mask2Former and GLPN, respectively; these models are better suited for those tasks.


Image classification

ViT and ConvNeXT can both be used for image classification; the main difference is that ViT uses an attention mechanism while ConvNeXT uses convolutions.
 Transformer
ViT replaces convolutions entirely with a pure Transformer architecture. If you’re familiar with the original Transformer, then you’re already most of the way toward understanding ViT.
The main change ViT introduced was in how images are fed to a Transformer:
An image is split into square non-overlapping patches, each of which gets turned into a vector or patch embedding. The patch embeddings are generated from a convolutional 2D layer which creates the proper input dimensions (which for a base Transformer is 768 values for each patch embedding). If you had a 224x224 pixel image, you could split it into 196 16x16 image patches. Just like how text is tokenized into words, an image is “tokenized” into a sequence of patches. A learnable embedding - a spec


Image classification

ial [CLS] token - is added to the beginning of the patch embeddings just like BERT. The final hidden state of the [CLS] token is used as the input to the attached classification head; other outputs are ignored. This token helps the model learn how to encode a representation of the image. The last thing to add to the patch and learnable embeddings are the position embeddings because the model doesn’t know how the image patches are ordered. The position embeddings are also learnable and have the same size as the patch embeddings. Finally, all of the embeddings are passed to the Transformer encoder. The output, specifically only the output with the [CLS] token, is passed to a multilayer perceptron head (MLP). ViT’s pretraining objective is simply classification. Like other classification heads, the MLP head converts the output into logits over the class labels and calculates the cross-entropy loss to find the most like


Image classification

ly class.
Ready to try your hand at image classification? Check out our complete image classification guide to learn how to finetune ViT and use it for inference!
 CNN
This section briefly explains convolutions, but it’d be helpful to have a prior understanding of how they change an image’s shape and size. If you’re unfamiliar with convolutions, check out the Convolution Neural Networks chapter from the fastai book!
ConvNeXT is a CNN architecture that adopts new and modern network designs to improve performance. However, convolutions are still at the core of the model. From a high-level perspective, a convolution is an operation where a smaller matrix (kernel) is multiplied by a small window of the image pixels. It computes some features from it, such as a particular texture or curvature of a line. Then it slides over to the next window of pixels; the distance the convolution travels is known as the stride.
A basic


Image classification

convolution without padding or stride, taken from A guide to convolution arithmetic for deep learning.
You can feed this output to another convolutional layer, and with each successive layer, the network learns more complex and abstract things like hotdogs or rockets. Between convolutional layers, it is common to add a pooling layer to reduce dimensionality and make the model more robust to variations of a feature’s position.
ConvNeXT modernizes a CNN in five ways:
Change the number of blocks in each stage and “patchify” an image with a larger stride and corresponding kernel size. The non-overlapping sliding window makes this patchifying strategy similar to how ViT splits an image into patches. A bottleneck layer shrinks the number of channels and then restores it because it is faster to do a 1x1 convolution, and you can increase the depth. An inverted bottleneck does the opposite by expanding the number of channels


Image classification

 and shrinking them, which is more memory efficient. Replace the typical 3x3 convolutional layer in the bottleneck layer with depthwise convolution, which applies a convolution to each input channel separately and then stacks them back together at the end. This widens the network width for improved performance. ViT has a global receptive field which means it can see more of an image at once thanks to its attention mechanism. ConvNeXT attempts to replicate this effect by increasing the kernel size to 7x7. ConvNeXT also makes several layer design changes that imitate Transformer models. There are fewer activation and normalization layers,  the activation function is switched to GELU instead of ReLU, and it uses LayerNorm instead of BatchNorm.
The output from the convolution blocks is passed to a classification head which converts the outputs into logits and calculates the cross-entropy loss to find the most likely lab


Object detection

DETR, DEtection TRansformer, is an end-to-end object detection model that combines a CNN with a Transformer encoder-decoder.
A pretrained CNN backbone takes an image, represented by its pixel values, and creates a low-resolution feature map of it. A 1x1 convolution is applied to the feature map to reduce dimensionality and it creates a new feature map with a high-level image representation. Since the Transformer is a sequential model, the feature map is flattened into a sequence of feature vectors that are combined with positional embeddings. The feature vectors are passed to the encoder, which learns the image representations using its attention layers. Next, the encoder hidden states are combined with object queries in the decoder. Object queries are learned embeddings that focus on the different


Object detection

regions of an image, and they’re updated as they progress through each attention layer. The decoder hidden states are passed to a feedforward network that predicts the bounding box coordinates and class label for each object query, or no object if there isn’t one. DETR decodes each object query in parallel to output N final predictions, where N is the number of queries. Unlike a typical autoregressive model that predicts one element at a time, object detection is a set prediction task (bounding box, class label) that makes N predictions in a single pass. DETR uses a bipartite matching loss during training to compare a fixed number of predictions with a fixed set of ground truth labels. If there are fewer ground truth labels in the set of N labels, then they’re padded with a no object class. This los


Object detection

s function encourages DETR to find a one-to-one assignment between the predictions and ground truth labels. If either the bounding boxes or class labels aren’t correct, a loss is incurred. Likewise, if DETR predicts an object that doesn’t exist, it is penalized. This encourages DETR to find other objects in an image instead of focusing on one really prominent object.
An object detection head is added on top of DETR to find the class label and the coordinates of the bounding box. There are two components to the object detection head: a linear layer to transform the decoder hidden states into logits over the class labels, and a MLP to predict the bounding box.
Ready to try your hand at object detection? Check out our complete object detection guide to learn how to finetune DETR and use it for inferenc


Image segmentation

Mask2Former is a universal architecture for solving all types of image segmentation tasks. Traditional segmentation models are typically tailored towards a particular subtask of image segmentation, like instance, semantic or panoptic segmentation. Mask2Former frames each of those tasks as a mask classification problem. Mask classification groups pixels into N segments, and predicts N masks and their corresponding class label for a given image. We’ll explain how Mask2Former works in this section, and then you can try finetuning SegFormer at the end.
There are three main components to Mask2Former:
A Swin backbone accepts an image and creates a low-resolution image feature map from 3 consecutive 3x3 convolutions. The feature map is passed to a pixel decoder whic


Image segmentation

h gradually upsamples the low-resolution features into high-resolution per-pixel embeddings. The pixel decoder actually generates multi-scale features (contains both low- and high-resolution features) with resolutions 1/32, 1/16, and 1/8th of the original image. Each of these feature maps of differing scales is fed successively to one Transformer decoder layer at a time in order to capture small objects from the high-resolution features. The key to Mask2Former is the masked attention mechanism in the decoder. Unlike cross-attention which can attend to the entire image, masked attention only focuses on a certain area of the image. This is faster and leads to better performance because the local features of an image are enough for the model to learn from. Like


Image segmentation

DETR, Mask2Former also uses learned object queries and combines them with the image features from the pixel decoder to make a set prediction (class label, mask prediction). The decoder hidden states are passed into a linear layer and transformed into logits over the class labels. The cross-entropy loss is calculated between the logits and class label to find the most likely one. The mask predictions are generated by combining the pixel-embeddings with the final decoder hidden states. The sigmoid cross-entropy and dice loss is calculated between the logits and the ground truth mask to find the most likely mask.
Ready to try your hand at object detection? Check out our complete image segmentation guide to learn how to finetune SegFormer and use it for inference


Depth estimation

GLPN, Global-Local Path Network, is a Transformer for depth estimation that combines a SegFormer encoder with a lightweight decoder.
Like ViT, an image is split into a sequence of patches, except these image patches are smaller. This is better for dense prediction tasks like segmentation or depth estimation. The image patches are transformed into patch embeddings (see the image classification section for more details about how patch embeddings are created), which are fed to the encoder. The encoder accepts the patch embeddings, and passes them through several encoder blocks. Each block consists of attention and Mix-FFN layers. The purpose of the latter is to provide positional information. At the end of each encoder block is a patch merging layer for creating hierarchical representations. The features of each group of neighbor


Depth estimation

ing patches are concatenated, and a linear layer is applied to the concatenated features to reduce the number of patches to a resolution of 1/4. This becomes the input to the next encoder block, where this whole process is repeated until you have image features with resolutions of 1/8, 1/16, and 1/32. A lightweight decoder takes the last feature map (1/32 scale) from the encoder and upsamples it to 1/16 scale. From here, the feature is passed into a Selective Feature Fusion (SFF) module, which selects and combines local and global features from an attention map for each feature and then upsamples it to 1/8th. This process is repeated until the decoded features are the same size as the original image. The output is passed through two convolution layers and then a sigmoid activation is applied to predict the depth of each pixel.


Natural language processing

The Transformer was initially designed for machine translation, and since then, it has practically become the default architecture for solving all NLP tasks. Some tasks lend themselves to the Transformer’s encoder structure, while others are better suited for the decoder. Still, other tasks make use of both the Transformer’s encoder-decoder structure.


Text classification

BERT is an encoder-only model and is the first model to effectively implement deep bidirectionality to learn richer representations of the text by attending to words on both sides.
BERT uses WordPiece tokenization to generate a token embedding of the text. To tell the difference between a single sentence and a pair of sentences, a special [SEP] token is added to differentiate them. A special [CLS] token is added to the beginning of every sequence of text. The final output with the [CLS] token is used as the input to the classification head for classification tasks. BERT also adds a segment embedding to denote whether a token belongs to the first or second sentence in a pair of sentences. BERT is pretrained with two objective


Text classification

s: masked language modeling and next-sentence prediction. In masked language modeling, some percentage of the input tokens are randomly masked, and the model needs to predict these. This solves the issue of bidirectionality, where the model could cheat and see all the words and “predict” the next word. The final hidden states of the predicted mask tokens are passed to a feedforward network with a softmax over the vocabulary to predict the masked word. The second pretraining object is next-sentence prediction. The model must predict whether sentence B follows sentence A. Half of the time sentence B is the next sentence, and the other half of the time, sentence B is a random sentence. The prediction, whether it is the next sen


Text classification

tence or not, is passed to a feedforward network with a softmax over the two classes (IsNext and NotNext). The input embeddings are passed through multiple encoder layers to output some final hidden states.
To use the pretrained model for text classification, add a sequence classification head on top of the base BERT model. The sequence classification head is a linear layer that accepts the final hidden states and performs a linear transformation to convert them into logits. The cross-entropy loss is calculated between the logits and target to find the most likely label.
Ready to try your hand at text classification? Check out our complete text classification guide to learn how to finetune DistilBERT and use it for inference


Token classification

To use BERT for token classification tasks like named entity recognition (NER), add a token classification head on top of the base BERT model. The token classification head is a linear layer that accepts the final hidden states and performs a linear transformation to convert them into logits. The cross-entropy loss is calculated between the logits and each token to find the most likely label.
Ready to try your hand at token classification? Check out our complete token classification guide to learn how to finetune DistilBERT and use it for inference!


Question answering

To use BERT for question answering, add a span classification head on top of the base BERT model. This linear layer accepts the final hidden states and performs a linear transformation to compute the span start and end logits corresponding to the answer. The cross-entropy loss is calculated between the logits and the label position to find the most likely span of text corresponding to the answer.
Ready to try your hand at question answering? Check out our complete question answering guide to learn how to finetune DistilBERT and use it for inference!
💡 Notice how easy it is to use BERT for different tasks once it’s been pretrained. You only need to add a specific head to the pretrained model to manipulate the hidden states into your desired output!


Text generation

GPT-2 is a decoder-only model pretrained on a large amount of text. It can generate convincing (though not always true!) text given a prompt and complete other NLP tasks like question answering despite not being explicitly trained to.
GPT-2 uses byte pair encoding (BPE) to tokenize words and generate a token embedding. Positional encodings are added to the token embeddings to indicate the position of each token in the sequence. The input embeddings are passed through multiple decoder blocks to output some final hidden state. Within each decoder block, GPT-2 uses a masked self-attention layer which means GPT-2 can’t attend to future tokens. It is only allowed to attend to tokens on the left. This is different from BERT’s mask token because, in masked self-attention, an attention mask is used to set the score to 0 for


Text generation

 future tokens. The output from the decoder is passed to a language modeling head, which performs a linear transformation to convert the hidden states into logits. The label is the next token in the sequence, which are created by shifting the logits to the right by one. The cross-entropy loss is calculated between the shifted logits and the labels to output the next most likely token.
GPT-2’s pretraining objective is based entirely on causal language modeling, predicting the next word in a sequence. This makes GPT-2 especially good at tasks that involve generating text.
Ready to try your hand at text generation? Check out our complete causal language modeling guide to learn how to finetune DistilGPT-2 and use it for inference!
For more information about text generation, check out the text generation strategies guide


Summarization

Encoder-decoder models like BART and T5 are designed for the sequence-to-sequence pattern of a summarization task. We’ll explain how BART works in this section, and then you can try finetuning T5 at the end.
BART’s encoder architecture is very similar to BERT and accepts a token and positional embedding of the text. BART is pretrained by corrupting the input and then reconstructing it with the decoder. Unlike other encoders with specific corruption strategies, BART can apply any type of corruption. The text infilling corruption strategy works the best though. In text infilling, a number of text spans are replaced with a single mask token. This is important because the model has to predict the masked tokens, and it teaches the model to predict the number of missing tokens. The input embeddings and masked spans are passed through the


Summarization

encoder to output some final hidden states, but unlike BERT, BART doesn’t add a final feedforward network at the end to predict a word. The encoder’s output is passed to the decoder, which must predict the masked tokens and any uncorrupted tokens from the encoder’s output. This gives additional context to help the decoder restore the original text. The output from the decoder is passed to a language modeling head, which performs a linear transformation to convert the hidden states into logits. The cross-entropy loss is calculated between the logits and the label, which is just the token shifted to the right.
Ready to try your hand at summarization? Check out our complete summarization guide to learn how to finetune T5 and use it for inference!
For more information about text generation, check out the text generation strategies guide


Translation

Translation is another example of a sequence-to-sequence task, which means you can use an encoder-decoder model like BART or T5 to do it. We’ll explain how BART works in this section, and then you can try finetuning T5 at the end.
BART adapts to translation by adding a separate randomly initialized encoder to map a source language to an input that can be decoded into the target language. This new encoder’s embeddings are passed to the pretrained encoder instead of the original word embeddings. The source encoder is trained by updating the source encoder, position


Translation

al embeddings, and input embeddings with the cross-entropy loss from the model output. The model parameters are frozen in this first step, and all the model parameters are trained together in the second step.
BART has since been followed up by a multilingual version, mBART, intended for translation and pretrained on many different languages.
Ready to try your hand at translation? Check out our complete translation guide to learn how to finetune T5 and use it for inference!
For more information about text generation, check out the text generation strategies guide!


What 🤗 Transformers can do

🤗 Transformers is a library of pretrained state-of-the-art models for natural language processing (NLP), computer vision, and audio and speech processing tasks. Not only does the library contain Transformer models, but it also has non-Transformer models like modern convolutional networks for computer vision tasks. If you look at some of the most popular consumer products today, like smartphones, apps, and televisions, odds are that some kind of deep learning technology is behind it. Want to remove a background object from a picture taken by your smartphone? This is an example of a panoptic segmentation task (don’t worry if you don’t know what this means yet, we’ll describe it in the following sections!).
This page provides an overview of the different speech and audio, computer vision, and NLP tasks that can be solved with the 🤗 Transformers library in just three lines of code!


Audio

Audio and speech processing tasks are a little different from the other modalities mainly because audio as an input is a continuous signal. Unlike text, a raw audio waveform can’t be neatly split into discrete chunks the way a sentence can be divided into words. To get around this, the raw audio signal is typically sampled at regular intervals. If you take more samples within an interval, the sampling rate is higher, and the audio more closely resembles the original audio source.
Previous approaches preprocessed the audio to extract useful features from it. It is now more common to start audio and speech processing tasks by directly feeding the raw audio waveform to a feature encoder to extract an audio representation. This simplifies the preprocessing step and allows the model to learn the most essential features.


Audio classification

Audio classification is a task that labels audio data from a predefined set of classes. It is a broad category with many specific applications, some of which include:
acoustic scene classification: label audio with a scene label (“office”, “beach”, “stadium”) acoustic event detection: label audio with a sound event label (“car horn”, “whale calling”, “glass breaking”) tagging: label audio containing multiple sounds (birdsongs, speaker identification in a meeting) music classification: label music with a genre la


Audio classification

bel (“metal”, “hip-hop”, “country”)
  Copied >>> from transformers import pipeline
>>> classifier = pipeline(task="audio-classification", model="superb/hubert-base-superb-er")
>>> preds = classifier("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac")
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> preds
[{'score': 0.4532, 'label': 'hap'},
 {'score': 0.3622, 'label': 'sad'},
 {'score': 0.0943, 'label': 'neu'},
 {'score': 0.0903, 'label': 'ang'}]


Automatic speech recognition

Automatic speech recognition (ASR) transcribes speech into text. It is one of the most common audio tasks due partly to speech being such a natural form of human communication. Today, ASR systems are embedded in “smart” technology products like speakers, phones, and cars. We can ask our virtual assistants to play music, set reminders, and tell us the weather.
But one of the key challenges Transformer architectures have helped with is in low-resource languages. By pretraining on large amounts of speech data, finetuning


Automatic speech recognition

the model on only one hour of labeled speech data in a low-resource language can still produce high-quality results compared to previous ASR systems trained on 100x more labeled data.
  Copied >>> from transformers import pipeline
>>> transcriber = pipeline(task="automatic-speech-recognition", model="openai/whisper-small")
>>> transcriber("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac")
{'text': ' I have a dream that one day this nation will rise up and live out the true meaning of its creed.'}


Computer vision

One of the first and earliest successful computer vision tasks was recognizing images of zip code numbers using a convolutional neural network (CNN). An image is composed of pixels, and each pixel has a numerical value. This makes it easy to represent an image as a matrix of pixel values. Each particular combination of pixel values describes the colors of an image.
Two general ways computer vision tasks can be solved are:
Use convolutions to learn the hierarchical features of an image from low-level features to high-level abstract things. Split an image into patches and use a Transformer to gradually learn how each image patch is related to each other to form an image. Unlike the bottom-up approach favored by a CNN, this is kind of like starting out with a blurry image and then gradually bringing it into focus.


Image classification

Image classification labels an entire image from a predefined set of classes. Like most classification tasks, there are many practical use cases for image classification, some of which include:
healthcare: label medical images to detect disease or monitor patient health environment: label satellite images to monitor deforestation, inform wildland management or detect wildfires agriculture: label images of crops to monitor plant health or satellite images for land use monitoring ecology: label images of animal or plant species to monitor wildlife populations or track endangered species
  Copied >>> from transformers imp


Image classification

ort pipeline
>>> classifier = pipeline(task="image-classification")
>>> preds = classifier(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> print(*preds, sep="\n")
{'score': 0.4335, 'label': 'lynx, catamount'}
{'score': 0.0348, 'label': 'cougar, puma, catamount, mountain lion, painter, panther, Felis concolor'}
{'score': 0.0324, 'label': 'snow leopard, ounce, Panthera uncia'}
{'score': 0.0239, 'label': 'Egyptian cat'}
{'score': 0.0229, 'label': 'tiger cat'}


Object detection

Unlike image classification, object detection identifies multiple objects within an image and the objects’ positions in an image (defined by the bounding box). Some example applications of object detection include:
self-driving vehicles: detect everyday traffic objects such as other vehicles, pedestrians, and traffic lights remote sensing: disaster monitoring, urban planning, and weather forecasting defect detection: detect cracks or structural damage in buildings, and manufacturing defects
  Copied >>> from transformers import pipeline
>>> detector = pipeline(task="object-detection")
>>> preds = detector(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"], "box": pred["box"]} for pred in preds]
>>> preds
[{'score': 0.9865,
  'label': 'cat',
  'box': {'xmin': 178, 'ymin': 154, 'xmax': 882, 'ymax': 598}}]


Image segmentation

Image segmentation is a pixel-level task that assigns every pixel in an image to a class. It differs from object detection, which uses bounding boxes to label and predict objects in an image because segmentation is more granular. Segmentation can detect objects at a pixel-level. There are several types of image segmentation:
instance segmentation: in addition to labeling the class of an object, it also labels each distinct instance of an object (“dog-1”, “dog-2”) panoptic segmentation: a combination of semantic and instance segmentation; it labels each pixel with a semantic class and each distinct instance of an object
Segmentation tasks are helpful in self-driving vehicles to create a pixel-level map of the world around them so they can navigate safely around pedestrians and


Image segmentation

other vehicles. It is also useful for medical imaging, where the task’s finer granularity can help identify abnormal cells or organ features. Image segmentation can also be used in ecommerce to virtually try on clothes or create augmented reality experiences by overlaying objects in the real world through your camera.
  Copied >>> from transformers import pipeline
>>> segmenter = pipeline(task="image-segmentation")
>>> preds = segmenter(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> print(*preds, sep="\n")
{'score': 0.9879, 'label': 'LABEL_184'}
{'score': 0.9973, 'label': 'snow'}
{'score': 0.9972, 'label': 'cat'}


Depth estimation

Depth estimation predicts the distance of each pixel in an image from the camera. This computer vision task is especially important for scene understanding and reconstruction. For example, in self-driving cars, vehicles need to understand how far objects like pedestrians, traffic signs, and other vehicles are to avoid obstacles and collisions. Depth information is also helpful for constructing 3D representations from 2D images and can be used to create high-quality 3D representations of biological structures or buildings.
There are two approaches to depth estimation:
stereo: depths are estimated by comparing two images of the same image from slightly different angles monocular: depths are estimated from a single image
  Copied >>> from transformers import pipeline
>>> depth_estimator = pipeline(task="depth-estimation")
>>> preds = depth_estimator(
...     "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
... )


Natural language processing

NLP tasks are among the most common types of tasks because text is such a natural way for us to communicate. To get text into a format recognized by a model, it needs to be tokenized. This means dividing a sequence of text into separate words or subwords (tokens) and then converting these tokens into numbers. As a result, you can represent a sequence of text as a sequence of numbers, and once you have a sequence of numbers, it can be input into a model to solve all sorts of NLP tasks!


Text classification

Like classification tasks in any modality, text classification labels a sequence of text (it can be sentence-level, a paragraph, or a document) from a predefined set of classes. There are many practical applications for text classification, some of which include:
sentiment analysis: label text according to some polarity like positive or negative which can inform and support decision-making in fields like politics, finance, and marketing content classification: label text according to some topic to help organize and filter information in news and social media feeds (weather, sports, finance, etc.)
  Copied >>> from transformers import pipeline
>>> classifier = pipeline(task="sentiment-analysis")
>>> preds = classifier("Hugging Face is the best thing since sliced bread!")
>>> preds = [{"score": round(pred["score"], 4), "label": pred["label"]} for pred in preds]
>>> preds
[{'score': 0.9991, 'label': 'POSITIVE'}]


Token classification

In any NLP task, text is preprocessed by separating the sequence of text into individual words or subwords. These are known as tokens. Token classification assigns each token a label from a predefined set of classes.
Two common types of token classification are:
named entity recognition (NER): label a token according to an entity category like organization, person, location or date. NER is especially popular in biomedical settings, where it can label genes, proteins, and drug names. part-of-speech tagging (POS): label a token according to its part-of-speech like noun, verb, or adjective. POS is useful for helping translation systems understand how two identical words are grammatically different (bank as a noun versus bank as a verb).
  Copied >>> from transformers import pipeline
>>> classifier = pipeline(task="ner")
>>> preds = classifier("Hugging Face is a French company based in New York City.")
>>> preds = [


Token classification

...     {
...         "entity": pred["entity"],
...         "score": round(pred["score"], 4),
...         "index": pred["index"],
...         "word": pred["word"],
...         "start": pred["start"],
...         "end": pred["end"],
...     }
...     for pred in preds
... ]
>>> print(*preds, sep="\n")
{'entity': 'I-ORG', 'score': 0.9968, 'index': 1, 'word': 'Hu', 'start': 0, 'end': 2}
{'entity': 'I-ORG', 'score': 0.9293, 'index': 2, 'word': '##gging', 'start': 2, 'end': 7}
{'entity': 'I-ORG', 'score': 0.9763, 'index': 3, 'word': 'Face', 'start': 8, 'end': 12}
{'entity': 'I-MISC', 'score': 0.9983, 'index': 6, 'word': 'French', 'start': 18, 'end': 24}
{'entity': 'I-LOC', 'score': 0.999, 'index': 10, 'word': 'New', 'start': 42, 'end': 45}
{'entity': 'I-LOC', 'score': 0.9987, 'index': 11, 'word': 'York', 'start': 46, 'end': 50}
{'entity': 'I-LOC', 'score': 0.9992, 'index': 12, 'word': 'City', 'start': 51, 'end': 55}


Question answering

Question answering is another token-level task that returns an answer to a question, sometimes with context (open-domain) and other times without context (closed-domain). This task happens whenever we ask a virtual assistant something like whether a restaurant is open. It can also provide customer or technical support and help search engines retrieve the relevant information you’re asking for.
There are two common types of question answering:
extractive: given a question and some context, the answer is a span of text from the context the model must extract abstractive: given a question and some context, the answer


Question answering

 is generated from the context; this approach is handled by the Text2TextGenerationPipeline instead of the QuestionAnsweringPipeline shown below
  Copied >>> from transformers import pipeline
>>> question_answerer = pipeline(task="question-answering")
>>> preds = question_answerer(
...     question="What is the name of the repository?",
...     context="The name of the repository is huggingface/transformers",
... )
>>> print(
...     f"score: {round(preds['score'], 4)}, start: {preds['start']}, end: {preds['end']}, answer: {preds['answer']}"
... )
score: 0.9327, start: 30, end: 54, answer: huggingface/transformers


Summarization

Summarization creates a shorter version of a text from a longer one while trying to preserve most of the meaning of the original document. Summarization is a sequence-to-sequence task; it outputs a shorter text sequence than the input. There are a lot of long-form documents that can be summarized to help readers quickly understand the main points. Legislative bills, legal and financial documents, patents, and scientific papers are a few examples of documents that could be summarized to save readers time and serve as a reading aid.
Like question answering, there are two types of summarization:
extractive: identify and extract the most important sentences from the original text abstractive: generate the target summary (which may include new words not in the input document) from the original text; the SummarizationPipeline uses the abstractive approach
  Copied >>> from transformers import pipeline
>>> summarizer = pipeline(task="summarization")
>>> sum


Summarization

marizer(
...     "In this work, we presented the Transformer, the first sequence transduction model based entirely on attention, replacing the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention. For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers. On both WMT 2014 English-to-German and WMT 2014 English-to-French translation tasks, we achieve a new state of the art. In the former task our best model outperforms even all previously reported ensembles."
... )
[{'summary_text': ' The Transformer is the first sequence transduction model based entirely on attention . It replaces the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention . For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers .'}


Translation

Translation converts a sequence of text in one language to another. It is important in helping people from different backgrounds communicate with each other, help translate content to reach wider audiences, and even be a learning tool to help people learn a new language. Along with summarization, translation is a sequence-to-sequence task, meaning the model receives an input sequence and returns a target output sequence.
In the early days, translation models were mostly monolingual, but recently, there has been increasing interest in multilingual models that can translate between many pairs of languages.
  Copied >>> from transformers import pipeline
>>> text = "translate English to French: Hugging Face is a community-based open-source platform for machine learning."
>>> translator = pipeline(task="translation", model="t5-small")
>>> translator(text)
[{'translation_text': "Hugging Face est une tribune communautaire de l'apprentissage des machines."}]


Language modeling

Language modeling is a task that predicts a word in a sequence of text. It has become a very popular NLP task because a pretrained language model can be finetuned for many other downstream tasks. Lately, there has been a lot of interest in large language models (LLMs) which demonstrate zero- or few-shot learning. This means the model can solve tasks it wasn’t explicitly trained to do! Language models can be used to generate fluent and convincing text, though you need to be careful since the text may not always be accurate.
There are two types of language modeling:
causal: the model’s objective is to predict the next token in a sequence, and future tokens are masked   Copied >>> from transformers import pipeline
>>> prompt = "Hugging Face is a community-based open-source platform for machin


Language modeling

e learning."
>>> generator = pipeline(task="text-generation")
>>> generator(prompt)  # doctest: +SKIP masked: the model’s objective is to predict a masked token in a sequence with full access to the tokens in the sequence   Copied >>> text = "Hugging Face is a community-based open-source <mask> for machine learning."
>>> fill_mask = pipeline(task="fill-mask")
>>> preds = fill_mask(text, top_k=1)
>>> preds = [
...     {
...         "score": round(pred["score"], 4),
...         "token": pred["token"],
...         "token_str": pred["token_str"],
...         "sequence": pred["sequence"],
...     }
...     for pred in preds
... ]
>>> preds
[{'score': 0.2236,
  'token': 1761,
  'token_str': ' platform',
  'sequence': 'Hugging Face is a community-based open-source platform for machine learning.'}


Multimodal

Multimodal tasks require a model to process multiple data modalities (text, image, audio, video) to solve a particular problem. Image captioning is an example of a multimodal task where the model takes an image as input and outputs a sequence of text describing the image or some properties of the image.
Although multimodal models work with different data types or modalities, internally, the preprocessing steps help the model convert all the data types into embeddings (vectors or list of numbers that holds meaningful information about the data). For a task like image captioning, the model learns relationships between image embeddings and text embeddings.


Document question answering

Document question answering is a task that answers natural language questions from a document. Unlike a token-level question answering task which takes text as input, document question answering takes an image of a document as input along with a question about the document and returns an answer. Document question answering can be used to parse structured documents and extract key information from it. In the example below, the total amount and change due can be extracted from a receipt.
  Copied >>> from transformers import pipeline
>>> from PIL import Image
>>> import requests
>>> url = "https://datasets-server.huggingface.co/assets/hf-internal-testing/ex


Document question answering

ample-documents/--/hf-internal-testing--example-documents/test/2/image/image.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> doc_question_answerer = pipeline("document-question-answering", model="magorshunov/layoutlm-invoices")
>>> preds = doc_question_answerer(
...     question="What is the total amount?",
...     image=image,
... )
>>> preds
[{'score': 0.8531, 'answer': '17,000', 'start': 4, 'end': 4}]
Hopefully, this page has given you some more background information about all the types of tasks in each modality and the practical importance of each one. In the next section, you’ll learn how 🤗 Transformers work to solve these tasks.


BetterTransformer for faster inference

We have recently integrated BetterTransformer for faster inference on CPU for text, image and audio models. Check the documentation about this integration here for more details.


PyTorch JIT-mode (TorchScript)

TorchScript is a way to create serializable and optimizable models from PyTorch code. Any TorchScript program can be saved from a Python process and loaded in a process where there is no Python dependency.
Comparing to default eager mode, jit mode in PyTorch normally yields better performance for model inference from optimization methodologies like operator fusion.
For a gentle introduction to TorchScript, see the Introduction to PyTorch TorchScript tutorial.


IPEX Graph Optimization with JIT-mode

Intel® Extension for PyTorch provides further optimizations in jit mode for Transformers series models. It is highly recommended for users to take advantage of Intel® Extension for PyTorch with jit mode. Some frequently used operator patterns from Transformers models are already supported in Intel® Extension for PyTorch with jit mode fusions. Those fusion patterns like Multi-head-attention fusion, Concat Linear, Linear+Add, Linear+Gelu, Add+LayerNorm fusion and etc. are enabled and perform well. The benefit of the fusion is delivered to users in a transparent fashion. According to the analysis, ~70% of most popular NLP tasks in question-answering, text-classification, and token-classification can get performance benefits with these fusion patterns for both Float32 precision and BFloat16 Mixed precision.
Check more detailed information for IPEX Graph Optimization.
 IPEX installation:
IPEX release is following PyTorch, check the approaches for IPEX installation.


Usage of JIT-mode

To enable JIT-mode in Trainer for evaluaion or prediction, users should add jit_mode_eval in Trainer command arguments.
for PyTorch >= 1.14.0. JIT-mode could benefit any models for prediction and evaluaion since dict input is supported in jit.trace for PyTorch < 1.14.0. JIT-mode could benefit models whose forward parameter order matches the tuple input order in jit.trace, like question-answering model
In the case where the forward parameter order does not match the tuple input order in jit.trace, like text-classification models, jit.trace will fail and we are capturing this with the exception here to mak


Usage of JIT-mode

e it fallback. Logging is used to notify users.
Take an example of the use cases on Transformers question-answering
Inference using jit mode on CPU: python run_qa.py \
--model_name_or_path csarron/bert-base-uncased-squad-v1 \
--dataset_name squad \
--do_eval \
--max_seq_length 384 \
--doc_stride 128 \
--output_dir /tmp/ \
--no_cuda \
--jit_mode_eval  Inference with IPEX using jit mode on CPU: python run_qa.py \
--model_name_or_path csarron/bert-base-uncased-squad-v1 \
--dataset_name squad \
--do_eval \
--max_seq_length 384 \
--doc_stride 128 \
--output_dir /tmp/ \
--no_cuda \
--use_ipex \
--jit_mode_eval


Export to ONNX

Deploying 🤗 Transformers models in production environments often requires, or can benefit from exporting the models into
a serialized format that can be loaded and executed on specialized runtimes and hardware.
🤗 Optimum is an extension of Transformers that enables exporting models from PyTorch or TensorFlow to serialized formats
such as ONNX and TFLite through its exporters module. 🤗 Optimum also provides a set of performance optimization tools to train
and run models on targeted hardware with maximum efficiency.
This guide demonstrates how you can export 🤗 Transformers models to ONNX with 🤗 Optimum, for the guide on exporting models to TFLite,
please refer to the Export to TFLite page.


Export to ONNX

ONNX (Open Neural Network eXchange) is an open standard that defines a common set of operators and a
common file format to represent deep learning models in a wide variety of frameworks, including PyTorch and
TensorFlow. When a model is exported to the ONNX format, these operators are used to
construct a computational graph (often called an intermediate representation) which
represents the flow of data through the neural network.
By exposing a graph with standardized operators and data types, ONNX makes it easy to
switch between frameworks. For example, a model trained in PyTorch can be exported to
ONNX format and then imported in TensorFlow (and vice versa).
Once exported to ONNX format, a model can be:
optimized for inference via techniques su


Export to ONNX

ch as graph optimization and quantization. run with ONNX Runtime via ORTModelForXXX classes,
which follow the same AutoModel API as the one you are used to in 🤗 Transformers. run with optimized inference pipelines,
which has the same API as the pipeline() function in 🤗 Transformers.
🤗 Optimum provides support for the ONNX export by leveraging configuration objects. These configuration objects come
ready-made for a number of model architectures, and are designed to be easily extendable to other architectures.
For the list of ready-made configurations, please refer to 🤗 Optimum documentation.
There are two ways to export a 🤗 Transformers model to ONNX, here we show both:
export with 🤗 Optimum via CLI. export with 🤗 Optimum with optimum.onnxruntime


Exporting a 🤗 Transformers model to ONNX with CLI

To export a 🤗 Transformers model to ONNX, first install an extra dependency:
  Copied pip install optimum[exporters]
To check out all available arguments, refer to the 🤗 Optimum docs,
or view help in command line:
  Copied optimum-cli export onnx --help
To export a model’s checkpoint from the 🤗 Hub, for example, distilbert-base-uncased-distilled-squad, run the following command:
  Copied optimum-cli export onnx --model distilbert-base-uncased-distilled-squad distilbert_base_uncased_squad_onnx/
You should see the logs indicating progress and showing where the resulting model.onnx is saved, like this:
  Copied Validating ONNX model distilbert_base_uncased_squad_onnx/model.onnx...
-[✓] ONNX model output names match reference model (start_logits, end_logits)
- Validating ONNX Model output "start_logits":
-[✓] (2, 16) matches (2, 16)
-[✓] all values close (a


tol: 0.0001)
- Validating ONNX Model output "end_logits":
-[✓] (2, 16) matches (2, 16)
-[✓] all values close (atol: 0.0001)
The ONNX export succeeded and the exported model was saved at: distilbert_base_uncased_squad_onnx
The example above illustrates exporting a checkpoint from 🤗 Hub. When exporting a local model, first make sure that you
saved both the model’s weights and tokenizer files in the same directory (local_path). When using CLI, pass the
local_path to the model argument instead of the checkpoint name on 🤗 Hub and provide the --task argument.
You can review the list of supported tasks in the 🤗 Optimum documentation.
If task argument is not provided, it will default to the model architecture without any task specific head.
  Copied optimum-cli export onnx --model local_path --task question-answering distilbert_base_uncased_squad_onnx/
The resu


lting model.onnx file can then be run on one of the many
accelerators that support the ONNX
standard. For example, we can load and run the model with ONNX
Runtime as follows:
  Copied >>> from transformers import AutoTokenizer
>>> from optimum.onnxruntime import ORTModelForQuestionAnswering
>>> tokenizer = AutoTokenizer.from_pretrained("distilbert_base_uncased_squad_onnx")
>>> model = ORTModelForQuestionAnswering.from_pretrained("distilbert_base_uncased_squad_onnx")
>>> inputs = tokenizer("What am I using?", "Using DistilBERT with ONNX Runtime!", return_tensors="pt")
>>> outputs = model(**inputs)
The process is identical for TensorFlow checkpoints on the Hub. For instance, here’s how you would
export a pure TensorFlow checkpoint from the Keras organization:
  Copied optimum-cli export onnx --model keras-io/transformers-qa distilbert_base_cased_squad_onn