ngth models should be evaluated with a sliding-window strategy. This involves repeatedly
sliding the context window so that the model has more context when making each prediction.
This is a closer approximation to the true decomposition of the sequence probability and will typically yield a more
favorable score. The downside is that it requires a separate forward pass for each token in the corpus. A good
practical compromise is to employ a strided sliding window, moving the context by larger strides rather than sliding by
1 token a time. This allows computation to proceed much faster while still giving the model a large context to make
predictions at each step


Example: Calculating perplexity with GPT-2 in 🤗 Transformers

Let’s demonstrate this process with GPT-2.
  Copied from transformers import GPT2LMHeadModel, GPT2TokenizerFast
device = "cuda"
model_id = "gpt2-large"
model = GPT2LMHeadModel.from_pretrained(model_id).to(device)
tokenizer = GPT2TokenizerFast.from_pretrained(model_id)
We’ll load in the WikiText-2 dataset and evaluate the perplexity using a few different sliding-window strategies. Since
this dataset is small and we’re just doing one forward pass over the set, we can just load and encode the entire
dataset in memory.
  Copied from datasets import load_dataset
test = load_dataset("wikitext", "wikitext-2-raw-v1", split="test")
encodings = tokenizer("\n\n".join(test["text"]), return_tensors="pt")
With 🤗 Transformers, we can simply pass the input_ids as the labels to


our model, and the average negative
log-likelihood for each token is returned as the loss. With our sliding window approach, however, there is overlap in
the tokens we pass to the model at each iteration. We don’t want the log-likelihood for the tokens we’re just treating
as context to be included in our loss, so we can set these targets to -100 so that they are ignored. The following
is an example of how we could do this with a stride of 512. This means that the model will have at least 512 tokens
for context when calculating the conditional likelihood of any one token (provided there are 512 preceding tokens
available to condition on).
  Copied import torch
from tqdm import tqdm
max_length = model.config.n_positions
stride = 512
seq_len = encodings.input_ids.s


ize(1)
nlls = []
prev_end_loc = 0
for begin_loc in tqdm(range(0, seq_len, stride)):
    end_loc = min(begin_loc + max_length, seq_len)
    trg_len = end_loc - prev_end_loc  # may be different from stride on last loop
    input_ids = encodings.input_ids[:, begin_loc:end_loc].to(device)
    target_ids = input_ids.clone()
    target_ids[:, :-trg_len] = -100
    with torch.no_grad():
        outputs = model(input_ids, labels=target_ids)
        # loss is calculated using CrossEntropyLoss which averages over valid labels
        # N.B. the model only calculates loss over trg_len - 1 labels, because it internally shifts the labels
        # to the left by 1.
        neg_log_likelihood = outputs.loss
    nlls.append(neg_log_likelihood)
    prev_end_loc = end_loc
    if


 end_loc == seq_len:
        break
ppl = torch.exp(torch.stack(nlls).mean())
Running this with the stride length equal to the max input length is equivalent to the suboptimal, non-sliding-window
strategy we discussed above. The smaller the stride, the more context the model will have in making each prediction,
and the better the reported perplexity will typically be.
When we run the above with stride = 1024, i.e. no overlap, the resulting PPL is 19.44, which is about the same
as the 19.93 reported in the GPT-2 paper. By using stride = 512 and thereby employing our striding window
strategy, this jumps down to 16.45. This is not only a more favorable score, but is calculated in a way that is
closer to the true autoregressive decomposition of a sequence likelihood.


Training on TPUs

Note: Most of the strategies introduced in the single GPU section (such as mixed precision training or gradient accumulation) and multi-GPU section are generic and apply to training models in general so make sure to have a look at it before diving into this section.
This document will be completed soon with information on how to train on TPUs.


Efficient Inference on a Single GPU

In addition to this guide, relevant information can be found as well in the guide for training on a single GPU and the guide for inference on CPUs.


Flash Attention 2

Note that this feature is experimental and might considerably change in future versions. For instance, the Flash Attention 2 API might migrate to BetterTransformer API in the near future.
Flash Attention 2 can considerably speed up transformer-based models’ training and inference speed. Flash Attention 2 has been introduced in the official Flash Attention repository by Tri Dao et al. The scientific paper on Flash Attention can be found here.
Make sure to follow the installation guide on the repository mentioned above to properly install Flash Attention 2. Once that package is installed, you can benefit from this feature.
W


Flash Attention 2

e natively support Flash Attention 2 for the following models:
Llama Mistral Falcon
You can request to add Flash Attention 2 support for more models by opening an issue on GitHub, and even open a Pull Request to integrate the changes. The supported models can be used for inference and training, including training with padding tokens - which is currently not supported for BetterTransformer API below.
Flash Attention 2 can only be used when the models’ dtype is  fp16 or bf16 and runs only on NVIDIA-GPU devices. Make sure to cast your model to the appropriate dtype and load them on a supported device before using that feature


Quick usage

To enable Flash Attention 2 in your model, add use_flash_attention_2 in the from_pretrained arguments:
  Copied import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaForCausalLM
model_id = "tiiuae/falcon-7b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    use_flash_attention_2=True,
)
And use it for generation or fine-tuning.


Expected speedups

You can benefit from considerable speedups for fine-tuning and inference, especially for long sequences. However, since Flash Attention does not support computing attention scores with padding tokens under the hood, we must manually pad / unpad the attention scores for batched inference when the sequence contains padding tokens. This leads to a significant slowdown for batched generations with padding tokens.
To overcome this, one should use Flash Attention without padding tokens in the sequence for training (e.g., by packing a dataset, i.e., concatenating sequences until reaching the maximum sequence length. An example is provided here.
Below is the expected speedup you can get for a simple forward pass on tiiuae/falcon-7b with a sequence length of 4096 and various batch sizes, without padding tokens:
Below is the expected speedup you can get for a simple forward pas


Expected speedups

s on meta-llama/Llama-7b-hf with a sequence length of 4096 and various batch sizes, without padding tokens:
For sequences with padding tokens (training with padding tokens or generating with padding tokens), we need to unpad / pad the input sequences to compute correctly the attention scores. For relatively small sequence length, on pure forward pass, this creates an overhead leading to a small speedup (below 30% of the input has been filled with padding tokens).
But for large sequence length you can benefit from interesting speedup for pure inference (also training)
Note that Flash Attention makes the attention computation more memory efficient, meaning you can train with much larger sequence lengths without facing CUDA OOM issues. It can lead up to memory reduction up to 20 for large sequence length. Check out the official flash attention repository for more details


Advanced usage

You can combine this feature with many exisiting feature for model optimization. Check out few examples below:


Combining Flash Attention 2 and 8-bit models

You can combine this feature together with 8-bit quantization:
  Copied import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaForCausalLM
model_id = "tiiuae/falcon-7b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_8bit=True,
    use_flash_attention_2=True,
)


Combining Flash Attention 2 and 4-bit models

You can combine this feature together with 4-bit quantization:
  Copied import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaForCausalLM
model_id = "tiiuae/falcon-7b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_4bit=True,
    use_flash_attention_2=True,
)


Combining Flash Attention 2 and PEFT

You can combine this feature together with PEFT for training adapters using Flash Attention 2 under the hood:
  Copied import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaForCausalLM
from peft import LoraConfig
model_id = "tiiuae/falcon-7b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_4bit=True,
    use_flash_attention_2=True,
)
lora_config = LoraConfig(
    r=8,
    task_type="CAUSAL_LM"
)
model.add_adapter(lora_config)
... # train your model


BetterTransformer

BetterTransformer converts 🤗 Transformers models to use the PyTorch-native fastpath execution, which calls optimized kernels like Flash Attention under the hood.
BetterTransformer is also supported for faster inference on single and multi-GPU for text, image, and audio models.
Flash Attention can only be used for models using fp16 or bf16 dtype. Make sure to cast your model to the appropriate dtype before using BetterTransformer.


Encoder models

PyTorch-native nn.MultiHeadAttention attention fastpath, called BetterTransformer, can be used with Transformers through the integration in the 🤗 Optimum library.
PyTorch’s attention fastpath allows to speed up inference through kernel fusions and the use of nested tensors. Detailed benchmarks can be found in this blog post.
After installing the optimum package, to use Better Transformer during inference, the relevant internal modules are replaced by calling to_bettertransformer():
  Copied model = model.to_bettertransformer()
The method reverse_bettertransformer() allows to go back to the original modeling, which should be used before saving the model in order to use the canonical transformers modeling:
  Copied model = model.reverse_bettertransformer()
model.save_pretrained("saved_model")
Have a look at this blog post to learn more about what is possible to do with BetterTransformer API for encoder models.


Decoder models

For text models, especially decoder-based models (GPT, T5, Llama, etc.), the BetterTransformer API converts all attention operations to use the torch.nn.functional.scaled_dot_product_attention operator (SDPA) that is only available in PyTorch 2.0 and onwards.
To convert a model to BetterTransformer:
  Copied from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m")
# convert the model to BetterTransformer
model.to_bettertransformer()
# Use it for training or inference
SDPA can also call Flash Attention kernels under the hood. To enable Flash Attention or to check that it is available in a given setting (hardware, problem size), use torch.backends.cuda.sdp_kernel as a context manager:
  Copied import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m")
model = AutoModelForCausalLM.from_pretrained(


Decoder models

"facebook/opt-350m", torch_dtype=torch.float16).to("cuda")
# convert the model to BetterTransformer
model.to_bettertransformer()
input_text = "Hello my dog is cute and"
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")
+ with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):
    outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
If you see a bug with a traceback saying
  Copied RuntimeError: No available kernel.  Aborting execution.
try using the PyTorch nightly version, which may have a broader coverage for Flash Attention:
  Copied pip3 install -U --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu118
Or make sure your model is correctly casted in float16 or bfloat16
Have a look at this detailed blogpost to read more about what is possible to do with BetterTransformer + SDPA API.


bitsandbytes integration for FP4 mixed-precision inference

You can install bitsandbytes and benefit from easy model compression on GPUs. Using FP4 quantization you can expect to reduce up to 8x the model size compared to its native full precision version. Check out below how to get started.
Note that this feature can also be used in a multi GPU setup.


Requirements

Latest bitsandbytes library
pip install bitsandbytes>=0.39.0 Install latest accelerate from source
pip install git+https://github.com/huggingface/accelerate.git Install latest transformers from source
pip install git+https://github.com/huggingface/transformers.git


Running FP4 models - single GPU setup - Quickstart

You can quickly run a FP4 model on a single GPU by running the following code:
  Copied from transformers import AutoModelForCausalLM
model_name = "bigscience/bloom-2b5"
model_4bit = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", load_in_4bit=True)
Note that device_map is optional but setting device_map = 'auto' is prefered for inference as it will dispatch efficiently the model on the available ressources.


Running FP4 models - multi GPU setup

The way to load your mixed 4-bit model in multiple GPUs is as follows (same command as single GPU setup):
  Copied model_name = "bigscience/bloom-2b5"
model_4bit = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", load_in_4bit=True)
But you can control the GPU RAM you want to allocate on each GPU using accelerate. Use the max_memory argument as follows:
  Copied max_memory_mapping = {0: "600MB", 1: "1GB"}
model_name = "bigscience/bloom-3b"
model_4bit = AutoModelForCausalLM.from_pretrained(
    model_name, device_map="auto", load_in_4bit=True, max_memory=max_memory_mapping
)
In this example, the first GPU will use 600MB of memory and the second 1GB.


bitsandbytes integration for Int8 mixed-precision matrix decomposition

Note that this feature can also be used in a multi GPU setup.
From the paper LLM.int8() : 8-bit Matrix Multiplication for Transformers at Scale, we support Hugging Face integration for all models in the Hub with a few lines of code.
The method reduces nn.Linear size by 2 for float16 and bfloat16 weights and by 4 for float32 weights, with close to no impact to the quality by operating on the outliers in half-precision.
Int8 mixed-precision matrix decomposition works by separating a matrix multiplication into two streams: (1) a systematic feature outlier stream matrix multiplied in fp16 (0.01%), (2)


a regular stream of int8 matrix multiplication (99.9%). With this method, int8 inference with no predictive degradation is possible for very large models.
For more details regarding the method, check out the paper or our blogpost about the integration.
Note, that you would require a GPU to run mixed-8bit models as the kernels have been compiled for GPUs only. Make sure that you have enough GPU memory to store the quarter (or half if your model weights are in half precision) of the model before using this feature.
Below are some notes to help you use this module, or follow the demos on Google colab.


Requirements

If you have bitsandbytes<0.37.0, make sure you run on NVIDIA GPUs that support 8-bit tensor cores (Turing, Ampere or newer architectures - e.g. T4, RTX20s RTX30s, A40-A100). For bitsandbytes>=0.37.0, all GPUs should be supported. Install the correct version of bitsandbytes by running:
pip install bitsandbytes>=0.31.5 Install accelerate pip install accelerate>=0.12.0


Running mixed-Int8 models - single GPU setup

After installing the required libraries, the way to load your mixed 8-bit model is as follows:
  Copied from transformers import AutoModelForCausalLM
model_name = "bigscience/bloom-2b5"
model_8bit = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", load_in_8bit=True)
For text generation, we recommend:
using the model’s generate() method instead of the pipeline() function. Although inference is possible with the pipeline() function, it is not optimized for mixed-8bit models, and will be slower than using the generate() method. Moreover, some sampling strategies are like nucleaus sampling a


re not supported by the pipeline() function for mixed-8bit models. placing all inputs on the same device as the model.
Here is a simple example:
  Copied from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "bigscience/bloom-2b5"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model_8bit = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", load_in_8bit=True)
prompt = "Hello, my llama is cute"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
generated_ids = model.generate(**inputs)
outputs = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)


Running mixed-int8 models - multi GPU setup

The way to load your mixed 8-bit model in multiple GPUs is as follows (same command as single GPU setup):
  Copied model_name = "bigscience/bloom-2b5"
model_8bit = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", load_in_8bit=True)
But you can control the GPU RAM you want to allocate on each GPU using accelerate. Use the max_memory argument as follows:
  Copied max_memory_mapping = {0: "1GB", 1: "2GB"}
model_name = "bigscience/bloom-3b"
model_8bit = AutoModelForCausalLM.from_pretrained(
    model_name, device_map="auto", load_in_8bit=True, max_memory=max_memory_mapping
)
In this example, the first GPU will use 1GB of memory and the second 2GB.


Colab demos

With this method you can infer on models that were not possible to infer on a Google Colab before.
Check out the demo for running T5-11b (42GB in fp32)! Using 8-bit quantization on Google Colab:
Or this demo for BLOOM-3B:


Advanced usage: mixing FP4 (or Int8) and BetterTransformer

You can combine the different methods described above to get the best performance for your model. For example, you can use BetterTransformer with FP4 mixed-precision inference + flash attention:
  Copied import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16
)
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m")
model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m", quantization_config=quantization_config)
input_text = "Hello my dog is cute and"
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")
with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):
    outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))


Glossary

This glossary defines general machine learning and 🤗 Transformers terms to help you better understand the
documentation.


attention mask

The attention mask is an optional argument used when batching sequences together.
This argument indicates to the model which tokens should be attended to, and which should not.
For example, consider these two sequences:
  Copied >>> from transformers import BertTokenizer
>>> tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
>>> sequence_a = "This is a short sequence."
>>> sequence_b = "This is a rather long sequence. It is at least longer than the sequence A."
>>> encoded_sequence_a = tokenizer(sequence_a)["input_ids"]
>>> encoded_sequence_b = tokenizer(sequence_b)["input_ids"]
The encoded versions have different lengths:
  Copied >>> len(encoded_sequence_a), len(enco


attention mask

ded_sequence_b)
(8, 19)
Therefore, we can’t put them together in the same tensor as-is. The first sequence needs to be padded up to the length
of the second one, or the second one needs to be truncated down to the length of the first one.
In the first case, the list of IDs will be extended by the padding indices. We can pass a list to the tokenizer and ask
it to pad like this:
  Copied >>> padded_sequences = tokenizer([sequence_a, sequence_b], padding=True)
We can see that 0s have been added on the right of the first sentence to make it the same length as the second one:
  Copied >>> padded_sequences["input_ids"]
[[101, 1188, 1110, 170, 1603, 4954, 119, 102, 0, 0, 0, 0, 0, 0, 0,


attention mask

 0, 0, 0, 0], [101, 1188, 1110, 170, 1897, 1263, 4954, 119, 1135, 1110, 1120, 1655, 2039, 1190, 1103, 4954, 138, 119, 102]]
This can then be converted into a tensor in PyTorch or TensorFlow. The attention mask is a binary tensor indicating the
position of the padded indices so that the model does not attend to them. For the BertTokenizer, 1 indicates a
value that should be attended to, while 0 indicates a padded value. This attention mask is in the dictionary returned
by the tokenizer under the key “attention_mask”:
  Copied >>> padded_sequences["attention_mask"]
[[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]


backbone

The backbone is the network (embeddings and layers) that outputs the raw hidden states or features. It is usually connected to a head which accepts the features as its input to make a prediction. For example, ViTModel is a backbone without a specific head on top. Other models can also use VitModel as a backbone such as DPT.


causal language modeling

A pretraining task where the model reads the texts in order and has to predict the next word. It’s usually done by
reading the whole sentence but using a mask inside the model to hide the future tokens at a certain timestep.


channel

Color images are made up of some combination of values in three channels - red, green, and blue (RGB) - and grayscale images only have one channel. In 🤗 Transformers, the channel can be the first or last dimension of an image’s tensor: [n_channels, height, width] or [height, width, n_channels].


connectionist temporal classification (CTC)

An algorithm which allows a model to learn without knowing exactly how the input and output are aligned; CTC calculates the distribution of all possible outputs for a given input and chooses the most likely output from it. CTC is commonly used in speech recognition tasks because speech doesn’t always cleanly align with the transcript for a variety of reasons such as a speaker’s different speech rates.


convolution

A type of layer in a neural network where the input matrix is multiplied element-wise by a smaller matrix (kernel or filter) and the values are summed up in a new matrix. This is known as a convolutional operation which is repeated over the entire input matrix. Each operation is applied to a different segment of the input matrix. Convolutional neural networks (CNNs) are commonly used in computer vision.


decoder input IDs

This input is specific to encoder-decoder models, and contains the input IDs that will be fed to the decoder. These
inputs should be used for sequence to sequence tasks, such as translation or summarization, and are usually built in a
way specific to each model.
Most encoder-decoder models (BART, T5) create their decoder_input_ids on their own from the labels. In such models,
passing the labels is the preferred way to handle training.
Please check each model’s docs to see how they handle these input IDs for sequence to sequence training.


decoder models

Also referred to as autoregressive models, decoder models involve a pretraining task (called causal language modeling) where the model reads the texts in order and has to predict the next word. It’s usually done by
reading the whole sentence with a mask to hide future tokens at a certain timestep.


encoder models

Also known as autoencoding models, encoder models take an input (such as text or images) and transform them into a condensed numerical representation called an embedding. Oftentimes, encoder models are pretrained using techniques like masked language modeling, which masks parts of the input sequence and forces the model to create more meaningful representations.


feature extraction

The process of selecting and transforming raw data into a set of features that are more informative and useful for machine learning algorithms. Some examples of feature extraction include transforming raw text into word embeddings and extracting important features such as edges or shapes from image/video data.


feed forward chunking

In each residual attention block in transformers the self-attention layer is usually followed by 2 feed forward layers.
The intermediate embedding size of the feed forward layers is often bigger than the hidden size of the model (e.g., for
bert-base-uncased).
For an input of size [batch_size, sequence_length], the memory required to store the intermediate feed forward
embeddings [batch_size, sequence_length, config.intermediate_size] can account for a large fraction of the memory
use. The authors of Reformer: The Efficient Transformer noticed that since the
computation is independent of the sequence_length dimension, it is mathematically equiv


feed forward chunking

alent to compute the output
embeddings of both feed forward layers [batch_size, config.hidden_size]_0, ..., [batch_size, config.hidden_size]_n
individually and concat them afterward to [batch_size, sequence_length, config.hidden_size] with n = sequence_length, which trades increased computation time against reduced memory use, but yields a mathematically
equivalent result.
For models employing the function apply_chunking_to_forward(), the chunk_size defines the number of output
embeddings that are computed in parallel and thus defines the trade-off between memory and time complexity. If
chunk_size is set to 0, no feed forward chunking is done.


finetuned models

Finetuning is a form of transfer learning which involves taking a pretrained model, freezing its weights, and replacing the output layer with a newly added model head. The model head is trained on your target dataset.
See the Fine-tune a pretrained model tutorial for more details, and learn how to fine-tune models with 🤗 Transformers.


head

The model head refers to the last layer of a neural network that accepts the raw hidden states and projects them onto a different dimension. There is a different model head for each task. For example:
GPT2ForSequenceClassification is a sequence classification head - a linear layer - on top of the base GPT2Model. ViTForImageClassification is an image classification head - a linear layer on top of the final hidden state of the CLS token - on top of the base ViTModel. Wav2Vec2ForCTC ia a language modeling head with CTC on top of the base Wav2Vec2Model.


image patch

Vision-based Transformers models split an image into smaller patches which are linearly embedded, and then passed as a sequence to the model. You can find the patch_size - or resolution - of the model in its configuration.


inference

Inference is the process of evaluating a model on new data after training is complete. See the Pipeline for inference tutorial to learn how to perform inference with 🤗 Transformers.


input IDs

The input ids are often the only required parameters to be passed to the model as input. They are token indices,
numerical representations of tokens building the sequences that will be used as input by the model.
Each tokenizer works differently but the underlying mechanism remains the same. Here’s an example using the BERT
tokenizer, which is a WordPiece tokenizer:
  Copied >>> from transformers import BertTokenizer
>>> tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
>>> sequence = "A Titan RTX has 24GB of VRAM"
The tokenizer takes care of splitting the sequence into tokens available in the tokenizer vocabulary.
  Copied >>> tokenized_sequence = tokenizer.tokenize(sequence)


input IDs

The tokens are either words or subwords. Here for instance, “VRAM” wasn’t in the model vocabulary, so it’s been split
in “V”, “RA” and “M”. To indicate those tokens are not separate words but parts of the same word, a double-hash prefix
is added for “RA” and “M”:
  Copied >>> print(tokenized_sequence)
['A', 'Titan', 'R', '##T', '##X', 'has', '24', '##GB', 'of', 'V', '##RA', '##M']
These tokens can then be converted into IDs which are understandable by the model. This can be done by directly feeding
the sentence to the tokenizer, which leverages the Rust implementation of 🤗
Tokenizers for peak performance.
  Copied >>> inputs = tokenizer(sequence)
The tokenizer returns a dictionary with


input IDs

all the arguments necessary for its corresponding model to work properly. The
token indices are under the key input_ids:
  Copied >>> encoded_sequence = inputs["input_ids"]
>>> print(encoded_sequence)
[101, 138, 18696, 155, 1942, 3190, 1144, 1572, 13745, 1104, 159, 9664, 2107, 102]
Note that the tokenizer automatically adds “special tokens” (if the associated model relies on them) which are special
IDs the model sometimes uses.
If we decode the previous sequence of ids,
  Copied >>> decoded_sequence = tokenizer.decode(encoded_sequence)
we will see
  Copied >>> print(decoded_sequence)
[CLS] A Titan RTX has 24GB of VRAM [SEP]
because this is the way a BertModel is going to expect its input


labels

The labels are an optional argument which can be passed in order for the model to compute the loss itself. These labels
should be the expected prediction of the model: it will use the standard loss in order to compute the loss between its
predictions and the expected value (the label).
These labels are different according to the model head, for example:
For sequence classification models, (BertForSequenceClassification), the model expects a tensor of dimension
(batch_size) with each value of the batch corresponding to the expected label of the entire sequence. For token classification models, (BertForTokenClassification), the model expects a tensor of dimension
(batch_size, seq_length) with each value corresponding to the expected label of each individual token. For masked language modeling, (BertForMaskedLM), the model expects a tensor of dimension (batch_size, seq_length) with each value c


labels

orresponding to the expected label of each individual token: the labels being the token
ID for the masked token, and values to be ignored for the rest (usually -100). For sequence to sequence tasks, (BartForConditionalGeneration, MBartForConditionalGeneration), the model
expects a tensor of dimension (batch_size, tgt_seq_length) with each value corresponding to the target sequences
associated with each input sequence. During training, both BART and T5 will make the appropriate
decoder_input_ids and decoder attention masks internally. They usually do not need to be supplied. This does not
apply to models leveraging the Encoder-Decoder framework. For image classification models, (ViTForImageClassification), the model expects a tensor of dimension
(batch_size) with each value of the batch corresponding to the expected label of each individual image. For semantic segmentation models, (SegformerF


labels

orSemanticSegmentation), the model expects a tensor of dimension
(batch_size, height, width) with each value of the batch corresponding to the expected label of each individual pixel. For object detection models, (DetrForObjectDetection), the model expects a list of dictionaries with a
class_labels and boxes key where each value of the batch corresponds to the expected label and number of bounding boxes of each individual image. For automatic speech recognition models, (Wav2Vec2ForCTC), the model expects a tensor of dimension (batch_size, target_length) with each value corresponding to the expected label of each individual token.
Each model’s labels may be different, so be sure to always check the documentation of each model for more information
about their specific labels!
The base models (BertModel) do not accept labels, as these are the base transformer models, simply outputting
features.


large language models (LLM)

A generic term that refers to transformer language models (GPT-3, BLOOM, OPT) that were trained on a large quantity of data. These models also tend to have a large number of learnable parameters (e.g. 175 billion for GPT-3).


masked language modeling (MLM)

A pretraining task where the model sees a corrupted version of the texts, usually done by
masking some tokens randomly, and has to predict the original text.


Natural language understanding (NLU)

All tasks related to understanding what is in a text (for instance classifying the
whole text, individual words).


pipeline

A pipeline in 🤗 Transformers is an abstraction referring to a series of steps that are executed in a specific order to preprocess and transform data and return a prediction from a model. Some example stages found in a pipeline might be data preprocessing, feature extraction, and normalization.
For more details, see Pipelines for inference.


pixel values

A tensor of the numerical representations of an image that is passed to a model. The pixel values have a shape of [batch_size, num_channels, height, width], and are generated from an image processor.


pooling

An operation that reduces a matrix into a smaller matrix, either by taking the maximum or average of the pooled dimension(s). Pooling layers are commonly found between convolutional layers to downsample the feature representation.


position IDs

Contrary to RNNs that have the position of each token embedded within them, transformers are unaware of the position of
each token. Therefore, the position IDs (position_ids) are used by the model to identify each token’s position in the
list of tokens.
They are an optional parameter. If no position_ids are passed to the model, the IDs are automatically created as
absolute positional embeddings.
Absolute positional embeddings are selected in the range [0, config.max_position_embeddings - 1]. Some models use
other types of positional embeddings, such as sinusoidal position embeddings or relative position embeddings.


preprocessing

The task of preparing raw data into a format that can be easily consumed by machine learning models. For example, text is typically preprocessed by tokenization. To gain a better idea of what preprocessing looks like for other input types, check out the Preprocess tutorial.


pretrained model

A model that has been pretrained on some data (for instance all of Wikipedia). Pretraining methods involve a
self-supervised objective, which can be reading the text and trying to predict the next word (see causal language
modeling) or masking some words and trying to predict them (see masked language
modeling).
Speech and vision models have their own pretraining objectives. For example, Wav2Vec2 is a speech model pretrained on a contrastive task which requires the model to identify the “true” speech representation from a set of “false” speech representations. On the other hand, BEiT is a vision model pretrained on a masked image modeling task which masks some of the image patches and requires the model to predict the masked patches (similar to the masked language modeling objective).


representation learning

A subfield of machine learning which focuses on learning meaningful representations of raw data. Some examples of representation learning techniques include word embeddings, autoencoders, and Generative Adversarial Networks (GANs).


sampling rate

A measurement in hertz of the number of samples (the audio signal) taken per second. The sampling rate is a result of discretizing a continuous signal such as speech.


self-supervised learning

A category of machine learning techniques in which a model creates its own learning objective from unlabeled data. It differs from unsupervised learning and supervised learning in that the learning process is supervised, but not explicitly from the user.
One example of self-supervised learning is masked language modeling, where a model is passed sentences with a proportion of its tokens removed and learns to predict the missing tokens.


semi-supervised learning

A broad category of machine learning training techniques that leverages a small amount of labeled data with a larger quantity of unlabeled data to improve the accuracy of a model, unlike supervised learning and unsupervised learning.
An example of a semi-supervised learning approach is “self-training”, in which a model is trained on labeled data, and then used to make predictions on the unlabeled data. The portion of the unlabeled data that the model predicts with the most confidence gets added to the labeled dataset and used to retrain the model.


sequence-to-sequence (seq2seq)

Models that generate a new sequence from an input, like translation models, or summarization models (such as
Bart or T5).


stride

In convolution or pooling, the stride refers to the distance the kernel is moved over a matrix. A stride of 1 means the kernel is moved one pixel over at a time, and a stride of 2 means the kernel is moved two pixels over at a time.


supervised learning

A form of model training that directly uses labeled data to correct and instruct model performance. Data is fed into the model being trained, and its predictions are compared to the known labels. The model updates its weights based on how incorrect its predictions were, and the process is repeated to optimize model performance.


token

A part of a sentence, usually a word, but can also be a subword (non-common words are often split in subwords) or a
punctuation symbol.


token Type IDs

Some models’ purpose is to do classification on pairs of sentences or question answering.
These require two different sequences to be joined in a single “input_ids” entry, which usually is performed with the
help of special tokens, such as the classifier ([CLS]) and separator ([SEP]) tokens. For example, the BERT model
builds its two sequence input as such:
  Copied >>> # [CLS] SEQUENCE_A [SEP] SEQUENCE_B [SEP]
We can use our tokenizer to automatically generate such a sentence by passing the two sequences to tokenizer as two
arguments (and not a list, like before) like this:
  Copied >>> from transformers import BertTokenizer
>>> tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
>>> sequence_a = "HuggingFace is based in NYC"
>>> sequence_b = "Where is HuggingFace based?"
>>> encoded_dict = tokenizer(sequence_a, sequence_b)
>>> decoded = tokeniz


token Type IDs

er.decode(encoded_dict["input_ids"])
which will return:
  Copied >>> print(decoded)
[CLS] HuggingFace is based in NYC [SEP] Where is HuggingFace based? [SEP]
This is enough for some models to understand where one sequence ends and where another begins. However, other models,
such as BERT, also deploy token type IDs (also called segment IDs). They are represented as a binary mask identifying
the two types of sequence in the model.
The tokenizer returns this mask as the “token_type_ids” entry:
  Copied >>> encoded_dict["token_type_ids"]
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1]
The first sequence, the “context” used for the question, has all its tokens represented by a 0, whereas the second
sequence, corresponding to the “question”, has all its tokens represented by a 1.
Some models, like XLNetModel use an additional token represented by a 2


transfer learning

A technique that involves taking a pretrained model and adapting it to a dataset specific to your task. Instead of training a model from scratch, you can leverage knowledge obtained from an existing model as a starting point. This speeds up the learning process and reduces the amount of training data needed.


unsupervised learning

A form of model training in which data provided to the model is not labeled. Unsupervised learning techniques leverage statistical information of the data distribution to find patterns useful for the task at hand.


Run training on Amazon SageMaker

The documentation has been moved to hf.co/docs/sagemaker. This page will be removed in transformers 5.0.


Table of Content

Train Hugging Face models on Amazon SageMaker with the SageMaker Python SDK Deploy Hugging Face models to Amazon SageMaker with the SageMaker Python SDK Frequently Asked Questions


Summary of the tokenizers

On this page, we will have a closer look at tokenization.
As we saw in the preprocessing tutorial, tokenizing a text is splitting it into words or
subwords, which then are converted to ids through a look-up table. Converting words or subwords to ids is
straightforward, so in this summary, we will focus on splitting a text into words or subwords (i.e. tokenizing a text).
More specifically, we will look at the three main types of tokenizers used in 🤗 Transformers: Byte-Pair Encoding
(BPE), WordPiece, and SentencePiece, and show examples
of which tokenizer type is used by which model.
Note that on each model page, you can look at the documentation of the associated tokenizer to know which tokenizer
type was used by the pretrained model. For instance, if we look at BertTokenizer, we can see
that the model uses WordPiece.


Introduction

Splitting a text into smaller chunks is a task that is harder than it looks, and there are multiple ways of doing so.
For instance, let’s look at the sentence "Don't you love 🤗 Transformers? We sure do."
A simple way of tokenizing this text is to split it by spaces, which would give:
  Copied ["Don't", "you", "love", "🤗", "Transformers?", "We", "sure", "do."]
This is a sensible first step, but if we look at the tokens "Transformers?" and "do.", we notice that the
punctuation is attached to the words "Transformer" and "do", which is suboptimal. We should take the
punctuation into account so that a model does not have to learn a different representation of a word and every possible
punctuation symbol that could follow it, which would explode the number of representations the model has to learn.
Taking punctuation into account, tokenizi


Introduction

ng our exemplary text would give:
  Copied ["Don", "'", "t", "you", "love", "🤗", "Transformers", "?", "We", "sure", "do", "."]
Better. However, it is disadvantageous, how the tokenization dealt with the word "Don't". "Don't" stands for
"do not", so it would be better tokenized as ["Do", "n't"]. This is where things start getting complicated, and
part of the reason each model has its own tokenizer type. Depending on the rules we apply for tokenizing a text, a
different tokenized output is generated for the same text. A pretrained model only performs properly if you feed it an
input that was tokenized with the same rules that were used to tokenize its training data.
spaCy and Moses are two popular
rule-based tokenizers. Applying them on our example, spaCy and Moses would output something like:
  Copied ["Do", "n't", "you", "love", "🤗",


Introduction

 "Transformers", "?", "We", "sure", "do", "."]
As can be seen space and punctuation tokenization, as well as rule-based tokenization, is used here. Space and
punctuation tokenization and rule-based tokenization are both examples of word tokenization, which is loosely defined
as splitting sentences into words. While it’s the most intuitive way to split texts into smaller chunks, this
tokenization method can lead to problems for massive text corpora. In this case, space and punctuation tokenization
usually generates a very big vocabulary (the set of all unique words and tokens used). E.g., Transformer XL uses space and punctuation tokenization, resulting in a vocabulary size of 267,735!
Such a big vocabulary size forces the model to have an enormous embedding matrix as the input and output layer, which
causes both an increased memory a


Introduction

nd time complexity. In general, transformers models rarely have a vocabulary size
greater than 50,000, especially if they are pretrained only on a single language.
So if simple space and punctuation tokenization is unsatisfactory, why not simply tokenize on characters?
While character tokenization is very simple and would greatly reduce memory and time complexity it makes it much harder
for the model to learn meaningful input representations. E.g. learning a meaningful context-independent
representation for the letter "t" is much harder than learning a context-independent representation for the word
"today". Therefore, character tokenization is often accompanied by a loss of performance. So to get the best of
both worlds, transformers models use a hybrid between word-level and character-level tokenization called subword
tokenization.


Subword tokenization

Subword tokenization algorithms rely on the principle that frequently used words should not be split into smaller
subwords, but rare words should be decomposed into meaningful subwords. For instance "annoyingly" might be
considered a rare word and could be decomposed into "annoying" and "ly". Both "annoying" and "ly" as
stand-alone subwords would appear more frequently while at the same time the meaning of "annoyingly" is kept by the
composite meaning of "annoying" and "ly". This is especially useful in agglutinative languages such as Turkish,
where you can form (almost) arbitrarily long complex words by stringing together subwords.
Subword tokenization allows the model to have a reasonable vocabulary size while being able to learn meaningful
context-independent representations. In addition, subwor


Subword tokenization

d tokenization enables the model to process words it has never
seen before, by decomposing them into known subwords. For instance, the BertTokenizer tokenizes
"I have a new GPU!" as follows:
  Copied >>> from transformers import BertTokenizer
>>> tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
>>> tokenizer.tokenize("I have a new GPU!")
["i", "have", "a", "new", "gp", "##u", "!"]
Because we are considering the uncased model, the sentence was lowercased first. We can see that the words ["i", "have", "a", "new"] are present in the tokenizer’s vocabulary, but the word "gpu" is not. Consequently, the
tokenizer splits "gpu" into known subwords: ["gp" and "##u"]. "##" means that the rest of the token should
be attached to the previous one, without space (for decoding or reversal of the tok


Subword tokenization

enization).
As another example, XLNetTokenizer tokenizes our previously exemplary text as follows:
  Copied >>> from transformers import XLNetTokenizer
>>> tokenizer = XLNetTokenizer.from_pretrained("xlnet-base-cased")
>>> tokenizer.tokenize("Don't you love 🤗 Transformers? We sure do.")
["▁Don", "'", "t", "▁you", "▁love", "▁", "🤗", "▁", "Transform", "ers", "?", "▁We", "▁sure", "▁do", "."]
We’ll get back to the meaning of those "▁" when we look at SentencePiece. As one can see,
the rare word "Transformers" has been split into the more frequent subwords "Transform" and "ers".
Let’s now look at how the different subword tokenization algorithms work. Note that all of those tokenization
algorithms rely on some form of training which is usually done on the corpus the corresponding model will be trained
o


Byte-Pair Encoding (BPE)

Byte-Pair Encoding (BPE) was introduced in Neural Machine Translation of Rare Words with Subword Units (Sennrich et
al., 2015). BPE relies on a pre-tokenizer that splits the training data into
words. Pretokenization can be as simple as space tokenization, e.g. GPT-2, RoBERTa. More advanced pre-tokenization include rule-based tokenization, e.g. XLM,
FlauBERT which uses Moses for most languages, or GPT which uses
Spacy and ftfy, to count the frequency of each word in the training corpus.
After pre-tokenization, a set of unique words has been created and the frequency with which each word occurred in the
training data has been determined. Next, BPE creates a base vocabulary consisting of all symbols that occur in the set
of unique words and learns merge rules to form a new symbol from two symbols of the base vocabulary. It do


Byte-Pair Encoding (BPE)

es so until
the vocabulary has attained the desired vocabulary size. Note that the desired vocabulary size is a hyperparameter to
define before training the tokenizer.
As an example, let’s assume that after pre-tokenization, the following set of words including their frequency has been
determined:
  Copied ("hug", 10), ("pug", 5), ("pun", 12), ("bun", 4), ("hugs", 5)
Consequently, the base vocabulary is ["b", "g", "h", "n", "p", "s", "u"]. Splitting all words into symbols of the
base vocabulary, we obtain:
  Copied ("h" "u" "g", 10), ("p" "u" "g", 5), ("p" "u" "n", 12), ("b" "u" "n", 4), ("h" "u" "g" "s", 5)
BPE then counts the frequency of each possible symbol pair and picks the symbol pair that occurs most frequently. In
the example above "h" followed by "u" is present 10 + 5 = 15 times (10 times in the 10 occurrences of


Byte-Pair Encoding (BPE)

"hug", 5 times in the 5 occurrences of "hugs"). However, the most frequent symbol pair is "u" followed by
"g", occurring 10 + 5 + 5 = 20 times in total. Thus, the first merge rule the tokenizer learns is to group all
"u" symbols followed by a "g" symbol together. Next, "ug" is added to the vocabulary. The set of words then
becomes
  Copied ("h" "ug", 10), ("p" "ug", 5), ("p" "u" "n", 12), ("b" "u" "n", 4), ("h" "ug" "s", 5)
BPE then identifies the next most common symbol pair. It’s "u" followed by "n", which occurs 16 times. "u",
"n" is merged to "un" and added to the vocabulary. The next most frequent symbol pair is "h" followed by
"ug", occurring 15 times. Again the pair is merged and "hug" can be added to the vocabulary.
At this stage, the vocabulary is ["b", "g", "h", "n", "p", "s", "u", "ug", "un", "hug"] and our set


Byte-Pair Encoding (BPE)

 of unique words
is represented as
  Copied ("hug", 10), ("p" "ug", 5), ("p" "un", 12), ("b" "un", 4), ("hug" "s", 5)
Assuming, that the Byte-Pair Encoding training would stop at this point, the learned merge rules would then be applied
to new words (as long as those new words do not include symbols that were not in the base vocabulary). For instance,
the word "bug" would be tokenized to ["b", "ug"] but "mug" would be tokenized as ["<unk>", "ug"] since
the symbol "m" is not in the base vocabulary. In general, single letters such as "m" are not replaced by the
"<unk>" symbol because the training data usually includes at least one occurrence of each letter, but it is likely
to happen for very special characters like emojis.
As mentioned earlier, the vocabulary size, i.e. the base vocabulary size + the number of merges, is a


Byte-Pair Encoding (BPE)

hyperparameter
to choose. For instance GPT has a vocabulary size of 40,478 since they have 478 base characters
and chose to stop training after 40,000 merges.
 Byte-level BPE
A base vocabulary that includes all possible base characters can be quite large if e.g. all unicode characters are
considered as base characters. To have a better base vocabulary, GPT-2 uses bytes
as the base vocabulary, which is a clever trick to force the base vocabulary to be of size 256 while ensuring that
every base character is included in the vocabulary. With some additional rules to deal with punctuation, the GPT2’s
tokenizer can tokenize every text without the need for the <unk> symbol. GPT-2 has a vocabulary
size of 50,257, which corresponds to the 256 bytes base tokens, a special end-of-text token and the symbols learned
with 50,000 merges.


WordPiece

WordPiece is the subword tokenization algorithm used for BERT, DistilBERT, and Electra. The algorithm was outlined in Japanese and Korean
Voice Search (Schuster et al., 2012) and is very similar to
BPE. WordPiece first initializes the vocabulary to include every character present in the training data and
progressively learns a given number of merge rules. In contrast to BPE, WordPiece does not choose the most frequent
symbol pair, but the one that maximizes the likelihood of the training data once added to the vocabulary.
So what does this mean exact


WordPiece

ly? Referring to the previous example, maximizing the likelihood of the training data is
equivalent to finding the symbol pair, whose probability divided by the probabilities of its first symbol followed by
its second symbol is the greatest among all symbol pairs. E.g. "u", followed by "g" would have only been
merged if the probability of "ug" divided by "u", "g" would have been greater than for any other symbol
pair. Intuitively, WordPiece is slightly different to BPE in that it evaluates what it loses by merging two symbols
to ensure it’s worth it.


Unigram

Unigram is a subword tokenization algorithm introduced in Subword Regularization: Improving Neural Network Translation
Models with Multiple Subword Candidates (Kudo, 2018). In contrast to BPE or
WordPiece, Unigram initializes its base vocabulary to a large number of symbols and progressively trims down each
symbol to obtain a smaller vocabulary. The base vocabulary could for instance correspond to all pre-tokenized words and
the most common substrings. Unigram is not used directly for any of the models in the transformers, but it’s used in
conjunction with SentencePiece.
At each training step, the Unigram algorithm defines a loss (often defined as the log-likelihood) over the training
data given the current vocabulary and a unigram language model. Then, for each symbol in the vocabulary, the algorithm
computes


Unigram

 how much the overall loss would increase if the symbol was to be removed from the vocabulary. Unigram then
removes p (with p usually being 10% or 20%) percent of the symbols whose loss increase is the lowest, i.e. those
symbols that least affect the overall loss over the training data. This process is repeated until the vocabulary has
reached the desired size. The Unigram algorithm always keeps the base characters so that any word can be tokenized.
Because Unigram is not based on merge rules (in contrast to BPE and WordPiece), the algorithm has several ways of
tokenizing new text after training. As an example, if a trained Unigram tokenizer exhibits the vocabulary:
  Copied ["b", "g", "h", "n", "p", "s", "u", "ug", "un", "hug"],
"hugs" could be tokenized both as ["hug", "s"], ["h", "ug", "s"] or ["h", "u", "g


Unigram

", "s"]. So which one
to choose? Unigram saves the probability of each token in the training corpus on top of saving the vocabulary so that
the probability of each possible tokenization can be computed after training. The algorithm simply picks the most
likely tokenization in practice, but also offers the possibility to sample a possible tokenization according to their
probabilities.
Those probabilities are defined by the loss the tokenizer is trained on. Assuming that the training data consists of
the words x1,…,xNx_{1}, \dots, x_{N}x1,…,xN and that the set of all possible tokenizations for a word xix_{i}xi is
defined as S(xi)S(x_{i})S(xi), then the overall loss is defined as
L=−∑i=1Nlog(∑x∈S(xi)p(x))\mathcal{L} = -\sum_{i=1}^{N} \log \left ( \sum_{x \in S(x_{i})} p(x) \right )L=−i=1∑Nlogx∈S(xi)∑p(x)


SentencePiece

All tokenization algorithms described so far have the same problem: It is assumed that the input text uses spaces to
separate words. However, not all languages use spaces to separate words. One possible solution is to use language
specific pre-tokenizers, e.g. XLM uses a specific Chinese, Japanese, and Thai pre-tokenizer).
To solve this problem more generally, SentencePiece: A simple and language independent subword tokenizer and
detokenizer for Neural Text Processing (Kudo et al., 2018) treats the input
as a raw input stream, thus including the


SentencePiece

 space in the set of characters to use. It then uses the BPE or unigram
algorithm to construct the appropriate vocabulary.
The XLNetTokenizer uses SentencePiece for example, which is also why in the example earlier the
"▁" character was included in the vocabulary. Decoding with SentencePiece is very easy since all tokens can just be
concatenated and "▁" is replaced by a space.
All transformers models in the library that use SentencePiece use it in combination with unigram. Examples of models
using SentencePiece are ALBERT, XLNet, Marian, and T5.


Inference on Specialized Hardware

This document will be completed soon with information on how to infer on specialized hardware. In the meantime you can check out the guide for inference on CPUs.


Multilingual models for inference

There are several multilingual models in 🤗 Transformers, and their inference usage differs from monolingual models. Not all multilingual model usage is different though. Some models, like bert-base-multilingual-uncased, can be used just like a monolingual model. This guide will show you how to use multilingual models whose usage differs for inference.


XLM

XLM has ten different checkpoints, only one of which is monolingual. The nine remaining model checkpoints can be split into two categories: the checkpoints that use language embeddings and those that don’t.


XLM with language embeddings

The following XLM models use language embeddings to specify the language used at inference:
xlm-mlm-ende-1024 (Masked language modeling, English-German) xlm-mlm-enfr-1024 (Masked language modeling, English-French) xlm-mlm-enro-1024 (Masked language modeling, English-Romanian) xlm-mlm-xnli15-1024 (Masked language modeling, XNLI languages) xlm-mlm-tlm-xnli15-1024 (Masked language modeling + translation, XNLI languages) xlm-clm-enfr-1024 (Causal language modeling, English-French) xlm-clm-ende-1024 (Causal language modeling, English-German)
Language embeddings are represented as a tensor of the same shape as the input_ids passed to the model. The values in these tensors depend on the language used and


XLM with language embeddings

are identified by the tokenizer’s lang2id and id2lang attributes.
In this example, load the xlm-clm-enfr-1024 checkpoint (Causal language modeling, English-French):
  Copied >>> import torch
>>> from transformers import XLMTokenizer, XLMWithLMHeadModel
>>> tokenizer = XLMTokenizer.from_pretrained("xlm-clm-enfr-1024")
>>> model = XLMWithLMHeadModel.from_pretrained("xlm-clm-enfr-1024")
The lang2id attribute of the tokenizer displays this model’s languages and their ids:
  Copied >>> print(tokenizer.lang2id)
{'en': 0, 'fr': 1}
Next, create an example input:
  Copied >>> input_ids = torch.tensor([tokenizer.encode("Wikipedia was used to")])  # batch size of 1
Set the language id as "en" and use it to de


XLM with language embeddings

fine the language embedding. The language embedding is a tensor filled with 0 since that is the language id for English. This tensor should be the same size as input_ids.
  Copied >>> language_id = tokenizer.lang2id["en"]  # 0
>>> langs = torch.tensor([language_id] * input_ids.shape[1])  # torch.tensor([0, 0, 0, ..., 0])
>>> # We reshape it to be of size (batch_size, sequence_length)
>>> langs = langs.view(1, -1)  # is now of shape [1, sequence_length] (we have a batch size of 1)
Now you can pass the input_ids and language embedding to the model:
  Copied >>> outputs = model(input_ids, langs=langs)
The run_generation.py script can generate text with language embeddings using the xlm-clm checkpoints


XLM without language embeddings

The following XLM models do not require language embeddings during inference:
xlm-mlm-17-1280 (Masked language modeling, 17 languages) xlm-mlm-100-1280 (Masked language modeling, 100 languages)
These models are used for generic sentence representations, unlike the previous XLM checkpoints.


BERT

The following BERT models can be used for multilingual tasks:
bert-base-multilingual-uncased (Masked language modeling + Next sentence prediction, 102 languages) bert-base-multilingual-cased (Masked language modeling + Next sentence prediction, 104 languages)
These models do not require language embeddings during inference. They should identify the language from the
context and infer accordingly.


XLM-RoBERTa

The following XLM-RoBERTa models can be used for multilingual tasks:
xlm-roberta-base (Masked language modeling, 100 languages) xlm-roberta-large (Masked language modeling, 100 languages)
XLM-RoBERTa was trained on 2.5TB of newly created and cleaned CommonCrawl data in 100 languages. It provides strong gains over previously released multilingual models like mBERT or XLM on downstream tasks like classification, sequence labeling, and question answering.


M2M100

The following M2M100 models can be used for multilingual translation:
facebook/m2m100_418M (Translation) facebook/m2m100_1.2B (Translation)
In this example, load the facebook/m2m100_418M checkpoint to translate from Chinese to English. You can set the source language in the tokenizer:
  Copied >>> from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer
>>> en_text = "Do not meddle in the affairs of wizards, for they are subtle and quick to anger."
>>> chinese_text = "不要插手巫師的事務, 因為他們是微妙的, 很快就會發怒."
>>> tokenizer = M2M100Tokenizer.from_pretrained("facebook/m2m100_418M", src_lang="zh")
>>> model = M2M10


M2M100

0ForConditionalGeneration.from_pretrained("facebook/m2m100_418M")
Tokenize the text:
  Copied >>> encoded_zh = tokenizer(chinese_text, return_tensors="pt")
M2M100 forces the target language id as the first generated token to translate to the target language. Set the forced_bos_token_id to en in the generate method to translate to English:
  Copied >>> generated_tokens = model.generate(**encoded_zh, forced_bos_token_id=tokenizer.get_lang_id("en"))
>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
'Do not interfere with the matters of the witches, because they are delicate and will soon be angry.'


MBart

The following MBart models can be used for multilingual translation:
facebook/mbart-large-50-one-to-many-mmt (One-to-many multilingual machine translation, 50 languages) facebook/mbart-large-50-many-to-many-mmt (Many-to-many multilingual machine translation, 50 languages) facebook/mbart-large-50-many-to-one-mmt (Many-to-one multilingual machine translation, 50 languages) facebook/mbart-large-50 (Multilingual translation, 50 languages) facebook/mbart-large-cc25
In this example, load the facebook/mbart-large-50-many-to-many-mmt checkpoint to translate Finnish to English. You can set the source language in the tokenizer:
  Copied >>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
>>> en_text = "Do not meddle in the affairs of wizards, for they are subtle and quick to anger."
>>> fi_text = "Älä sekaannu velhojen asioihin, sillä ne ovat hienovaraisia ja nopeasti vihaisia."
>>> tokenizer = A


MBart

utoTokenizer.from_pretrained("facebook/mbart-large-50-many-to-many-mmt", src_lang="fi_FI")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")
Tokenize the text:
  Copied >>> encoded_en = tokenizer(en_text, return_tensors="pt")
MBart forces the target language id as the first generated token to translate to the target language. Set the forced_bos_token_id to en in the generate method to translate to English:
  Copied >>> generated_tokens = model.generate(**encoded_en, forced_bos_token_id=tokenizer.lang_code_to_id["en_XX"])
>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
"Don't interfere with the wizard's affairs, because they are subtle, will soon get angry."
If you are using the facebook/mbart-large-50-many-to-one-mmt checkpoint, you don’t need to force the target language id as the first generated token otherwise the usage is the same


Efficient Training on Multiple CPUs

When training on a single CPU is too slow, we can use multiple CPUs. This guide focuses on PyTorch-based DDP enabling distributed CPU training efficiently.


Intel® oneCCL Bindings for PyTorch

Intel® oneCCL (collective communications library) is a library for efficient distributed deep learning training implementing such collectives like allreduce, allgather, alltoall. For more information on oneCCL, please refer to the oneCCL documentation and oneCCL specification.
Module oneccl_bindings_for_pytorch (torch_ccl before version 1.12)  implements PyTorch C10D ProcessGroup API and can be dynamically loaded as external ProcessGroup and only works on Linux platform now
Check more detailed information for oneccl_bind_pt.


Intel® oneCCL Bindings for PyTorch installation:

Wheel files are available for the following Python versions:
| Extension Version   | Python 3.6   | Python 3.7   | Python 3.8   | Python 3.9   | Python 3.10   |
|---------------------|--------------|--------------|--------------|--------------|---------------|
| 1.13.0              | nan          | √            | √            | √            | √             |
| 1.12.100            | nan          | √            | √            | √            | √             |
| 1.12.0              | nan          | √            | √            | √            | √             |
| 1.11.0              | nan          | √


   | √            | √            | √             |
| 1.10.0              | √            | √            | √            | √            | nan           |
  Copied pip install oneccl_bind_pt=={pytorch_version} -f https://developer.intel.com/ipex-whl-stable-cpu
where {pytorch_version} should be your PyTorch version, for instance 1.13.0.
Check more approaches for oneccl_bind_pt installation.
Versions of oneCCL and PyTorch must match.
oneccl_bindings_for_pytorch 1.12.0 prebuilt wheel does not work with PyTorch 1.12.1 (it is for PyTorch 1.12.0)
PyTorch 1.12.1 should work with oneccl_bindings_for_pytorch 1.12.10


Intel® MPI library

Use this standards-based MPI implementation to deliver flexible, efficient, scalable cluster messaging on Intel® architecture. This component is part of the Intel® oneAPI HPC Toolkit.
oneccl_bindings_for_pytorch is installed along with the MPI tool set. Need to source the environment before using it.
for Intel® oneCCL >= 1.12.0
  Copied oneccl_bindings_for_pytorch_path=$(python -c "from oneccl_bindings_for_pytorch import cwd; print(cwd)")
source $oneccl_bindings_for_pytorch_path/env/setvars.sh
for Intel® oneCCL whose version < 1.12.0
  Copied torch_ccl_path=$(python -c "import torch; import torch_ccl; import os;  print(os.path.abspath(os.path.dirname(torch_ccl.__file__)))")
source $torch_ccl_path/env/setvars.sh
 IPEX installation:
IPEX provides performance optimizations for CPU training with both Float32 and BFloat16, you could refer single CPU section.
The following “Usage in Trainer” takes mpirun in Intel® MPI library as an example.


Usage in Trainer

To enable multi CPU distributed training in the Trainer with the ccl backend, users should add --ddp_backend ccl in the command arguments.
Let’s see an example with the question-answering example
The following command enables training with 2 processes on one Xeon node, with one process running per one socket. The variables OMP_NUM_THREADS/CCL_WORKER_COUNT can be tuned for optimal performance.
  Copied  export CCL_WORKER_COUNT=1
 export MASTER_ADDR=127.0.0.1
 mpirun -n 2 -genv OMP_NUM_THREADS=23 \
 python3 run_qa.py \
 --model_name_or_path bert-large-uncased \
 --dataset_name squad \
 --do_train \
 --do_eval \
 --per_device_train_batch_size 12  \
 --learning_rate 3e-5  \
 --num_train_epochs 2  \
 --max_seq_length 384 \
 --doc_stride 128  \
 --output_dir /tmp/debug_squad/ \
 --no_cuda \
 --ddp_backend ccl \
 --use_ipex
The following command enables training with a total of four processes on two Xeons (node0 and node1, taking node0 as the main process), ppn (processes per nod


Usage in Trainer

e) is set to 2, with one process running per one socket. The variables OMP_NUM_THREADS/CCL_WORKER_COUNT can be tuned for optimal performance.
In node0, you need to create a configuration file which contains the IP addresses of each node (for example hostfile) and pass that configuration file path as an argument.
  Copied  cat hostfile
 xxx.xxx.xxx.xxx #node0 ip
 xxx.xxx.xxx.xxx #node1 ip
Now, run the following command in node0 and 4DDP will be enabled in node0 and node1 with BF16 auto mixed precision:
  Copied  export CCL_WORKER_COUNT=1
 export MASTER_ADDR=xxx.xxx.xxx.xxx #node0 ip
 mpirun -f hostfile -n 4 -ppn 2 \
 -genv OMP_NUM_THREADS=23 \
 python3 run_qa.py \
 --model_name_or_path bert-large-uncased \
 --dataset_name squad \
 --do_train \
 --do_eval \
 --per_device_train_batch_size 12  \
 --learning_rate 3e-5  \
 --num_train_epochs 2  \
 --max_seq_length 384 \
 --doc_stride 128  \
 --output_dir /tmp/debug_squad/ \
 --no_cuda \
 --ddp_backend ccl \
 --use_ipex \
 --bf16


Troubleshoot

Sometimes errors occur, but we are here to help! This guide covers some of the most common issues we’ve seen and how you can resolve them. However, this guide isn’t meant to be a comprehensive collection of every 🤗 Transformers issue. For more help with troubleshooting your issue, try:
Asking for help on the forums. There are specific categories you can post your question to, like Beginners or 🤗 Transformers. Make sure you write a good descriptive forum post with some reproducible code to maximize the


Troubleshoot

likelihood that your problem is solved!
Create an Issue on the 🤗 Transformers repository if it is a bug related to the library. Try to include as much information describing the bug as possible to help us better figure out what’s wrong and how we can fix it. Check the Migration guide if you use an older version of 🤗 Transformers since some important changes have been introduced between versions.
For more details about troubleshooting and getting help, take a look at Chapter 8 of the Hugging Face course


Firewalled environments

Some GPU instances on cloud and intranet setups are firewalled to external connections, resulting in a connection error. When your script attempts to download model weights or datasets, the download will hang and then timeout with the following message:
  Copied ValueError: Connection error, and we cannot find the requested files in the cached path.
Please try again or make sure your Internet connection is on.
In this case, you should try to run 🤗 Transformers on offline mode to avoid the connection error.


CUDA out of memory

Training large models with millions of parameters can be challenging without the appropriate hardware. A common error you may encounter when the GPU runs out of memory is:
  Copied CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 11.17 GiB total capacity; 9.70 GiB already allocated; 179.81 MiB free; 9.85 GiB reserved in total by PyTorch)
Here are some potential solutions you can try to lessen memory use:
Reduce the per_device_train_batch_size value in TrainingArguments. Try using gradient_accumulation_steps in TrainingArguments to effectively increase overall batch size.
Refer to the Performance guide for more details about memory-saving techniques.


Unable to load a saved TensorFlow model

TensorFlow’s model.save method will save the entire model - architecture, weights, training configuration - in a single file. However, when you load the model file again, you may run into an error because 🤗 Transformers may not load all the TensorFlow-related objects in the model file. To avoid issues with saving and loading TensorFlow models, we recommend you:
Save the model weights as a h5 file extension with model.save_weights and then reload the model with from_pretrained():
  Copied >>> from transformers import TFPreTrainedModel
>>> from tensorflow import keras
>>> model.save_weights("some_folder/tf_model.h5")
>>> model = TFPreTrainedModel.from_pretrained("some_folder")
Save the model with ~TFPretrainedModel.save_pretrained and load it again with from_pretrained():
  Copied >>> from transformers import TFPreTrainedModel
>>> model.save_pretrained("path_to/model")
>>> model = TFPreTrainedModel.from_pretrained("path_to/model")


ImportError

Another common error you may encounter, especially if it is a newly released model, is ImportError:
  Copied ImportError: cannot import name 'ImageGPTImageProcessor' from 'transformers' (unknown location)
For these error types, check to make sure you have the latest version of 🤗 Transformers installed to access the most recent models:
  Copied pip install transformers --upgrade


Sometimes you may run into a generic CUDA error about an error in the device code.
  Copied RuntimeError: CUDA error: device-side assert triggered
You should try to run the code on a CPU first to get a more descriptive error message. Add the following environment variable to the beginning of your code to switch to a CPU:
  Copied >>> import os
>>> os.environ["CUDA_VISIBLE_DEVICES"] = ""
Another option is to get a better traceback from the GPU. Add the following environment variable to the beginning of your code to get the traceback to point to the source of the error:
  Copied >>> import os
>>> os.environ["CUDA_LAUNCH_BLOCKING"] = "1"


Incorrect output when padding tokens aren’t masked

In some cases, the output hidden_state may be incorrect if the input_ids include padding tokens. To demonstrate, load a model and tokenizer. You can access a model’s pad_token_id to see its value. The pad_token_id may be None for some models, but you can always manually set it.
  Copied >>> from transformers import AutoModelForSequenceClassification
>>> import torch
>>> model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
>>> model.config.pad_token_id
0
The following example shows the output without masking the padding tokens:
  Copied >>> input_ids = torch.tensor([[7592, 2057, 2097, 2393, 9611, 2115], [7592, 0, 0, 0, 0, 0]])
>>> output = model(input_ids)
>>> print(output.logits)
tensor([[ 0.0082, -0.2307],
        [ 0.1317, -0.1683]], grad_fn=<AddmmBackward0>)
Here is the actual output of the second sequence:
  Copied >>> input_ids = torch.tensor([[


7592]])
>>> output = model(input_ids)
>>> print(output.logits)
tensor([[-0.1008, -0.4061]], grad_fn=<AddmmBackward0>)
Most of the time, you should provide an attention_mask to your model to ignore the padding tokens to avoid this silent error. Now the output of the second sequence matches its actual output:
By default, the tokenizer creates an attention_mask for you based on your specific tokenizer’s defaults.
  Copied >>> attention_mask = torch.tensor([[1, 1, 1, 1, 1, 1], [1, 0, 0, 0, 0, 0]])
>>> output = model(input_ids, attention_mask=attention_mask)
>>> print(output.logits)
tensor([[ 0.0082, -0.2307],
        [-0.1008, -0.4061]], grad_fn=<AddmmBackward0>)
🤗 Transformers doesn’t automatically create an attention_mask to mask a padding token if it is provided because:
Some models don’t have a padding token. For some use-cases, users want a model to attend to a padding token.


ValueError: Unrecognized configuration class XYZ for this kind of AutoModel

Generally, we recommend using the AutoModel class to load pretrained instances of models. This class
can automatically infer and load the correct architecture from a given checkpoint based on the configuration. If you see
this ValueError when loading a model from a checkpoint, this means the Auto class couldn’t find a mapping from
the configuration in the given checkpoint to the kind of model you are trying to load. Most commonly, this happens when a
checkpoint doesn’t support a given task.
For instance, you’ll see this error in the following exampl


e because there is no GPT2 for question answering:
  Copied >>> from transformers import AutoProcessor, AutoModelForQuestionAnswering
>>> processor = AutoProcessor.from_pretrained("gpt2-medium")
>>> model = AutoModelForQuestionAnswering.from_pretrained("gpt2-medium")
ValueError: Unrecognized configuration class <class 'transformers.models.gpt2.configuration_gpt2.GPT2Config'> for this kind of AutoModel: AutoModelForQuestionAnswering.
Model type should be one of AlbertConfig, BartConfig, BertConfig, BigBirdConfig, BigBirdPegasusConfig, BloomConfig, ..


Export to TFLite

TensorFlow Lite is a lightweight framework for deploying machine learning models
on resource-constrained devices, such as mobile phones, embedded systems, and Internet of Things (IoT) devices.
TFLite is designed to optimize and run models efficiently on these devices with limited computational power, memory, and
power consumption.
A TensorFlow Lite model is represented in a special efficient portable format identified by the .tflite file extension.
🤗 Optimum offers functionality to export 🤗 Transformers models to TFLite through the exporters.tflite module.
For the list of supported model architectures, please refer to 🤗 Optimum documentation.
To export a model to TFLite, install


Export to TFLite

the required dependencies:
  Copied pip install optimum[exporters-tf]
To check out all available arguments, refer to the 🤗 Optimum docs,
or view help in command line:
  Copied optimum-cli export tflite --help
To export a model’s checkpoint from the 🤗 Hub, for example, bert-base-uncased, run the following command:
  Copied optimum-cli export tflite --model bert-base-uncased --sequence_length 128 bert_tflite/
You should see the logs indicating progress and showing where the resulting model.tflite is saved, like this:
  Copied Validating TFLite model...
-[✓] TFLite model output names match reference model (logits)
- Validating TFLite Model output "logits":
-[✓] (1, 128, 30522) match


Export to TFLite

es (1, 128, 30522)
-[x] values not close enough, max diff: 5.817413330078125e-05 (atol: 1e-05)
The TensorFlow Lite export succeeded with the warning: The maximum absolute difference between the output of the reference model and the TFLite exported model is not within the set tolerance 1e-05:
- logits: max diff = 5.817413330078125e-05.
 The exported model was saved at: bert_tflite
The example above illustrates exporting a checkpoint from 🤗 Hub. When exporting a local model, first make sure that you
saved both the model’s weights and tokenizer files in the same directory (local_path). When using CLI, pass the
local_path to the model argument instead of the checkpoint name on 🤗 Hub.


Efficient Inference on a Multiple GPUs

This document contains information on how to efficiently infer on a multiple GPUs.
Note: A multi GPU setup can use the majority of the strategies described in the single GPU section. You must be aware of simple techniques, though, that can be used for a better usage.


Flash Attention 2

Flash Attention 2 integration also works in a multi-GPU setup, check out the appropriate section in the single GPU section


BetterTransformer

BetterTransformer converts 🤗 Transformers models to use the PyTorch-native fastpath execution, which calls optimized kernels like Flash Attention under the hood.
BetterTransformer is also supported for faster inference on single and multi-GPU for text, image, and audio models.
Flash Attention can only be used for models using fp16 or bf16 dtype. Make sure to cast your model to the appropriate dtype before using BetterTransformer.


Decoder models

For text models, especially decoder-based models (GPT, T5, Llama, etc.), the BetterTransformer API converts all attention operations to use the torch.nn.functional.scaled_dot_product_attention operator (SDPA) that is only available in PyTorch 2.0 and onwards.
To convert a model to BetterTransformer:
  Copied from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m")
# convert the model to BetterTransformer
model.to_bettertransformer()
# Use it for training or inference
SDPA can also call Flash Attention kernels under the hood. To enable Flash Attention or to check that it is available in a given setting (hardware, problem size), use torch.backends.cuda.sdp_kernel as a context manager:
  Copied import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-


Decoder models

350m")
model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m").to("cuda")
# convert the model to BetterTransformer
model.to_bettertransformer()
input_text = "Hello my dog is cute and"
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")
+ with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):
    outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
If you see a bug with a traceback saying
  Copied RuntimeError: No available kernel.  Aborting execution.
try using the PyTorch nightly version, which may have a broader coverage for Flash Attention:
  Copied pip3 install -U --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu118
Have a look at this blog post to learn more about what is possible with the BetterTransformer + SDPA API


Encoder models

For encoder models during inference, BetterTransformer dispatches the forward call of encoder layers to an equivalent of torch.nn.TransformerEncoderLayer that will execute the fastpath implementation of the encoder layers.
Because torch.nn.TransformerEncoderLayer fastpath does not support training, it is dispatched to torch.nn.functional.scaled_dot_product_attention instead, which does not leverage nested tensors but can use Flash Attention or Memory-Efficient Attention fused kernels.
More details about BetterTransformer performance can be found in this blog post, and you can learn more about BetterTransformer for encoder models in this blog.


You can combine the different methods described above to get the best performance for your model. For example, you can use BetterTransformer with FP4 mixed-precision inference + flash attention:
  Copied import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16
)
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m")
model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m", quantization_config=quantization_config)
input_text = "Hello my dog is cute and"
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")
with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):
    outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))


Export to TorchScript

This is the very beginning of our experiments with TorchScript and we are still
exploring its capabilities with variable-input-size models. It is a focus of interest to
us and we will deepen our analysis in upcoming releases, with more code examples, a more
flexible implementation, and benchmarks comparing Python-based codes with compiled
TorchScript.
According to the TorchScript documentation:
TorchScript is a way to create serializable and optimizable models from PyTorch code.
There are two PyTorch modules, JIT and
TRACE, that allow develop


Export to TorchScript

ers to export their
models to be reused in other programs like efficiency-oriented C++ programs.
We provide an interface that allows you to export 🤗 Transformers models to TorchScript
so they can be reused in a different environment than PyTorch-based Python programs.
Here, we explain how to export and use our models using TorchScript.
Exporting a model requires two things:
model instantiation with the torchscript flag a forward pass with dummy inputs
These necessities imply several things developers should be careful about as detailed
below.


TorchScript flag and tied weights

The torchscript flag is necessary because most of the 🤗 Transformers language models
have tied weights between their Embedding layer and their Decoding layer.
TorchScript does not allow you to export models that have tied weights, so it is
necessary to untie and clone the weights beforehand.
Models instantiated with the torchscript flag have their Embedding layer and
Decoding layer separated, which means that they should not be trained down the line.
Training would desynchronize the two layers, leading to unexpected results.
This is not the case for models that do not have a language model head, as those do not
have tied weights. These models can be safely exported without the torchscript flag.


Dummy inputs and standard lengths

The dummy inputs are used for a models forward pass. While the inputs’ values are
propagated through the layers, PyTorch keeps track of the different operations executed
on each tensor. These recorded operations are then used to create the trace of the
model.
The trace is created relative to the inputs’ dimensions. It is therefore constrained by
the dimensions of the dummy input, and will not work for any other sequence length or
batch size. When trying with a different size, the following error is raised:
  Copied `The expanded size of the t


Dummy inputs and standard lengths

ensor (3) must match the existing size (7) at non-singleton dimension 2`
We recommended you trace the model with a dummy input size at least as large as the
largest input that will be fed to the model during inference. Padding can help fill the
missing values. However, since the model is traced with a larger input size, the
dimensions of the matrix will also be large, resulting in more calculations.
Be careful of the total number of operations done on each input and follow the
performance closely when exporting varying sequence-length models.


Saving a model

To export a BertModel with TorchScript, instantiate BertModel from the BertConfig
class and then save it to disk under the filename traced_bert.pt:
  Copied from transformers import BertModel, BertTokenizer, BertConfig
import torch
enc = BertTokenizer.from_pretrained("bert-base-uncased")
# Tokenizing input text
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = enc.tokenize(text)
# Masking one of the input tokens
masked_index = 8
tokenized_text[masked_index] = "[MASK]"
indexed_tokens = enc.convert_tokens_to_ids(tokenized_text)
segments_ids = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]
# Creating a dummy input
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
dummy_input = [tokens_tenso


Saving a model

r, segments_tensors]
# Initializing the model with the torchscript flag
# Flag set to True even though it is not necessary as this model does not have an LM Head.
config = BertConfig(
    vocab_size_or_config_json_file=32000,
    hidden_size=768,
    num_hidden_layers=12,
    num_attention_heads=12,
    intermediate_size=3072,
    torchscript=True,
)
# Instantiating the model
model = BertModel(config)
# The model needs to be in evaluation mode
model.eval()
# If you are instantiating the model with *from_pretrained* you can also easily set the TorchScript flag
model = BertModel.from_pretrained("bert-base-uncased", torchscript=True)
# Creating the trace
traced_model = torch.jit.trace(model, [tokens_tensor, segments_tensors])
torch.jit.save(traced_model, "traced_bert.pt")


Loading a model

Now you can load the previously saved BertModel, traced_bert.pt, from disk and use
it on the previously initialised dummy_input:
  Copied loaded_model = torch.jit.load("traced_bert.pt")
loaded_model.eval()
all_encoder_layers, pooled_output = loaded_model(*dummy_input)


Using a traced model for inference

Use the traced model for inference by using its __call__ dunder method:
  Copied traced_model(tokens_tensor, segments_tensors)


Deploy Hugging Face TorchScript models to AWS with the Neuron SDK

AWS introduced the Amazon EC2 Inf1
instance family for low cost, high performance machine learning inference in the cloud.
The Inf1 instances are powered by the AWS Inferentia chip, a custom-built hardware
accelerator, specializing in deep learning inferencing workloads. AWS
Neuron is the SDK for
Inferentia that supports tracing and optimizing transformers models for deployment on
Inf1. The Neuron SDK provides:
Easy-to-use API with one line of code change to trace and optimize a TorchScript
model for inference in the cloud. Out of the box performance optimizations for improved
cost-performance. Support for Hugging Face transformers models built with either
PyTorch
or
TensorFlow.


Implications

Transformers models based on the BERT (Bidirectional Encoder Representations from
Transformers)
architecture, or its variants such as
distilBERT and
roBERTa run best on
Inf1 for non-generative tasks such as extractive question answering, sequence
classification, and token classification. However, text generation tasks can still be
adapted to run on Inf1 according to this AWS Neuron MarianMT
tutorial.
More information about models that can be converted out of the box on Inferentia can be
found in the Model Architecture
Fit
section of the Neuron documentation.


Dependencies

Using AWS Neuron to convert models requires a Neuron SDK
environment
which comes preconfigured on AWS Deep Learning
AMI.


Converting a model for AWS Neuron

Convert a model for AWS NEURON using the same code from Using TorchScript in
Python to trace a BertModel. Import the
torch.neuron framework extension to access the components of the Neuron SDK through a
Python API:
  Copied from transformers import BertModel, BertTokenizer, BertConfig
import torch
import torch.neuron
You only need to modify the following line:
  Copied - torch.jit.trace(model, [tokens_tensor, segments_tensors])
+ torch.neuron.trace(model, [token_tensor, segments_tensors])
This enables the Neuron SDK to trace the model and optimize it for Inf1 instances.
To learn more about AWS Neuron SDK features, tools, example tutorials and latest
updates, please see the AWS NeuronSDK
documentation.


BERTology

There is a growing field of study concerned with investigating the inner working of large-scale transformers like BERT
(that some call “BERTology”). Some good examples of this field are:
BERT Rediscovers the Classical NLP Pipeline by Ian Tenney, Dipanjan Das, Ellie Pavlick:
https://arxiv.org/abs/1905.05950 Are Sixteen Heads Really Better than One? by Paul Michel, Omer Levy, Graham Neubig: https://arxiv.org/abs/1905.10650 What Does BERT Look At? An Analysis of BERT’s Attention by Kevin Clark, Urvashi Khandelwal, Omer Levy, Christopher D.
Manning: https://arxiv.org/abs/1906.04341 CAT-probing: A Metric-based Approach to Interpret How Pre-trained Models for Programming Language Attend Code Structure: https:/


BERTology

/arxiv.org/abs/2210.04633
In order to help this new field develop, we have included a few additional features in the BERT/GPT/GPT-2 models to
help people access the inner representations, mainly adapted from the great work of Paul Michel
(https://arxiv.org/abs/1905.10650):
accessing all the hidden-states of BERT/GPT/GPT-2, accessing all the attention weights for each head of BERT/GPT/GPT-2, retrieving heads output values and gradients to be able to compute head importance score and prune head as explained
in https://arxiv.org/abs/1905.10650.
To help you understand and use these features, we have added a specific example script: bertology.py while extract information and prune a model pre-trained on
GLUE.


Load adapters with 🤗 PEFT

Parameter-Efficient Fine Tuning (PEFT) methods freeze the pretrained model parameters during fine-tuning and add a small number of trainable parameters (the adapters) on top of it. The adapters are trained to learn task-specific information. This approach has been shown to be very memory-efficient with lower compute usage while producing results comparable to a fully fine-tuned model.
Adapters trained with PEFT are also usually an order of magnitude smaller than the full model, making it convenient to share, store, and load them.
 The adapter weights for a OPTForCausalLM model stored on the Hub are only ~6MB compared to the full size of the model weights, which can be ~700MB.
If you’re interested in learning more about the 🤗 PEFT library, check out the documentation.


Setup

Get started by installing 🤗 PEFT:
  Copied pip install peft
If you want to try out the brand new features, you might be interested in installing the library from source:
  Copied pip install git+https://github.com/huggingface/peft.git


Supported PEFT models

🤗 Transformers natively supports some PEFT methods, meaning you can load adapter weights stored locally or on the Hub and easily run or train them with a few lines of code. The following methods are supported:
Low Rank Adapters IA3 AdaLoRA
If you want to use other PEFT methods, such as prompt learning or prompt tuning, or about the 🤗 PEFT library in general, please refer to the documentation.


Load a PEFT adapter

To load and use a PEFT adapter model from 🤗 Transformers, make sure the Hub repository or local directory contains an adapter_config.json file and the adapter weights, as shown in the example image above. Then you can load the PEFT adapter model using the AutoModelFor class. For example, to load a PEFT adapter model for causal language modeling:
specify the PEFT model id pass it to the AutoModelForCausalLM class
  Copied from transformers import AutoModelForCausalLM, AutoTokenizer
peft_model_id = "ybe


Load a PEFT adapter

lkada/opt-350m-lora"
model = AutoModelForCausalLM.from_pretrained(peft_model_id)
You can load a PEFT adapter with either an AutoModelFor class or the base model class like OPTForCausalLM or LlamaForCausalLM.
You can also load a PEFT adapter by calling the load_adapter method:
  Copied from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "facebook/opt-350m"
peft_model_id = "ybelkada/opt-350m-lora"
model = AutoModelForCausalLM.from_pretrained(model_id)
model.load_adapter(peft_model_id


Load in 8bit or 4bit

The bitsandbytes integration supports 8bit and 4bit precision data types, which are useful for loading large models because it saves memory (see the bitsandbytes integration guide to learn more). Add the load_in_8bit or load_in_4bit parameters to from_pretrained() and set device_map="auto" to effectively distribute the model to your hardware:
  Copied from transformers import AutoModelForCausalLM, AutoTokenizer
peft_model_id = "ybelkada/opt-350m-lora"
model = AutoModelForCausalLM.from_pretrained(peft_model_id, device_map="auto", load_in_8bit=True)


Add a new adapter

You can use ~peft.PeftModel.add_adapter to add a new adapter to a model with an existing adapter as long as the new adapter is the same type as the current one. For example, if you have an existing LoRA adapter attached to a model:
  Copied from transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer
from peft import PeftConfig
model_id = "facebook/opt-350m"
model = AutoModelForCausalLM.from_pretrained(model_id)
lora_config = LoraConfig(
    target_modules=["q_proj", "k_proj"],
    init_lora_weights=False
)
model.add_adapter(lora


Add a new adapter

_config, adapter_name="adapter_1")
To add a new adapter:
  Copied # attach new adapter with same config
model.add_adapter(lora_config, adapter_name="adapter_2")
Now you can use ~peft.PeftModel.set_adapter to set which adapter to use:
  Copied # use adapter_1
model.set_adapter("adapter_1")
output = model.generate(**inputs)
print(tokenizer.decode(output_disabled[0], skip_special_tokens=True))
# use adapter_2
model.set_adapter("adapter_2")
output_enabled = model.generate(**inputs)
print(tokenizer.decode(output_enabled[0], skip_special_tokens=True))


Enable and disable adapters

Once you’ve added an adapter to a model, you can enable or disable the adapter module. To enable the adapter module:
  Copied from transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer
from peft import PeftConfig
model_id = "facebook/opt-350m"
adapter_model_id = "ybelkada/opt-350m-lora"
tokenizer = AutoTokenizer.from_pretrained(model_id)
text = "Hello"
inputs = tokenizer(text, return_tensors="pt")
model = AutoModelForCausalLM.from_pretrained(model_id)
peft_config = PeftConfig.from_pretrained(adapter_model_id)
# to initiate with random weights
peft_config.init_lora_weights = False
model.add_adapter(peft_config)
model.enable_adapters()
output = model.generate(**inputs)
To disable the adapter module:
  Copied model.disable_adapters()
output = model.generate(**inputs)


Train a PEFT adapter

PEFT adapters are supported by the Trainer class so that you can train an adapter for your specific use case. It only requires adding a few more lines of code. For example, to train a LoRA adapter:
If you aren’t familiar with fine-tuning a model with Trainer, take a look at the Fine-tune a pretrained model tutorial.
Define your adapter configuration with the task type and hyperparameters (see ~peft.LoraConfig for more details about what the hyperparameters do).
  Copied from peft import LoraConfig
peft_config = LoraConfig(
    lora_alpha=16,
    lora_dropout=0.1,
    r=64,
    bias="none",
    task_type="CAUSAL_LM",
)
Add adapter to the model.
  Copied model.add_adapter(peft_config)
Now you can pass the model to Trainer!
  Copied trainer = Trainer(model=model, ...)
trainer.train()
To save your trained adapter and load it back:
  Copied model.save_pretrained(save_dir)
model = AutoModelForCausalLM.from_pretrained(save_dir)


Contribute to 🤗 Transformers

Everyone is welcome to contribute, and we value everybody’s contribution. Code
contributions are not the only way to help the community. Answering questions, helping
others, and improving the documentation are also immensely valuable.
It also helps us if you spread the word! Reference the library in blog posts
about the awesome projects it made possible, shout out on Twitter every time it has
helped you, or simply ⭐ the repository to say thank you.
However you choose to contribute, please be mindful and respect our
code of conduct.
This guide was heavily inspired by the awesome scikit-learn guide to contributing.


Ways to contribute

There are several ways you can contribute to 🤗 Transformers:
Fix outstanding issues with the existing code. Submit issues related to bugs or desired new features. Implement new models. Contribute to the examples or to the documentation.
If you don’t know where to start, there is a special Good First
Issue listing. It will give you a list of
open issues that are beginner-friendly and help you start contributing to open-source. Just comment in the issue that you’d like to work
on it.
For something slightly more challenging, you can also take a look at the Good Second Issue list. In general though, if you feel like you know what you’re doing, go for it and we’ll help you get there! 🚀
All contributions are equally valuable to the community. 🥰


Fixing outstanding issues

If you notice an issue with the existing code and have a fix in mind, feel free to start contributing and open a Pull Request!


Submitting a bug-related issue or feature request

Do your best to follow these guidelines when submitting a bug-related issue or a feature
request. It will make it easier for us to come back to you quickly and with good
feedback.