Supported models and frameworks

      | ❌                    | ❌              |
| VideoMAE                      | ✅                 | ❌                    | ❌              |
| ViLT                          | ✅                 | ❌                    | ❌              |
| Vision Encoder decoder        | ✅                 | ✅                    | ✅              |
| VisionTextDualEncoder         | ✅                 | ✅                    | ✅              |
| VisualBERT                    | ✅                 | ❌                    | ❌              |
| ViT                           | ✅                 | ✅                    | ✅              |
| ViT Hybrid                    | ✅                 | ❌                    | ❌              |
| VitDet                        | ✅                 | ❌                    | ❌              |
| ViTMAE                        | ✅                 | ✅                    | ❌              |
| ViTMatte                      | ✅                 | ❌                    | ❌


Supported models and frameworks

  |
| ViTMSN                        | ✅                 | ❌                    | ❌              |
| VITS                          | ✅                 | ❌                    | ❌              |
| ViViT                         | ✅                 | ❌                    | ❌              |
| Wav2Vec2                      | ✅                 | ✅                    | ✅              |
| Wav2Vec2-Conformer            | ✅                 | ❌                    | ❌              |
| Wav2Vec2Phoneme               | ✅                 | ✅                    | ✅              |
| WavLM                         | ✅                 | ❌                    | ❌              |
| Whisper                       | ✅                 | ✅                    | ✅              |
| X-CLIP                        | ✅                 | ❌                    | ❌              |
| X-MOD                         | ✅                 | ❌                    | ❌              |
| XGLM                          | ✅


Supported models and frameworks

            | ✅                    | ✅              |
| XLM                           | ✅                 | ✅                    | ❌              |
| XLM-ProphetNet                | ✅                 | ❌                    | ❌              |
| XLM-RoBERTa                   | ✅                 | ✅                    | ✅              |
| XLM-RoBERTa-XL                | ✅                 | ❌                    | ❌              |
| XLM-V                         | ✅                 | ✅                    | ✅              |
| XLNet                         | ✅                 | ✅                    | ❌              |
| XLS-R                         | ✅                 | ✅                    | ✅              |
| XLSR-Wav2Vec2                 | ✅                 | ✅                    | ✅              |
| YOLOS                         | ✅                 | ❌                    | ❌              |
| YOSO                          | ✅                 | ❌                    | ❌


Efficient Training on Multiple GPUs

When training on a single GPU is too slow or the model weights don’t fit in a single GPUs memory we use a multi-GPU setup. Switching from a single GPU to multiple requires some form of parallelism as the work needs to be distributed. There are several techniques to achieve parallism such as data, tensor, or pipeline parallism. However, there is no one solution to fit them all and which settings works best depends on the hardware you are running on. While the main concepts most likely will apply to any other framework, this article is focused on PyTorch-based implementations.


Efficient Training on Multiple GPUs

Note: Most of the strategies introduced in the single GPU section (such as mixed precision training or gradient accumulation) are generic and apply to training models in general so make sure to have a look at it before diving into the following sections such as multi-GPU or CPU training.
We will first discuss in depth various 1D parallelism techniques and their pros and cons and then look at how they can be combined into 2D and 3D parallelism to enable an even faster training and to support even bigger models. Various other powerful alternative approaches will be presented.


Concepts

The following is the brief description of the main concepts that will be described later in depth in this document.
DataParallel (DP) - the same setup is replicated multiple times, and each being fed a slice of the data. The processing is done in parallel and all setups are synchronized at the end of each training step. TensorParallel (TP) - each tensor is split up into multiple chunks, so instead of having the whole tensor reside on a single gpu, each shard of the tensor resides on its designated gpu. During processing each shard gets processed separately and in parallel on different GPUs and the results are synced at the end of the step. This is what one may call horizontal parallelism, as the splitting happens on horizontal level. PipelineParallel (PP) - the model is split up vertically (layer


Concepts

-level) across multiple GPUs, so that only one or several layers of the model are places on a single gpu. Each gpu processes in parallel different stages of the pipeline and working on a small chunk of the batch. Zero Redundancy Optimizer (ZeRO) - Also performs sharding of the tensors somewhat similar to TP, except the whole tensor gets reconstructed in time for a forward or backward computation, therefore the model doesn’t need to be modified. It also supports various offloading techniques to compensate for limited GPU memory. Sharded DDP - is another name for the foundational ZeRO concept as used by various other implementations of ZeRO.
Before diving deeper into the specifics of each concept we first have a look at the rough decision process when training large models on a large infrastructure


Scalability Strategy

⇨ Single Node / Multi-GPU
Model fits onto a single GPU: DDP - Distributed DP ZeRO - may or may not be faster depending on the situation and configuration used Model doesn’t fit onto a single GPU: PP ZeRO TP With very fast intra-node connectivity of NVLINK or NVSwitch all three should be mostly on par, without these PP will be faster than TP or ZeRO. The degree of TP may also make a difference. Best to experiment to find the winner on your particular setup. TP is almost always used within a single node. That is TP size <= gpus per node. Largest Layer not fitting into a single GPU: If not using ZeRO - must use TP, as PP alone won’t be able to fit. With ZeRO see the same entry for “Single GPU” above
⇨ Multi-Node / Multi-GPU
When you have fast inter-node connectivity: ZeRO - as it requires close to no modifications to the model PP+TP+DP - less communications, but requires massive changes to the model when you have slow inter-node connectivity and still low on GPU memory: DP+PP+TP+ZeRO-1


Data Parallelism

Most users with just 2 GPUs already enjoy the increased training speed up thanks to DataParallel (DP) and DistributedDataParallel (DDP) that are almost trivial to use. This is a built-in feature of Pytorch. Note that in general it is advised to use DDP as it is better maintained and works for all models while DP might fail for some models. PyTorch documentation itself recommends the use of DDP.


DP vs DDP

DistributedDataParallel (DDP) is typically faster than DataParallel (DP), but it is not always the case:
while DP is python threads-based, DDP is multiprocess-based - and as such it has no python threads limitations, such as GIL on the other hand a slow inter-connectivity between the GPU cards could lead to an actual slower outcome with DDP
Here are the main differences in the inter-GPU communication overhead between the two modes:
DDP:
At the start time the main process replicates the model once from gpu 0 to the rest of gpus Then for each batch:each gpu consumes each own mini-batch of data directly during backward, once the local gradients are ready, they are then averaged across all processes
DP:
For each batch:
gpu 0 reads the batch of data and then sends a mini-batch to each gpu replicates the up-to-date model from gpu 0 to each gpu runs forward and sends output from each gpu to gpu 0, computes loss scatters loss from g


DP vs DDP

pu 0 to all gpus, runs backward sends gradients from each gpu to gpu 0 and averages those
The only communication DDP performs per batch is sending gradients, whereas DP does 5 different data exchanges per batch.
DP copies data within the process via python threads, whereas DDP copies data via torch.distributed.
Under DP gpu 0 performs a lot more work than the rest of the gpus, thus resulting in under-utilization of gpus.
You can use DDP across multiple machines, but this is not the case with DP.
There are other differences between DP and DDP but they aren’t relevant to this discussion.
If you want to go really deep into understanding these 2 modes, this article is highly recommended, as it has great diagrams, includes multiple benchmarks and profiler outputs on various hardware, explains all the nuances that you may need to know.
Let’s look at an actual benchmark:
| Type   | NVlink   | Time   |
|--------|----------|--------|


DP vs DDP

| 2:DP   | Y        | 110s   |
| 2:DDP  | Y        | 101s   |
| 2:DDP  | N        | 131s   |
Analysis:
Here DP is ~10% slower than DDP w/ NVlink, but ~15% faster than DDP w/o NVlink
The real difference will depend on how much data each GPU needs to sync with the others - the more there is to sync, the more a slow link will slow down the total runtime.
Here is the full benchmark code and outputs:
NCCL_P2P_DISABLE=1 was used to disable the NVLink feature on the corresponding benchmark.
  Copied
# DP
rm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 \
python examples/pytorch/language-modeling/run_clm.py \
--model_name_or_path gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
--do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200
{'train_runtime': 110.5948, 'train_samples_per_second': 1.808, 'epoch': 0.69}
# DDP w/ NVlink
rm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 \
python -m


DP vs DDP

torch.distributed.launch --nproc_per_node 2 examples/pytorch/language-modeling/run_clm.py \
--model_name_or_path gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
--do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200
{'train_runtime': 101.9003, 'train_samples_per_second': 1.963, 'epoch': 0.69}
# DDP w/o NVlink
rm -r /tmp/test-clm; NCCL_P2P_DISABLE=1 CUDA_VISIBLE_DEVICES=0,1 \
python -m torch.distributed.launch --nproc_per_node 2 examples/pytorch/language-modeling/run_clm.py \
--model_name_or_path gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
--do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200
{'train_runtime': 131.4367, 'train_samples_per_second': 1.522, 'epoch': 0.69}
Hardware: 2x TITAN RTX 24GB each + NVlink with 2 NVLinks (NV2 in nvidia-smi topo -m)
Software: pytorch-1.8-to-be + cuda-11.0 / transformers==4.3.0.d


ZeRO Data Parallelism

ZeRO-powered data parallelism (ZeRO-DP) is described on the following diagram from this blog post
It can be difficult to wrap one’s head around it, but in reality the concept is quite simple. This is just the usual DataParallel (DP), except, instead of replicating the full model params, gradients and optimizer states, each GPU stores only a slice of it.  And then at run-time when the full layer params are needed just for the given layer, all GPUs synchronize to give each other parts that they miss - this is it.
Consider this simple model with 3 layers, where each layer has 3 params:
  Copied La | Lb | Lc
---|----|---
a0 | b0 | c0
a1 | b1 | c1
a2 | b2 | c2
Layer La has weights a0, a1 and a2.
If we have 3 GPUs, the Sharded DDP (= Zero-DP) splits the model onto 3 GPUs like so:
  Copied G


ZeRO Data Parallelism

PU0:
La | Lb | Lc
---|----|---
a0 | b0 | c0
GPU1:
La | Lb | Lc
---|----|---
a1 | b1 | c1
GPU2:
La | Lb | Lc
---|----|---
a2 | b2 | c2
In a way this is the same horizontal slicing, as tensor parallelism, if you imagine the typical DNN diagram. Vertical slicing is where one puts whole layer-groups on different GPUs. But it’s just the starting point.
Now each of these GPUs will get the usual mini-batch as it works in DP:
  Copied x0 => GPU0
x1 => GPU1
x2 => GPU2
The inputs are unmodified - they think they are going to be processed by the normal model.
First, the inputs hit the layer La.
Let’s focus just on GPU0: x0 needs a0, a1, a2 params to do its forward path, but GPU0 has only a0 - it gets sent a1 from GPU1 and a2 from GPU2, bringing all pieces of the model together.
In parallel, GPU1


ZeRO Data Parallelism

gets mini-batch x1 and it only has a1, but needs a0 and a2 params, so it gets those from GPU0 and GPU2.
Same happens to GPU2 that gets input x2. It gets a0 and a1 from GPU0 and GPU1, and with its a2 it reconstructs the full tensor.
All 3 GPUs get the full tensors reconstructed and a forward happens.
As soon as the calculation is done, the data that is no longer needed gets dropped - it’s only used during the calculation. The reconstruction is done efficiently via a pre-fetch.
And the whole process is repeated for layer Lb, then Lc forward-wise, and then backward Lc -> Lb -> La.
To me this sounds like an efficient group backpacking weight distribution strategy:
person A carries the tent person B carries the stove person C carries the axe
Now each night they all share what they have with


ZeRO Data Parallelism

 others and get from others what they don’t have, and in the morning they pack up their allocated type of gear and continue on their way. This is Sharded DDP / Zero DP.
Compare this strategy to the simple one where each person has to carry their own tent, stove and axe, which would be far more inefficient. This is DataParallel (DP and DDP) in Pytorch.
While reading the literature on this topic you may encounter the following synonyms: Sharded, Partitioned.
If you pay close attention the way ZeRO partitions the model’s weights - it looks very similar to tensor parallelism which will be discussed later. This is because it partitions/shards each layer’s weights, unlike vertical model parallelism which is discussed next.
Implementations:
DeepSpeed ZeRO-DP stages 1+2+3 transformers integrat


Naive Model Parallelism (Vertical) and Pipeline Parallelism

Naive Model Parallelism (MP) is where one spreads groups of model layers across multiple GPUs. The mechanism is relatively simple - switch the desired layers .to() the desired devices and now whenever the data goes in and out those layers switch the data to the same device as the layer and leave the rest unmodified.
We refer to it as Vertical MP, because if you remember how most models are drawn, we slice the layers vertically. For example, if the following diagram shows an 8-layer model:
  Copied ===================  ===================
|  0 | 1 | 2 | 3  |  |  4 | 5 | 6 | 7  |
===================  ===================
        gpu0                 gpu1
we just sliced it in 2 vertically, placing layers 0-3 onto GPU0 and 4-7 to GPU1.
Now while data travels from layer 0 to 1, 1 to 2 and 2 to 3 this is just the normal model. But when data needs to pass from layer 3 to layer 4 it needs to travel from GPU0 to GPU1 which int


roduces a communication overhead. If the participating GPUs are on the same compute node (e.g. same physical machine) this copying is pretty fast, but if the GPUs are located on different compute nodes (e.g. multiple machines) the communication overhead could be significantly larger.
Then layers 4 to 5 to 6 to 7 are as a normal model would have and when the 7th layer completes we often need to send the data back to layer 0 where the labels are (or alternatively send the labels to the last layer). Now the loss can be computed and the optimizer can do its work.
Problems:
the main deficiency and why this one is called “naive” MP, is that all but one GPU is idle at any given moment. So if 4 GPUs are used, it’s almost identical to quadrupling the amount of memory of a single GPU, and ignoring the rest of the hardware. Plus there is the overhead of copying the data between devices. So 4x 6GB cards will be able to accommoda


te the same size as 1x 24GB card using naive MP, except the latter will complete the training faster, since it doesn’t have the data copying overhead. But, say, if you have 40GB cards and need to fit a 45GB model you can with 4x 40GB cards (but barely because of the gradient and optimizer states) shared embeddings may need to get copied back and forth between GPUs.
Pipeline Parallelism (PP) is almost identical to a naive MP, but it solves the GPU idling problem, by chunking the incoming batch into micro-batches and artificially creating a pipeline, which allows different GPUs to concurrently participate in the computation process.
The following illustration from the GPipe paper shows the naive MP on the top, and PP on the bottom:
It’s easy to see from the bottom diagram how PP has less dead zones, where GPUs are idle. The idle parts are referred to as the “bubble”.
Both parts of the diagram show a parallelism that is


 of degree 4. That is 4 GPUs are participating in the pipeline. So there is the forward path of 4 pipe stages F0, F1, F2 and F3 and then the return reverse order backward path of B3, B2, B1 and B0.
PP introduces a new hyper-parameter to tune and it’s chunks which defines how many chunks of data are sent in a sequence through the same pipe stage. For example, in the bottom diagram you can see that chunks=4. GPU0 performs the same forward path on chunk 0, 1, 2 and 3 (F0,0, F0,1, F0,2, F0,3) and then it waits for other GPUs to do their work and only when their work is starting to be complete, GPU0 starts to work again doing the backward path for chunks 3, 2, 1 and 0 (B0,3, B0,2, B0,1, B0,0).
Note that conceptually this is the same concept as gradient accumulation steps (GAS). Pytorch uses chunks, whereas DeepSpeed refers to the same hyper-parameter as GAS.
Because of the chunks, PP introduces the concept of micro-batche


s (MBS). DP splits the global data batch size into mini-batches, so if you have a DP degree of 4, a global batch size of 1024 gets split up into 4 mini-batches of 256 each (1024/4). And if the number of chunks (or GAS) is 32 we end up with a micro-batch size of 8 (256/32). Each Pipeline stage works with a single micro-batch at a time.
To calculate the global batch size of the DP + PP setup we then do: mbs*chunks*dp_degree (8*32*4=1024).
Let’s go back to the diagram.
With chunks=1 you end up with the naive MP, which is very inefficient. With a very large chunks value you end up with tiny micro-batch sizes which could be not every efficient either. So one has to experiment to find the value that leads to the highest efficient utilization of the gpus.
While the diagram shows that there is a bubble of “dead” time that can’t be parallelized because the last forward stage has to wait for backward to complete the pipeline,


the purpose of finding the best value for chunks is to enable a high concurrent GPU utilization across all participating GPUs which translates to minimizing the size of the bubble.
There are 2 groups of solutions - the traditional Pipeline API and the more modern solutions that make things much easier for the end user.
Traditional Pipeline API solutions:
PyTorch DeepSpeed Megatron-LM
Modern solutions:
Varuna Sagemaker
Problems with traditional Pipeline API solutions:
have to modify the model quite heavily, because Pipeline requires one to rewrite the normal flow of modules into a nn.Sequential sequence of the same, which may require changes to the design of the model. currently the Pipeline API is very restricted. If you had a bunch of python variables being passed in the very first stage of the Pipeline, you will have to find a way around it. Currently, the pipeline interface requires either a single Tensor or a tup


le of Tensors as the only input and output. These tensors must have a batch size as the very first dimension, since pipeline is going to chunk the mini batch into micro-batches. Possible improvements are being discussed here https://github.com/pytorch/pytorch/pull/50693 conditional control flow at the level of pipe stages is not possible - e.g., Encoder-Decoder models like T5 require special workarounds to handle a conditional encoder stage. have to arrange each layer so that the output of one model becomes an input to the other model.
We are yet to experiment with Varuna and SageMaker but their papers report that they have overcome the list of problems mentioned above and that they require much smaller changes to the user’s model.
Implementations:
Pytorch (initial support in pytorch-1.8, and progressively getting improved in 1.9 and more so in 1.10). Some examples DeepSpeed Megatron-LM has an internal implementation


 - no API. Varuna SageMaker - this is a proprietary solution that can only be used on AWS. OSLO - this is implemented based on the Hugging Face Transformers.
🤗 Transformers status: as of this writing none of the models supports full-PP. GPT2 and T5 models have naive MP support. The main obstacle is being unable to convert the models to nn.Sequential and have all the inputs to be Tensors. This is because currently the models include many features that make the conversion very complicated, and will need to be removed to accomplish that.
Other approaches:
DeepSpeed, Varuna and SageMaker use the concept of an Interleaved Pipeline
Here the bubble (idle time) is further minimized by prioritizing backward passes.
Varuna further tries to improve the schedule by using simulations to discover the most efficient scheduling.
OSLO has pipeline parallelism implementation based on the Transformers without nn.Sequential converting.


Tensor Parallelism

In Tensor Parallelism each GPU processes only a slice of a tensor and only aggregates the full tensor for operations that require the whole thing.
In this section we use concepts and diagrams from the Megatron-LM paper: Efficient Large-Scale Language Model Training on GPU Clusters.
The main building block of any transformer is a fully connected nn.Linear followed by a nonlinear activation GeLU.
Following the Megatron’s paper notation, we can write the dot-product part of it as Y = GeLU(XA), where X and Y are the input and output vectors, and A is the weight matrix.
If we look at the computation in matrix form, it’s easy to see how the matrix multiplication can be split between multiple GPUs:
If we split the weight matrix A column-wise across N GPUs and perform matrix multiplications XA_1 through XA_n in parallel, then we w


Tensor Parallelism

ill end up with N output vectors Y_1, Y_2, ..., Y_n which can be fed into GeLU independently:
Using this principle, we can update an MLP of arbitrary depth, without the need for any synchronization between GPUs until the very end, where we need to reconstruct the output vector from shards. The Megatron-LM paper authors provide a helpful illustration for that:
Parallelizing the multi-headed attention layers is even simpler, since they are already inherently parallel, due to having multiple independent heads!
Special considerations: TP requires very fast network, and therefore it’s not advisable to do TP across more than one node. Practically, if a node has 4 GPUs, the highest TP degree is therefore 4. If you need a TP degree of 8, you need to use nodes that have at least 8 GPUs.
This section is based on the original much mo


Tensor Parallelism

re detailed TP overview.
by @anton-l.
SageMaker combines TP with DP for a more efficient processing.
Alternative names:
DeepSpeed calls it tensor slicing
Implementations:
Megatron-LM has an internal implementation, as it’s very model-specific parallelformers (only inference at the moment) SageMaker - this is a proprietary solution that can only be used on AWS. OSLO has the tensor parallelism implementation based on the Transformers.
🤗 Transformers status:
core: not yet implemented in the core but if you want inference parallelformers provides this support for most of our models. So until this is implemented in the core you can use theirs. And hopefully training mode will be supported too. Deepspeed-Inference also supports our BERT, GPT-2, and GPT-Neo models in their super-fast CUDA-kernel-based inference mode, see more her


DP+PP

The following diagram from the DeepSpeed pipeline tutorial demonstrates how one combines DP with PP.
Here it’s important to see how DP rank 0 doesn’t see GPU2 and DP rank 1 doesn’t see GPU3. To DP there is just GPUs 0 and 1 where it feeds data as if there were just 2 GPUs. GPU0 “secretly” offloads some of its load to GPU2 using PP. And GPU1 does the same by enlisting GPU3 to its aid.
Since each dimension requires at least 2 GPUs, here you’d need at least 4 GPUs.
Implementations:
DeepSpeed Megatron-LM Varuna SageMaker OSLO
🤗 Transformers status: not yet implemented


DP+PP+TP

To get an even more efficient training a 3D parallelism is used where PP is combined with TP and DP. This can be seen in the following diagram.
This diagram is from a blog post 3D parallelism: Scaling to trillion-parameter models, which is a good read as well.
Since each dimension requires at least 2 GPUs, here you’d need at least 8 GPUs.
Implementations:
DeepSpeed - DeepSpeed also includes an even more efficient DP, which they call ZeRO-DP. Megatron-LM Varuna SageMaker OSLO
🤗 Transformers status: not yet implemented, since we have no PP and TP.


ZeRO DP+PP+TP

One of the main features of DeepSpeed is ZeRO, which is a super-scalable extension of DP. It has already been discussed in ZeRO Data Parallelism. Normally it’s a standalone feature that doesn’t require PP or TP. But it can be combined with PP and TP.
When ZeRO-DP is combined with PP (and optionally TP) it typically enables only ZeRO stage 1 (optimizer sharding).
While it’s theoretically possible to use ZeRO stage 2 (gradient sharding) with Pipeline Parallelism, it will have bad performance impacts. There would need to be an additional reduce-scatter collective for every micro-batch to aggregate the gradients before sharding, which adds a potentially significant communication overhead. By nature of Pipeline Parallelism, small micro-batches are used and instead the focus is on trying to balance arithmetic intensity (micro-batch size) with minimizing the Pi


ZeRO DP+PP+TP

peline bubble (number of micro-batches). Therefore those communication costs are going to hurt.
In addition, There are already fewer layers than normal due to PP and so the memory savings won’t be huge. PP already reduces gradient size by 1/PP, and so gradient sharding savings on top of that are less significant than pure DP.
ZeRO stage 3 is not a good choice either for the same reason - more inter-node communications required.
And since we have ZeRO, the other benefit is ZeRO-Offload. Since this is stage 1 optimizer states can be offloaded to CPU.
Implementations:
Megatron-DeepSpeed and Megatron-Deepspeed from BigScience, which is the fork of the former repo. OSLO
Important papers:
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
🤗 Transformers status: not yet implemented, since we have no PP and TP


FlexFlow

FlexFlow also solves the parallelization problem in a slightly different approach.
Paper: “Beyond Data and Model Parallelism for Deep Neural Networks” by Zhihao Jia, Matei Zaharia, Alex Aiken
It performs a sort of 4D Parallelism over Sample-Operator-Attribute-Parameter.
Sample = Data Parallelism (sample-wise parallel) Operator = Parallelize a single operation into several sub-operations Attribute = Data Parallelism (length-wise parallel) Parameter = Model Parallelism (regardless of dimension - horizontal or vertical)
Examples:
Sample
Let’s take 10 batches of sequence length 512. If we parallelize them by sample dimension into 2 devices, we get 10 x 512 which becomes be 5 x 2 x 512.
Operator
If we perform layer normalization, we compute std first and mean second, and the


FlexFlow

n we can normalize data. Operator parallelism allows computing std and mean in parallel. So if we parallelize them by operator dimension into 2 devices (cuda:0, cuda:1), first we copy input data into both devices, and cuda:0 computes std, cuda:1 computes mean at the same time.
Attribute
We have 10 batches of 512 length. If we parallelize them by attribute dimension into 2 devices, 10 x 512 will be 10 x 2 x 256.
Parameter
It is similar with tensor model parallelism or naive layer-wise model parallelism.
The significance of this framework is that it takes resources like (1) GPU/TPU/CPU vs. (2) RAM/DRAM vs. (3) fast-intra-connect/slow-inter-connect and it automatically optimizes all these  algorithmically deciding which parallelisation to use where.
One very important aspe


FlexFlow

ct is that FlexFlow is designed for optimizing DNN parallelizations for models with static and fixed workloads, since models with dynamic behavior may prefer different parallelization strategies across iterations.
So the promise is very attractive - it runs a 30min simulation on the cluster of choice and it comes up with the best strategy to utilise this specific environment. If you add/remove/replace any parts it’ll run and re-optimize the plan for that. And then you can train. A different setup will have its own custom optimization.
🤗 Transformers status: not yet integrated. We already have our models FX-trace-able via transformers.utils.fx, which is a prerequisite for FlexFlow, so someone needs to figure out what needs to be done to make FlexFlow work with our models


Which Strategy To Use When

Here is a very rough outline at which parallelism strategy to use when. The first on each list is typically faster.
⇨ Single GPU
Model fits onto a single GPU: Normal use Model doesn’t fit onto a single GPU: ZeRO + Offload CPU and optionally NVMe as above plus Memory Centric Tiling (see below for details) if the largest layer can’t fit into a single GPU Largest Layer not fitting into a single GPU:
ZeRO - Enable Memory Centric Tiling (MCT). It allows you to run arbitrarily large layers by automatically splitting them and executing them sequentially. MCT reduces the number of parameters that are live on a GPU, but it does not affect the activation memory. As this need is very rare as of this writing a manual override of torch.nn.Linear needs to be done by the user.
⇨ Single Node / Multi-GPU
Model fits onto a single GPU: DDP - Distributed DP ZeRO - may or may not be faster dep


Which Strategy To Use When

ending on the situation and configuration used Model doesn’t fit onto a single GPU: PP ZeRO TP With very fast intra-node connectivity of NVLINK or NVSwitch all three should be mostly on par, without these PP will be faster than TP or ZeRO. The degree of TP may also make a difference. Best to experiment to find the winner on your particular setup. TP is almost always used within a single node. That is TP size <= gpus per node. Largest Layer not fitting into a single GPU: If not using ZeRO - must use TP, as PP alone won’t be able to fit. With ZeRO see the same entry for “Single GPU” above
⇨ Multi-Node / Multi-GPU
When you have fast inter-node connectivity: ZeRO - as it requires close to no modifications to the model PP+TP+DP - less communications, but requires massive changes to the model when you have slow inter-node connectivity and still low on GPU memory: DP+PP+TP+ZeRO-1


Mixed precision with IPEX

IPEX is optimized for CPUs with AVX-512 or above, and functionally works for CPUs with only AVX2. So, it is expected to bring performance benefit for Intel CPU generations with AVX-512 or above while CPUs with only AVX2 (e.g., AMD CPUs or older Intel CPUs) might result in a better performance under IPEX, but not guaranteed. IPEX provides performance optimizations for CPU training with both Float32 and BFloat16. The usage of BFloat16 is the main focus of the following sections.
Low precision data type BFloat16 has been natively supported on the 3rd Generation Xeon® Scalable Processors (aka Cooper Lake) with AVX512 instruction


Mixed precision with IPEX

set and will be supported on the next generation of Intel® Xeon® Scalable Processors with Intel® Advanced Matrix Extensions (Intel® AMX) instruction set with further boosted performance. The Auto Mixed Precision for CPU backend has been enabled since PyTorch-1.10. At the same time, the support of Auto Mixed Precision with BFloat16 for CPU and BFloat16 optimization of operators has been massively enabled in Intel® Extension for PyTorch, and partially upstreamed to PyTorch master branch. Users can get better performance and user experience with IPEX Auto Mixed Precision.
Check more detailed information for Auto Mixed Precision.


IPEX installation:

IPEX release is following PyTorch, to install via pip:
|   PyTorch Version | IPEX version   |
|-------------------|----------------|
|              1.13 | 1.13.0+cpu     |
|              1.12 | 1.12.300+cpu   |
|              1.11 | 1.11.200+cpu   |
|              1.1  | 1.10.100+cpu   |
  Copied pip install intel_extension_for_pytorch==<version_name> -f https://developer.intel.com/ipex-whl-stable-cpu
Check more approaches for IPEX installation.


Usage in Trainer

To enable auto mixed precision with IPEX in Trainer, users should add use_ipex, bf16 and no_cuda in training command arguments.
Take an example of the use cases on Transformers question-answering
Training with IPEX using BF16 auto mixed precision on CPU: python run_qa.py \
--model_name_or_path bert-base-uncased \
--dataset_name squad \
--do_train \
--do_eval \
--per_device_train_batch_size 12 \
--learning_rate 3e-5 \
--num_train_epochs 2 \
--max_seq_length 384 \
--doc_stride 128 \
--output_dir /tmp/debug_squad/ \
--use_ipex \
--bf16 --no_cuda


Methods and tools for efficient training on a single GPU

This guide demonstrates practical techniques that you can use to increase the efficiency of your model’s training by
optimizing memory utilization, speeding up the training, or both. If you’d like to understand how GPU is utilized during
training, please refer to the Model training anatomy conceptual guide first. This guide
focuses on practical techniques.
If you have access to a machine with multiple GPUs, these approaches are still valid, plus you can leverage additional methods outlined in the multi-GPU section.
When training large models, there are two aspects that should be considered at the same time:
Data throughput/training time Model performance
Maximizing the throughput (samples/second) leads to lower training cost. This is generally achieved by utilizing the GPU
as much as possible and thus filling GPU memory to its limit. If the


desired batch size exceeds the limits of the GPU memory,
the memory optimization techniques, such as gradient accumulation, can help.
However, if the preferred batch size fits into memory, there’s no reason to apply memory-optimizing techniques because they can
slow down the training. Just because one can use a large batch size, does not necessarily mean they should. As part of
hyperparameter tuning, you should determine which batch size yields the best results and then optimize resources accordingly.
The methods and tools covered in this guide can be classified based on the effect they have on the training process:
| Method/tool              | Improves training speed   | Optimizes memory utilization   |
|--------------------------|---------------------------|--------------------------------|
| Batch size choice        | Yes


      | Yes                            |
| Gradient accumulation    | No                        | Yes                            |
| Gradient checkpointing   | No                        | Yes                            |
| Mixed precision training | Yes                       | (No)                           |
| Optimizer choice         | Yes                       | Yes                            |
| Data preloading          | Yes                       | No                             |
| DeepSpeed Zero           | No                        | Yes                            |
| torch.compile            | Yes                       | No                             |
Note: when using mixed precision with a small model and a large batch size, there will be some memory savings but with a
large model and a small batch size, the memory use will be la


rger.
You can combine the above methods to get a cumulative effect. These techniques are available to you whether you are
training your model with Trainer or writing a pure PyTorch loop, in which case you can configure these optimizations
with 🤗 Accelerate.
If these methods do not result in sufficient gains, you can explore the following options:
Look into building your own custom Docker container with efficient softare prebuilds Consider a model that uses Mixture of Experts (MoE) Convert your model to BetterTransformer to leverage PyTorch native attention
Finally, if all of the above is still not enough, even after switching to a server-grade GPU like A100, consider moving
to a multi-GPU setup. All these approaches are still valid in a multi-GPU setup, plus you can leverage additional parallelism
techniques outlined in the multi-GPU section


Batch size choice

To achieve optimal performance, start by identifying the appropriate batch size. It is recommended to use batch sizes and
input/output neuron counts that are of size 2^N. Often it’s a multiple of 8, but it can be
higher depending on the hardware being used and the model’s dtype.
For reference, check out NVIDIA’s recommendation for input/output neuron counts and
batch size for
fully connected layers (which are involved in GEMMs (General Matrix Multiplications)).
Tensor Core Requirements
define the multiplier based on the dtype and the hardware. For instance, for fp16 data type a multiple of 8 is recommended, unless
it’s an A100 GPU, in which case use multiples of 64.
For parameters that are small, consider also Dimension Quantization Effects.
This is where tiling happens and the right multiplier can have a significant speedup.


Gradient Accumulation

The gradient accumulation method aims to calculate gradients in smaller increments instead of computing them for the
entire batch at once. This approach involves iteratively calculating gradients in smaller batches by performing forward
and backward passes through the model and accumulating the gradients during the process. Once a sufficient number of
gradients have been accumulated, the model’s optimization step is executed. By employing gradient accumulation, it
becomes possible to increase the effective batch size beyond the limitations imposed by the GPU’s memory capacity.
However, it is important to note that the additional forward and backward passes introduced by gradient accumulation can
slow down the training process.
You can enable gradient accumulation by adding the gradient_accumulation_steps argument to  TrainingArguments:
  Copied training_args = TrainingArguments(per_device_train_batch_size=1, gradient_accumulation_


Gradient Accumulation

steps=4, **default_args)
In the above example, your effective batch size becomes 4.
Alternatively, use 🤗 Accelerate to gain full control over the training loop. Find the 🤗 Accelerate example
further down in this guide.
While it is advised to max out GPU usage as much as possible, a high number of gradient accumulation steps can
result in a more pronounced training slowdown. Consider the following example. Let’s say, the per_device_train_batch_size=4
without gradient accumulation hits the GPU’s limit. If you would like to train with batches of size 64, do not set the
per_device_train_batch_size to 1 and gradient_accumulation_steps to 64. Instead, keep per_device_train_batch_size=4
and set gradient_accumulation_steps=16. This results in the same effective batch size while making better use of
the available GPU resources.
For additional information, please refer to batch size and gradient accumulation benchmarks for RTX-3090
and A100


Gradient Checkpointing

Some large models may still face memory issues even when the batch size is set to 1 and gradient accumulation is used.
This is because there are other components that also require memory storage.
Saving all activations from the forward pass in order to compute the gradients during the backward pass can result in
significant memory overhead. The alternative approach of discarding the activations and recalculating them when needed
during the backward pass, would introduce a considerable computational overhead and slow down the training process.
Gradient checkpointing offers a compromise between these two approaches and saves strategically selected


Gradient Checkpointing

activations
throughout the computational graph so only a fraction of the activations need to be re-computed for the gradients. For
an in-depth explanation of gradient checkpointing, refer to this great article.
To enable gradient checkpointing in the Trainer, pass the corresponding a flag to TrainingArguments:
  Copied training_args = TrainingArguments(
    per_device_train_batch_size=1, gradient_accumulation_steps=4, gradient_checkpointing=True, **default_args
)
Alternatively, use 🤗 Accelerate - find the 🤗 Accelerate example further in this guide.
While gradient checkpointing may improve memory efficiency, it slows training by approximately 20%.


Mixed precision training

Mixed precision training is a technique that aims to optimize the computational efficiency of training models by
utilizing lower-precision numerical formats for certain variables. Traditionally, most models use 32-bit floating point
precision (fp32 or float32) to represent and process variables. However, not all variables require this high precision
level to achieve accurate results. By reducing the precision of certain variables to lower numerical formats like 16-bit
floating point (fp16 or float


Mixed precision training

16), we can speed up the computations. Because in this approach some computations are performed
in half-precision, while some are still in full precision, the approach is called mixed precision training.
Most commonly mixed precision training is achieved by using fp16 (float16) data types, however, some GPU architectures
(such as the Ampere architecture) offer bf16 and tf32 (CUDA internal data type) data types. Check
out the NVIDIA Blog to learn more about
the differences between these data types.


fp16

The main advantage of mixed precision training comes from saving the activations in half precision (fp16).
Although the gradients are also computed in half precision they are converted back to full precision for the optimization
step so no memory is saved here.
While mixed precision training results in faster computations, it can also lead to more GPU memory being utilized, especially for small batch sizes.
This is because the model is now present on the GPU in both 16-bit and 32-bit precision (1.5x the original model on the GPU).
To enable mixed precision training, set the fp16 flag to True:
  Copied training_args = TrainingArguments(per_device_train_batch_size=4, fp16=True, **default_args)
If you prefer to use 🤗 Accelerate, find the 🤗 Accelerate example further in this guide.


BF16

If you have access to an Ampere or newer hardware you can use bf16 for mixed precision training and evaluation. While
bf16 has a worse precision than fp16, it has a much bigger dynamic range. In fp16 the biggest number you can have
is 65535 and any number above that will result in an overflow. A bf16 number can be as large as 3.39e+38 (!) which
is about the same as fp32 - because both have 8-bits used for the numerical range.
You can enable BF16 in the 🤗 Trainer with:
  Copied training_args = TrainingArguments(bf16=True, **default_args)


TF32

The Ampere hardware uses a magical data type called tf32. It has the same numerical range as fp32 (8-bits), but instead
of 23 bits precision it has only 10 bits (same as fp16) and uses only 19 bits in total. It’s “magical” in the sense that
you can use the normal fp32 training and/or inference code and by enabling tf32 support you can get up to 3x throughput
improvement. All you need to do is to add the following to your code:
  Copied import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True
CUDA will automatically switch to using tf32 instead of fp32 where possible, assuming that the used GPU


TF32

 is from the Ampere series.
According to NVIDIA research, the
majority of machine learning training workloads show the same perplexity and convergence with tf32 training as with fp32.
If you’re already using fp16 or bf16 mixed precision it may help with the throughput as well.
You can enable this mode in the 🤗 Trainer:
  Copied TrainingArguments(tf32=True, **default_args)
tf32 can’t be accessed directly via tensor.to(dtype=torch.tf32) because it is an internal CUDA data type. You need torch>=1.7 to use tf32 data types.
For additional information on tf32 vs other precisions, please refer to the following benchmarks:
RTX-3090 and
A100


Flash Attention 2

You can speedup the training throughput by using Flash Attention 2 integration in transformers. Check out the appropriate section in the single GPU section to learn more about how to load a model with Flash Attention 2 modules.


Optimizer choice

The most common optimizer used to train transformer models is Adam or AdamW (Adam with weight decay). Adam achieves
good convergence by storing the rolling average of the previous gradients; however, it adds an additional memory
footprint of the order of the number of model parameters. To remedy this, you can use an alternative optimizer.
For example if you have NVIDIA/apex installed, adamw_apex_fused will give you the
fastest training experience among all supported AdamW optimizers.
Trainer integrates a variety of optimizers that can be used out of box: adamw_hf, adamw_torch, adamw_torch_fused,
adamw_apex_fused, adamw_anyprecision, adafactor, or adamw_bnb_8bit. M


Optimizer choice

ore optimizers can be plugged in via a third-party implementation.
Let’s take a closer look at two alternatives to AdamW optimizer:
adafactor which is available in Trainer adamw_bnb_8bit is also available in Trainer, but a third-party integration is provided below for demonstration.
For comparison, for a 3B-parameter model, like “t5-3b”:
A standard AdamW optimizer will need 24GB of GPU memory because it uses 8 bytes for each parameter (8*3 => 24GB) Adafactor optimizer will need more than 12GB. It uses slightly more than 4 bytes for each parameter, so 4*3 and then some extra. 8bit BNB quantized optimizer will use only (2*3) 6GB if all optimizer states are quantized


Adafactor

Adafactor doesn’t store rolling averages for each element in weight matrices. Instead, it keeps aggregated information
(sums of rolling averages row- and column-wise), significantly reducing its footprint. However, compared to Adam,
Adafactor may have slower convergence in certain cases.
You can switch to Adafactor by setting optim="adafactor" in TrainingArguments:
  Copied training_args = TrainingArguments(per_device_train_batch_size=4, optim="adafactor", **default_args)
Combined with other approaches (gradient accumulation, gradient checkpointing, and mixed precision training)
you can notice up to 3x improvement while maintaining the throughput! However, as mentioned before, the convergence of
Adafactor can be worse than Adam.


8-bit Adam

Instead of aggregating optimizer states like Adafactor, 8-bit Adam keeps the full state and quantizes it. Quantization
means that it stores the state with lower precision and dequantizes it only for the optimization. This is similar to the
idea behind mixed precision training.
To use adamw_bnb_8bit, you simply need to set optim="adamw_bnb_8bit" in TrainingArguments:
  Copied training_args = TrainingArguments(per_device_train_batch_size=4, optim="adamw_bnb_8bit", **default_args)
However, we can also use a third-party implementation of the 8-bit optimizer for demonstration purposes to see how that can be integrated.
First, follow the installation guide in the GitHub repo to install the bitsandbytes library
that implements the 8-bit Adam optimizer.
Next you need to initialize the optimizer. This involves two steps:
First, group the model’s paramet


8-bit Adam

ers into two groups - one where weight decay should be applied, and the other one where it should not. Usually, biases and layer norm parameters are not weight decayed. Then do some argument housekeeping to use the same parameters as the previously used AdamW optimizer.
  Copied import bitsandbytes as bnb
from torch import nn
from transformers.trainer_pt_utils import get_parameter_names
training_args = TrainingArguments(per_device_train_batch_size=4, **default_args)
decay_parameters = get_parameter_names(model, [nn.LayerNorm])
decay_parameters = [name for name in decay_parameters if "bias" not in name]
optimizer_grouped_parameters = [
    {
        "params": [p for n, p in model.named_parameters() if n in decay_parameters],
        "weight_decay": training_args.weight_decay,
    },
    {
        "params": [p for n, p in model.named_parameters()


8-bit Adam

 if n not in decay_parameters],
        "weight_decay": 0.0,
    },
]
optimizer_kwargs = {
    "betas": (training_args.adam_beta1, training_args.adam_beta2),
    "eps": training_args.adam_epsilon,
}
optimizer_kwargs["lr"] = training_args.learning_rate
adam_bnb_optim = bnb.optim.Adam8bit(
    optimizer_grouped_parameters,
    betas=(training_args.adam_beta1, training_args.adam_beta2),
    eps=training_args.adam_epsilon,
    lr=training_args.learning_rate,
)
Finally, pass the custom optimizer as an argument to the Trainer:
  Copied trainer = Trainer(model=model, args=training_args, train_dataset=ds, optimizers=(adam_bnb_optim, None))
Combined with other approaches (gradient accumulation, gradient checkpointing, and mixed precision training),
you can expect to get about a 3x memory improvement and even slightly higher throughput as using Adafactor


multi_tensor

pytorch-nightly introduced torch.optim._multi_tensor which should significantly speed up the optimizers for situations
with lots of small feature tensors. It should eventually become the default, but if you want to experiment with it sooner, take a look at this GitHub issue.


Data preloading

One of the important requirements to reach great training speed is the ability to feed the GPU at the maximum speed it
can handle. By default, everything happens in the main process, and it might not be able to read the data from disk fast
enough, and thus create a bottleneck, leading to GPU under-utilization. Configure the following arguments to reduce the bottleneck:
DataLoader(pin_memory=True, ...) - ensures the data gets preloaded into the pinned memory on CPU and typically leads to much faster transfers from CPU to GPU memory. DataLoader(num_workers=4, ...) - spawn several workers to preload data faster. During training, watch the GPU utilization stats; if it’s far from 100%, experiment with increasing the number of workers. Of course, the problem could be elsewhere, so many workers won’t necessarily lead to better performance.
When using Trainer, the corresponding TrainingArguments are: dataloader_pin_memory (True by default), and dataloader_num_workers (defaults to 0).


DeepSpeed ZeRO

DeepSpeed is an open-source deep learning optimization library that is integrated with 🤗 Transformers and 🤗 Accelerate.
It provides a wide range of features and optimizations designed to improve the efficiency and scalability of large-scale
deep learning training.
If your model fits onto a single GPU and you have enough space to fit a small batch size, you don’t need to use DeepSpeed
as it’ll only slow things down. However, if the model doesn’t fit onto a single GPU or you can’t fit a small batch, you can
leverage DeepSpee


DeepSpeed ZeRO

d ZeRO + CPU Offload, or NVMe Offload for much larger models. In this case, you need to separately
install the library, then follow one of the guides to create a configuration file
and launch DeepSpeed:
For an in-depth guide on DeepSpeed integration with Trainer, review the corresponding documentation, specifically the
section for a single GPU. Some adjustments are required to use DeepSpeed in a notebook; please take a look at the corresponding guide. If you prefer to use 🤗 Accelerate, refer to 🤗 Accelerate DeepSpeed guide


Using torch.compile

PyTorch 2.0 introduced a new compile function that doesn’t require any modification to existing PyTorch code but can
optimize your code by adding a single line of code: model = torch.compile(model).
If using Trainer, you only need to pass the torch_compile option in the TrainingArguments:
  Copied training_args = TrainingArguments(torch_compile=True, **default_args)
torch.compile uses Python’s frame evaluation API to automatically create a graph from existing PyTorch programs. After
capturing the graph, different backends can be deployed to lower the graph to an optimized engine.
You can find more details and benchmarks in PyTorch documentation.
torch.compile has a growing list of backends, which can be found in by calling torc


Using torch.compile

hdynamo.list_backends(), each of which with its optional dependencies.
Choose which backend to use by specifying it via torch_compile_backend in the TrainingArguments.  Some of the most commonly used backends are:
Debugging backends:
dynamo.optimize("eager") - Uses PyTorch to run the extracted GraphModule. This is quite useful in debugging TorchDynamo issues. dynamo.optimize("aot_eager") - Uses AotAutograd with no compiler, i.e, just using PyTorch eager for the AotAutograd’s extracted forward and backward graphs. This is useful for debugging, and unlikely to give speedups.
Training & inference backends:
dynamo.optimize("inductor") - Uses TorchInductor backend with AotAutograd and cudagraphs by leveraging codegened Triton kernel


Using torch.compile

s  Read more dynamo.optimize("nvfuser") -  nvFuser with TorchScript. Read more dynamo.optimize("aot_nvfuser") -  nvFuser with AotAutograd. Read more dynamo.optimize("aot_cudagraphs") - cudagraphs with AotAutograd. Read more
Inference-only backends:
dynamo.optimize("ofi") -  Uses Torchscript optimize_for_inference.  Read more dynamo.optimize("fx2trt") -  Uses Nvidia TensorRT for inference optimizations.  Read more dynamo.optimize("onnxrt") -  Uses ONNXRT for inference on CPU/GPU.  Read more dynamo.optimize("ipex") -  Uses IPEX for inference on CPU.  Read more
For an example of using torch.compile with 🤗 Transformers, check out this blog post on fine-tuning a BERT model for Text Classification using the newest PyTorch 2.0 feature


Using 🤗 Accelerate

With 🤗 Accelerate you can use the above methods while gaining full
control over the training loop and can essentially write the loop in pure PyTorch with some minor modifications.
Suppose you have combined the methods in the TrainingArguments like so:
  Copied training_args = TrainingArguments(
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    gradient_checkpointing=True,
    fp16=True,
    **default_args,
)
The full example training loop with 🤗 Accelerate is only a handful of lines of code long:
  Copied from accelerate import Accelerator
from torch.utils.data.dataloader import DataLoader
dataloader = DataLoader(ds, batch_size=training_args.per_device_train_batch_size)
if training_args.gradient_checkpoint


Using 🤗 Accelerate

ing:
    model.gradient_checkpointing_enable()
accelerator = Accelerator(fp16=training_args.fp16)
model, optimizer, dataloader = accelerator.prepare(model, adam_bnb_optim, dataloader)
model.train()
for step, batch in enumerate(dataloader, start=1):
    loss = model(**batch).loss
    loss = loss / training_args.gradient_accumulation_steps
    accelerator.backward(loss)
    if step % training_args.gradient_accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()
First we wrap the dataset in a DataLoader.
Then we can enable gradient checkpointing by calling the model’s gradient_checkpointing_enable() method.
When we initialize the Accelerator
we can specify if we want to use mixed precision training and it wil


Using 🤗 Accelerate

l take care of it for us in the prepare call.
During the prepare
call the dataloader will also be distributed across workers should we use multiple GPUs. We use the same 8-bit optimizer from the earlier example.
Finally, we can add the main training loop. Note that the backward call is handled by 🤗 Accelerate. We can also see
how gradient accumulation works: we normalize the loss, so we get the average at the end of accumulation and once we have
enough steps we run the optimization.
Implementing these optimization techniques with 🤗 Accelerate only takes a handful of lines of code and comes with the
benefit of more flexibility in the training loop. For a full documentation of all features have a look at the
Accelerate documentatio


Efficient Software Prebuilds

PyTorch’s pip and conda builds come prebuilt with the cuda toolkit
which is enough to run PyTorch, but it is insufficient if you need to build cuda extensions.
At times, additional efforts may be required to pre-build some components. For instance, if you’re using libraries like apex that
don’t come pre-compiled. In other situations figuring out how to install the right cuda toolkit system-wide can be complicated.
To address these scenarios PyTorch and NVIDIA released a new version of NGC docker container which already comes with
everything prebuilt. You just need to install your programs on it, and it will run


Efficient Software Prebuilds

out of the box.
This approach is also useful if you want to tweak the pytorch source and/or make a new customized build.
To find the docker image version you want start with PyTorch release notes,
choose one of the latest monthly releases. Go into the release’s notes for the desired release, check that the environment’s
components are matching your needs (including NVIDIA Driver requirements!) and then at the very top of that document go
to the corresponding NGC page. If for some reason you get lost, here is the index of all PyTorch NGC images.
Next follow the instructions to download and deploy the docker image


Mixture of Experts

Some recent papers reported a 4-5x training speedup and a faster inference by integrating
Mixture of Experts (MoE) into the Transformer models.
Since it has been discovered that more parameters lead to better performance, this technique allows to increase the
number of parameters by an order of magnitude without increasing training costs.
In this approach every other FFN layer is replaced with a MoE Layer which consists of many experts, with a gated function
that trains each expert in a balanced way depending on the input token’s position in a sequence.
(source: GLAM)
You can find exhaustive details and comparison tables in the papers listed at the end of this section.
The main drawback of this approach is that it requires staggering amounts of GPU memory - almost an order of magnitude
larger than its dense equivalent. Various distillation and approaches are proposed to how to overcome the muc


Mixture of Experts

h higher memory requirements.
There is direct trade-off though, you can use just a few experts with a 2-3x smaller base model instead of dozens or
hundreds experts leading to a 5x smaller model and thus increase the training speed moderately while increasing the
memory requirements moderately as well.
Most related papers and implementations are built around Tensorflow/TPUs:
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity GLaM: Generalist Language Model (GLaM)
And for Pytorch DeepSpeed has built one as well: DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale, Mixture of Experts - blog posts:  1, 2 and specific deployment with large transformer-based natural language generation models: blog post, Megatron-Deepspeed branch.


Using PyTorch native attention and Flash Attention

PyTorch 2.0 released a native torch.nn.functional.scaled_dot_product_attention (SDPA),
that allows using fused GPU kernels such as memory-efficient attention and flash attention.
After installing the optimum package, the relevant internal modules can be
replaced to use PyTorch’s native attention with:
  Copied model = model.to_bettertransformer()
Once converted, train the model as usual.
The PyTorch-native scaled_dot_product_attention operator can only dispatch to Flash Attention if no attention_mask is provided. By default, in training mode, the BetterTransformer integration drops the mask support and can only be used for training that does not require a padding mask for batched training. This is the case, for example, during masked language modeling or causal language modeling. BetterTransformer is not suited for fine-tuning models on tasks that require a padding mask.
Check out this blogpost to learn more about acceleration and memory-savings with SDPA.


Use tokenizers from 🤗 Tokenizers

The PreTrainedTokenizerFast depends on the 🤗 Tokenizers library. The tokenizers obtained from the 🤗 Tokenizers library can be
loaded very simply into 🤗 Transformers.
Before getting in the specifics, let’s first start by creating a dummy tokenizer in a few lines:
  Copied >>> from tokenizers import Tokenizer
>>> from tokenizers.models import BPE
>>> from tokenizers.trainers import BpeTrainer
>>> from tokenizers.pre_tokenizers import Whitespace
>>> tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
>>> trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])
>>> tokenizer.pre_tokenizer = Whitespace()
>>> files = [...]
>>> tokenizer.train(files, trainer)
We now have a tokenizer trained on the files we defined. We can either continue using it in that runtime, or save it to
a JSON file for future re-use.


Loading directly from the tokenizer object

Let’s see how to leverage this tokenizer object in the 🤗 Transformers library. The
PreTrainedTokenizerFast class allows for easy instantiation, by accepting the instantiated
tokenizer object as an argument:
  Copied >>> from transformers import PreTrainedTokenizerFast
>>> fast_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer)
This object can now be used with all the methods shared by the 🤗 Transformers tokenizers! Head to the tokenizer
page for more information.


Loading from a JSON file

In order to load a tokenizer from a JSON file, let’s first start by saving our tokenizer:
  Copied >>> tokenizer.save("tokenizer.json")
The path to which we saved this file can be passed to the PreTrainedTokenizerFast initialization
method using the tokenizer_file parameter:
  Copied >>> from transformers import PreTrainedTokenizerFast
>>> fast_tokenizer = PreTrainedTokenizerFast(tokenizer_file="tokenizer.json")
This object can now be used with all the methods shared by the 🤗 Transformers tokenizers! Head to the tokenizer
page for more information.


Hyperparameter Search using Trainer API

🤗 Transformers provides a Trainer class optimized for training 🤗 Transformers models, making it easier to start training without manually writing your own training loop. The Trainer provides API for hyperparameter search. This doc shows how to enable it in example.


Hyperparameter Search backend

Trainer supports four hyperparameter search backends currently:
optuna, sigopt, raytune and wandb.
you should install them before using them as the hyperparameter search backend
  Copied pip install optuna/sigopt/wandb/ray[tune]


How to enable Hyperparameter search in example

Define the hyperparameter search space, different backends need different format.
For sigopt, see sigopt object_parameter, it’s like following:
  Copied >>> def sigopt_hp_space(trial):
...     return [
...         {"bounds": {"min": 1e-6, "max": 1e-4}, "name": "learning_rate", "type": "double"},
...         {
...             "categorical_values": ["16", "32", "64", "128"],
...             "name": "per_device_train_batch_size",
...             "type": "categorical",
...         },
...     ]
For optuna, see optuna object_parameter, it’s like following:
  Copied >>> def optuna_hp_space(trial):
...     return {
...         "learning_rate": trial.suggest_float("learning_rate", 1e-6, 1e-4, log=True),
...         "per_device_train_batch_size": trial.suggest_categorical("per_device_train_batch_size", [16, 32, 64, 128]),
...     }
Optuna provides multi-objective HPO. You can pass direction in hyperparameter_se


arch and define your own compute_objective to return multiple objective values. The Pareto Front (List[BestRun]) will be returned in hyperparameter_search, you should refer to the test case TrainerHyperParameterMultiObjectOptunaIntegrationTest in test_trainer. It’s like following
  Copied >>> best_trials = trainer.hyperparameter_search(
...     direction=["minimize", "maximize"],
...     backend="optuna",
...     hp_space=optuna_hp_space,
...     n_trials=20,
...     compute_objective=compute_objective,
... )
For raytune, see raytune object_parameter, it’s like following:
  Copied >>> def ray_hp_space(trial):
...     return {
...         "learning_rate": tune.loguniform(1e-6, 1e-4),
...         "per_device_train_batch_size": tune.choice([16, 32, 64, 128]),
...     }
For wandb, see wandb object_parameter, it’s like following:
  Copied >>> def wandb_hp_space(trial):
...     return {
...         "method"


: "random",
...         "metric": {"name": "objective", "goal": "minimize"},
...         "parameters": {
...             "learning_rate": {"distribution": "uniform", "min": 1e-6, "max": 1e-4},
...             "per_device_train_batch_size": {"values": [16, 32, 64, 128]},
...         },
...     }
Define a model_init function and pass it to the Trainer, as an example:
  Copied >>> def model_init(trial):
...     return AutoModelForSequenceClassification.from_pretrained(
...         model_args.model_name_or_path,
...         from_tf=bool(".ckpt" in model_args.model_name_or_path),
...         config=config,
...         cache_dir=model_args.cache_dir,
...         revision=model_args.model_revision,
...         use_auth_token=True if model_args.use_auth_token else None,
...     )
Create a Trainer with your model_init function, training arguments, training and test datasets, and evaluation function:
  Copied >


>> trainer = Trainer(
...     model=None,
...     args=training_args,
...     train_dataset=small_train_dataset,
...     eval_dataset=small_eval_dataset,
...     compute_metrics=compute_metrics,
...     tokenizer=tokenizer,
...     model_init=model_init,
...     data_collator=data_collator,
... )
Call hyperparameter search, get the best trial parameters, backend could be "optuna"/"sigopt"/"wandb"/"ray". direction can be"minimize" or "maximize", which indicates whether to optimize greater or lower objective.
You could define your own compute_objective function, if not defined, the default compute_objective will be called, and the sum of eval metric like f1 is returned as objective value.
  Copied >>> best_trial = trainer.hyperparameter_search(
...     direction="maximize",
...     backend="optuna",
...     hp_space=optuna_hp_space,
...     n_trials=20,
...     compute_objective=compute_objective,
... )


Hyperparameter search For DDP finetune

Currently, Hyperparameter search for DDP is enabled for optuna and sigopt. Only the rank-zero process will generate the search trial and pass the argument to other ranks.


Custom hardware for training

The hardware you use to run model training and inference can have a big effect on performance. For a deep dive into GPUs make sure to check out Tim Dettmer’s excellent blog post.
Let’s have a look at some practical advice for GPU setups.


GPU

When you train bigger models you have essentially three options:
bigger GPUs more GPUs more CPU and NVMe (offloaded to by DeepSpeed-Infinity)
Let’s start at the case where you have a single GPU.


Power and Cooling

If you bought an expensive high end GPU make sure you give it the correct power and sufficient cooling.
Power:
Some high end consumer GPU cards have 2 and sometimes 3 PCI-E 8-Pin power sockets. Make sure you have as many independent 12V PCI-E 8-Pin cables plugged into the card as there are sockets. Do not use the 2 splits at one end of the same cable (also known as pigtail cable). That is if you have 2 sockets on the GPU, you want 2 PCI-E 8-Pin cables going from your PSU to the card and not one that has 2 PCI-E 8-Pin connectors at the end! You won’t get the full performance out of your card otherwise.
Each PCI-E 8-Pin power cable needs to be plugged into a 12V rail on the PSU side and can supply up to 150W of power.
Some other cards may use a PCI-E 12-Pin connectors, and these can deliver up to 500-600W of power.
Low end cards may use 6-Pin connectors, which supply


Power and Cooling

up to 75W of power.
Additionally you want the high-end PSU that has stable voltage. Some lower quality ones may not give the card the stable voltage it needs to function at its peak.
And of course the PSU needs to have enough unused Watts to power the card.
Cooling:
When a GPU gets overheated it will start throttling down and will not deliver full performance and it can even shutdown if it gets too hot.
It’s hard to tell the exact best temperature to strive for when a GPU is heavily loaded, but probably anything under +80C is good, but lower is better - perhaps 70-75C is an excellent range to be in. The throttling down is likely to start at around 84-90C. But other than throttling performance a prolonged very high temperature is likely to reduce the lifespan of a GPU.
Next let’s have a look at one of the most important aspects when having multiple GPUs: connectivity


Multi-GPU Connectivity

If you use multiple GPUs the way cards are inter-connected can have a huge impact on the total training time. If the GPUs are on the same physical node, you can run:
  Copied nvidia-smi topo -m
and it will tell you how the GPUs are inter-connected. On a machine with dual-GPU and which are connected with NVLink, you will most likely see something like:
  Copied         GPU0    GPU1    CPU Affinity    NUMA Affinity
GPU0     X      NV2     0-23            N/A
GPU1    NV2      X      0-23            N/A
on a different machine w/o NVLink we may see:
  Copied         GPU0    GPU1    CPU Affinity    NUMA Affinity
GPU0     X      PHB     0-11            N/A
GPU1    PHB      X      0-11            N/A
The report includes this legend:
  Copied   X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between


Multi-GPU Connectivity

 NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks
So the first report NV2 tells us the GPUs are interconnected with 2 NVLinks, and the second report PHB we have a typical consumer-level PCIe+Bridge setup.
Check what type of connectivity you have on your setup. Some of these will make the communication between cards faster (e.g. NVLink), others slower (e.g. PHB).
Depending on the type of scalability solution used, the connectivity speed could ha


Multi-GPU Connectivity

ve a major or a minor impact. If the GPUs need to sync rarely, as in DDP, the impact of a slower connection will be less significant. If the GPUs need to send messages to each other often, as in ZeRO-DP, then faster connectivity becomes super important to achieve faster training.
 NVlink
NVLink is a wire-based serial multi-lane near-range communications link developed by Nvidia.
Each new generation provides a faster bandwidth, e.g. here is a quote from Nvidia Ampere GA102 GPU Architecture:
Third-Generation NVLink®
GA102 GPUs utilize NVIDIA’s third-generation NVLink interface, which includes four x4 links,
with each link providing 14.0625 GB/sec bandwidth in each direction between two GPUs. Four
links provide 56.25 GB/sec bandwidth in each direction, and 112.5 GB/sec total bandwidth
between two GPUs. Two RTX 3090 GPUs can


Multi-GPU Connectivity

be connected together for SLI using NVLink.
(Note that 3-Way and 4-Way SLI configurations are not supported.)
So the higher X you get in the report of NVX in the output of nvidia-smi topo -m the better. The generation will depend on your GPU architecture.
Let’s compare the execution of a gpt2 language model training over a small sample of wikitext.
The results are:
| NVlink   | Time   |
|----------|--------|
| Y        | 101s   |
| N        | 131s   |
You can see that NVLink completes the training ~23% faster. In the second benchmark we use NCCL_P2P_DISABLE=1 to tell the GPUs not to use NVLink.
Here is the full benchmark code and outputs:
  Copied # DDP w/ NVLink
rm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch \
--nproc_per_node 2 examples/pytorch/language-modeling/run_clm.py --model_name_


Multi-GPU Connectivity

or_path gpt2 \
--dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --do_train \
--output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200
{'train_runtime': 101.9003, 'train_samples_per_second': 1.963, 'epoch': 0.69}
# DDP w/o NVLink
rm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 NCCL_P2P_DISABLE=1 python -m torch.distributed.launch \
--nproc_per_node 2 examples/pytorch/language-modeling/run_clm.py --model_name_or_path gpt2 \
--dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --do_train
--output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200
{'train_runtime': 131.4367, 'train_samples_per_second': 1.522, 'epoch': 0.69}
Hardware: 2x TITAN RTX 24GB each + NVlink with 2 NVLinks (NV2 in nvidia-smi topo -m)
Software: pytorch-1.8-to-be + cuda-11.0 / transformers==4.3.0.d


Multi-GPU Network Issues Debug

When training or inferencing with DistributedDataParallel and multiple GPU, if you run into issue of inter-communication between processes and/or nodes, you can use the following script to diagnose network issues.
  Copied wget https://raw.githubusercontent.com/huggingface/transformers/main/scripts/distributed/torch-distributed-gpu-test.py
For example to test how 2 GPUs interact do:
  Copied python -m torch.distributed.run --nproc_per_node 2 --nnodes 1 torch-distributed-gpu-test.py
If both processes can talk to each and allocate GPU memory each will print an OK status.
For more GPUs or nod


Multi-GPU Network Issues Debug

es adjust the arguments in the script.
You will find a lot more details inside the diagnostics script and even a recipe to how you could run it in a SLURM environment.
An additional level of debug is to add NCCL_DEBUG=INFO environment variable as follows:
  Copied NCCL_DEBUG=INFO python -m torch.distributed.run --nproc_per_node 2 --nnodes 1 torch-distributed-gpu-test.py
This will dump a lot of NCCL-related debug information, which you can then search online if you find that some problems are reported. Or if you’re not sure how to interpret the output you can share the log file in an Issue.


Underflow and Overflow Detection

This feature is currently available for PyTorch-only.
For multi-GPU training it requires DDP (torch.distributed.launch).
This feature can be used with any nn.Module-based model.
If you start getting loss=NaN or the model inhibits some other abnormal behavior due to inf or nan in
activations or weights one needs to discover where the first underflow or overflow happens and what led to it. Luckily
you can accomplish that easily by activating a special module that will do the detection automatically.
If you’re using Trainer, you just need to add:
  Copied --debug underflow_overflow
to the normal command line arguments, or pass debug="underflow_overflow" when creating the
TrainingArguments object.
If you’re using your own training loop or another Trainer you can accomplish the same with:
  Copied from transformers.debug_utils import DebugUnderflowOverflow
debug_overflow = DebugUnderflowOverflow(model)
DebugUnderflowOverflow inserts hooks into the model that immediately after each
for


Underflow and Overflow Detection

ward call will test input and output variables and also the corresponding module’s weights. As soon as inf or
nan is detected in at least one element of the activations or weights, the program will assert and print a report
like this (this was caught with google/mt5-small under fp16 mixed precision):
  Copied Detected inf/nan during batch_number=0
Last 21 forward frames:
abs min  abs max  metadata
                  encoder.block.1.layer.1.DenseReluDense.dropout Dropout
0.00e+00 2.57e+02 input[0]
0.00e+00 2.85e+02 output
[...]
                  encoder.block.2.layer.0 T5LayerSelfAttention
6.78e-04 3.15e+03 input[0]
2.65e-04 3.42e+03 output[0]
             None output[1]
2.25e-01 1.00e+04 output[2]
                  encoder.block.2.layer.1.layer_norm T5LayerNorm
8.69e-02 4.18e-01 weight
2.65e-04 3.42e+03 input[0]
1.79e-06 4.65e+00 output
                  encoder.block.2.layer.1.DenseReluDense.wi_0 Linear
2.17e-07 4.50e+00 weight
1.79e-06 4.65e+00 input[0]
2.68e-06 3.70e+01 output


Underflow and Overflow Detection

                  encoder.block.2.layer.1.DenseReluDense.wi_1 Linear
8.08e-07 2.66e+01 weight
1.79e-06 4.65e+00 input[0]
1.27e-04 2.37e+02 output
                  encoder.block.2.layer.1.DenseReluDense.dropout Dropout
0.00e+00 8.76e+03 input[0]
0.00e+00 9.74e+03 output
                  encoder.block.2.layer.1.DenseReluDense.wo Linear
1.01e-06 6.44e+00 weight
0.00e+00 9.74e+03 input[0]
3.18e-04 6.27e+04 output
                  encoder.block.2.layer.1.DenseReluDense T5DenseGatedGeluDense
1.79e-06 4.65e+00 input[0]
3.18e-04 6.27e+04 output
                  encoder.block.2.layer.1.dropout Dropout
3.18e-04 6.27e+04 input[0]
0.00e+00      inf output
The example output has been trimmed in the middle for brevity.
The second column shows the value of the absolute largest element, so if you have a closer look at the last few frames,
the inputs and outputs were in the range of 1e4. So when this training was done under fp16 mixed precision the very
last step overflowed (since under fp16


Underflow and Overflow Detection

the largest number before inf is 64e3). To avoid overflows under
fp16 the activations must remain way below 1e4, because 1e4 * 1e4 = 1e8 so any matrix multiplication with
large activations is going to lead to a numerical overflow condition.
At the very start of the trace you can discover at which batch number the problem occurred (here Detected inf/nan during batch_number=0 means the problem occurred on the first batch).
Each reported frame starts by declaring the fully qualified entry for the corresponding module this frame is reporting
for. If we look just at this frame:
  Copied                   encoder.block.2.layer.1.layer_norm T5LayerNorm
8.69e-02 4.18e-01 weight
2.65e-04 3.42e+03 input[0]
1.79e-06 4.65e+00 output
Here, encoder.block.2.layer.1.layer_norm indicates that it was a layer norm for the first layer, of the second
block of the encoder. And the specific calls of the forward is T5LayerNorm.
Let’s look at the last few frames of that report:
  Copied Detected inf/nan


Underflow and Overflow Detection

during batch_number=0
Last 21 forward frames:
abs min  abs max  metadata
[...]
                  encoder.block.2.layer.1.DenseReluDense.wi_0 Linear
2.17e-07 4.50e+00 weight
1.79e-06 4.65e+00 input[0]
2.68e-06 3.70e+01 output
                  encoder.block.2.layer.1.DenseReluDense.wi_1 Linear
8.08e-07 2.66e+01 weight
1.79e-06 4.65e+00 input[0]
1.27e-04 2.37e+02 output
                  encoder.block.2.layer.1.DenseReluDense.wo Linear
1.01e-06 6.44e+00 weight
0.00e+00 9.74e+03 input[0]
3.18e-04 6.27e+04 output
                  encoder.block.2.layer.1.DenseReluDense T5DenseGatedGeluDense
1.79e-06 4.65e+00 input[0]
3.18e-04 6.27e+04 output
                  encoder.block.2.layer.1.dropout Dropout
3.18e-04 6.27e+04 input[0]
0.00e+00      inf output
The last frame reports for Dropout.forward function with the first entry for the only input and the second for the
only output. You can see that it was called from an attribute dropout inside DenseReluDense class. We can see
that it happe


Underflow and Overflow Detection

ned during the first layer, of the 2nd block, during the very first batch. Finally, the absolute largest
input elements was 6.27e+04 and same for the output was inf.
You can see here, that T5DenseGatedGeluDense.forward resulted in output activations, whose absolute max value was
around 62.7K, which is very close to fp16’s top limit of 64K. In the next frame we have Dropout which renormalizes
the weights, after it zeroed some of the elements, which pushes the absolute max value to more than 64K, and we get an
overflow (inf).
As you can see it’s the previous frames that we need to look into when the numbers start going into very large for fp16
numbers.
Let’s match the report to the code from models/t5/modeling_t5.py:
  Copied class T5DenseGatedGeluDense(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.wi_0 = nn.Linear(config.d_model, config.d_ff, bias=False)
        self.wi_1 = nn.Linear(config.d_model, config.d_ff, bias=False)
        self.wo = n


Underflow and Overflow Detection

n.Linear(config.d_ff, config.d_model, bias=False)
        self.dropout = nn.Dropout(config.dropout_rate)
        self.gelu_act = ACT2FN["gelu_new"]
    def forward(self, hidden_states):
        hidden_gelu = self.gelu_act(self.wi_0(hidden_states))
        hidden_linear = self.wi_1(hidden_states)
        hidden_states = hidden_gelu * hidden_linear
        hidden_states = self.dropout(hidden_states)
        hidden_states = self.wo(hidden_states)
        return hidden_states
Now it’s easy to see the dropout call, and all the previous calls as well.
Since the detection is happening in a forward hook, these reports are printed immediately after each forward
returns.
Going back to the full report, to act on it and to fix the problem, we need to go a few frames up where the numbers
started to go up and most likely switch to the fp32 mode here, so that the numbers don’t overflow when multiplied
or summed up. Of course, there might be other solutions. For example, we could turn off amp te


Underflow and Overflow Detection

mporarily if it’s
enabled, after moving the original forward into a helper wrapper, like so:
  Copied def _forward(self, hidden_states):
    hidden_gelu = self.gelu_act(self.wi_0(hidden_states))
    hidden_linear = self.wi_1(hidden_states)
    hidden_states = hidden_gelu * hidden_linear
    hidden_states = self.dropout(hidden_states)
    hidden_states = self.wo(hidden_states)
    return hidden_states
import torch
def forward(self, hidden_states):
    if torch.is_autocast_enabled():
        with torch.cuda.amp.autocast(enabled=False):
            return self._forward(hidden_states)
    else:
        return self._forward(hidden_states)
Since the automatic detector only reports on inputs and outputs of full frames, once you know where to look, you may
want to analyse the intermediary stages of any specific forward function as well. In such a case you can use the
detect_overflow helper function to inject the detector where you want it, for example:
  Copied from debug_utils import de


Underflow and Overflow Detection

tect_overflow
class T5LayerFF(nn.Module):
    [...]
    def forward(self, hidden_states):
        forwarded_states = self.layer_norm(hidden_states)
        detect_overflow(forwarded_states, "after layer_norm")
        forwarded_states = self.DenseReluDense(forwarded_states)
        detect_overflow(forwarded_states, "after DenseReluDense")
        return hidden_states + self.dropout(forwarded_states)
You can see that we added 2 of these and now we track if inf or nan for forwarded_states was detected
somewhere in between.
Actually, the detector already reports these because each of the calls in the example above is a nn.Module, but
let’s say if you had some local direct calculations this is how you’d do that.
Additionally, if you’re instantiating the debugger in your own code, you can adjust the number of frames printed from
its default, e.g.:
  Copied from transformers.debug_utils import DebugUnderflowOverflow
debug_overflow = DebugUnderflowOverflow(model, max_frames_to_save=100)


Specific batch absolute min and max value tracing

The same debugging class can be used for per-batch tracing with the underflow/overflow detection feature turned off.
Let’s say you want to watch the absolute min and max values for all the ingredients of each forward call of a given
batch, and only do that for batches 1 and 3. Then you instantiate this class as:
  Copied debug_overflow = DebugUnderflowOverflow(model, trace_batch_nums=[1, 3])
And now full batches 1 and 3 will be traced using the same format as the underflow/overflow detector does.
Batches are 0-indexed.
This is helpful if you know that the program starts misbehaving after a certain batch number, so you can fast-forward
right to that area. Here is a


 sample truncated output for such configuration:
  Copied                   *** Starting batch number=1 ***
abs min  abs max  metadata
                  shared Embedding
1.01e-06 7.92e+02 weight
0.00e+00 2.47e+04 input[0]
5.36e-05 7.92e+02 output
[...]
                  decoder.dropout Dropout
1.60e-07 2.27e+01 input[0]
0.00e+00 2.52e+01 output
                  decoder T5Stack
     not a tensor output
                  lm_head Linear
1.01e-06 7.92e+02 weight
0.00e+00 1.11e+00 input[0]
6.06e-02 8.39e+01 output
                   T5ForConditionalGeneration
     not a tensor output
                  *** Starting batch number=3 ***
abs min  abs max  metadata


          shared Embedding
1.01e-06 7.92e+02 weight
0.00e+00 2.78e+04 input[0]
5.36e-05 7.92e+02 output
[...]
Here you will get a huge number of frames dumped - as many as there were forward calls in your model, so it may or may
not what you want, but sometimes it can be easier to use for debugging purposes than a normal debugger. For example, if
a problem starts happening at batch number 150. So you can dump traces for batches 149 and 150 and compare where
numbers started to diverge.
You can also specify the batch number after which to stop the training, with:
  Copied debug_overflow = DebugUnderflowOverflow(model, trace_batch_nums=[1, 3], abort_after_batch_num=3


How to add a model to 🤗 Transformers?

The 🤗 Transformers library is often able to offer new models thanks to community contributors. But this can be a challenging project and requires an in-depth knowledge of the 🤗 Transformers library and the model to implement. At Hugging Face, we’re trying to empower more of the community to actively add models and we’ve put together this guide to walk you through the process of adding a PyTorch model (make sure you have PyTorch installed).
If you’re interested in implementing a TensorFlow model, take a look at the How to convert a 🤗 Transformers model to TensorFlow guide!
Along the way, you’ll:
get insights into open-source best practices understand the design principles behind o


How to add a model to 🤗 Transformers?

ne of the most popular deep learning libraries learn how to efficiently test large models learn how to integrate Python utilities like black, ruff, and make fix-copies to ensure clean and readable code
A Hugging Face team member will be available to help you along the way so you’ll never be alone. 🤗 ❤
To get started, open a New model addition issue for the model you want to see in 🤗 Transformers. If you’re not especially picky about contributing a specific model, you can filter by the New model label to see if there are any unclaimed model requests and work on it.
Once you’ve opened a new model request, the first step is to get familiar with 🤗 Transformers if you aren’t already!


General overview of 🤗 Transformers

First, you should get a general overview of 🤗 Transformers. 🤗 Transformers is a very opinionated library, so there is a
chance that you don’t agree with some of the library’s philosophies or design choices. From our experience, however, we
found that the fundamental design choices and philosophies of the library are crucial to efficiently scale 🤗
Transformers while keeping maintenance costs at a reasonable level.
A good first starting point to better understand the library is to read the documentation of our philosophy. As a result of our way of working, there are some choices that we try to apply to all models:
Composition is generally favored over-abstraction Duplicati


General overview of 🤗 Transformers

ng code is not always bad if it strongly improves the readability or accessibility of a model Model files are as self-contained as possible so that when you read the code of a specific model, you ideally only
have to look into the respective modeling_....py file.
In our opinion, the library’s code is not just a means to provide a product, e.g. the ability to use BERT for
inference, but also as the very product that we want to improve. Hence, when adding a model, the user is not only the
person who will use your model, but also everybody who will read, try to understand, and possibly tweak your code.
With this in mind, let’s go a bit deeper into the general library design


Overview of models

To successfully add a model, it is important to understand the interaction between your model and its config,
PreTrainedModel, and PretrainedConfig. For exemplary purposes, we will
call the model to be added to 🤗 Transformers BrandNewBert.
Let’s take a look:
As you can see, we do make use of inheritance in 🤗 Transformers, but we keep the level of abstraction to an absolute
minimum. There are never more than two levels of abstraction for any model in the library. BrandNewBertModel
inherits from BrandNewBertPreTrainedModel which in turn inherits from PreTrainedModel and
that’s it. As a general rule, we want to make sure that a new model only depends on
PreTrainedModel. The important fu


Overview of models

nctionalities that are automatically provided to every new
model are from_pretrained() and
save_pretrained(), which are used for serialization and deserialization. All of the
other important functionalities, such as BrandNewBertModel.forward should be completely defined in the new
modeling_brand_new_bert.py script. Next, we want to make sure that a model with a specific head layer, such as
BrandNewBertForMaskedLM does not inherit from BrandNewBertModel, but rather uses BrandNewBertModel
as a component that can be called in its forward pass to keep the level of abstraction low. Every new model requires a
configuration class, called BrandNewBertConfig. This configuration is always stor


Overview of models

ed as an attribute in
PreTrainedModel, and thus can be accessed via the config attribute for all classes
inheriting from BrandNewBertPreTrainedModel:
  Copied model = BrandNewBertModel.from_pretrained("brandy/brand_new_bert")
model.config  # model has access to its config
Similar to the model, the configuration inherits basic serialization and deserialization functionalities from
PretrainedConfig. Note that the configuration and the model are always serialized into two
different formats - the model to a pytorch_model.bin file and the configuration to a config.json file. Calling
save_pretrained() will automatically call
save_pretrained(), so that both model and configuration are saved


Code style

When coding your new model, keep in mind that Transformers is an opinionated library and we have a few quirks of our
own regarding how code should be written :-)
The forward pass of your model should be fully written in the modeling file while being fully independent of other
models in the library. If you want to reuse a block from another model, copy the code and paste it with a
# Copied from comment on top (see here
for a good example and there for more documentation on Copied from). The code should be fully understandable, even by a non-native English speaker. This means you should pic


Code style

k
descriptive variable names and avoid abbreviations. As an example, activation is preferred to act.
One-letter variable names are strongly discouraged unless it’s an index in a for loop. More generally we prefer longer explicit code to short magical one. Avoid subclassing nn.Sequential in PyTorch but subclass nn.Module and write the forward pass, so that anyone
using your code can quickly debug it by adding print statements or breaking points. Your function signature should be type-annotated. For the rest, good variable names are way more readable and
understandable than type annotations


Step-by-step recipe to add a model to 🤗 Transformers

Everyone has different preferences of how to port a model so it can be very helpful for you to take a look at summaries
of how other contributors ported models to Hugging Face. Here is a list of community blog posts on how to port a model:
Porting GPT2 Model by Thomas Porting WMT19 MT Model by Stas
From experience, we can tell you that the most important things to keep in mind when adding a model are:
Don’t reinvent the wheel! Most parts of the code you will add for the new 🤗 Transformers model already exist
somewhere in 🤗 Transformers. Take some time to find similar, already existing models and tokenizers you can copy
from. grep and rg are your
friends. Note that it might very well happen that your model’s tokenizer is based on one model implementation, and
your model’s modeling code on another one. E.g. FSMT’s modeling code is based on BART, while FSMT’s tokenizer code
is based on XLM. It’s more of an engineering challenge than


a scientific challenge. You should spend more time creating an
efficient debugging environment rather than trying to understand all theoretical aspects of the model in the paper. Ask for help, when you’re stuck! Models are the core component of 🤗 Transformers so we at Hugging Face are more
than happy to help you at every step to add your model. Don’t hesitate to ask if you notice you are not making
progress.
In the following, we try to give you a general recipe that we found most useful when porting a model to 🤗 Transformers.
The following list is a summary of everything that has to be done to add a model and can be used by you as a To-Do
List:
☐ (Optional) Understood the model’s theoretical aspects
☐ Prepared 🤗 Transformers dev environment
☐ Set up debugging environment of the original repository
☐ Created script that successfully runs the forward() pass using the original repository and checkpoint
☐ Successfully added the model


skeleton to 🤗 Transformers
☐ Successfully converted original checkpoint to 🤗 Transformers checkpoint
☐ Successfully ran forward() pass in 🤗 Transformers that gives identical output to original checkpoint
☐ Finished model tests in 🤗 Transformers
☐ Successfully added tokenizer in 🤗 Transformers
☐ Run end-to-end integration tests
☐ Finished docs
☐ Uploaded model weights to the Hub
☐ Submitted the pull request
☐ (Optional) Added a demo notebook
To begin with, we usually recommend starting by getting a good theoretical understanding of BrandNewBert. However,
if you prefer to understand the theoretical aspects of the model on-the-job, then it is totally fine to directly dive
into the BrandNewBert’s code-base. This option might suit you better if your engineering skills are better than
your theoretical skill, if you have trouble understanding BrandNewBert’s paper, or if you just enjoy programming
much more than reading scientific papers.


1. (Optional) Theoretical aspects of BrandNewBert

You should take some time to read BrandNewBert’s paper, if such descriptive work exists. There might be large
sections of the paper that are difficult to understand. If this is the case, this is fine - don’t worry! The goal is
not to get a deep theoretical understanding of the paper, but to extract the necessary information required to
effectively re-implement the model in 🤗 Transformers. That being said, you don’t have to spend too much time on the
theoretical aspects, but rather focus on the practical ones, namely:
What type of model is brand_new_bert? BERT-like encoder-only model? GPT2-like decoder-only model? BART-like
encoder-decoder model? Look at the model_summary if you’re not familiar with the differences between


 those. What are the applications of brand_new_bert? Text classification? Text generation? Seq2Seq tasks, e.g.,
summarization? What is the novel feature of the model that makes it different from BERT/GPT-2/BART? Which of the already existing 🤗 Transformers models is most
similar to brand_new_bert? What type of tokenizer is used? A sentencepiece tokenizer? Word piece tokenizer? Is it the same tokenizer as used
for BERT or BART?
After you feel like you have gotten a good overview of the architecture of the model, you might want to write to the
Hugging Face team with any questions you might have. This might include questions regarding the model’s architecture,
its attention layer, etc. We will be more than happy to help you.


2. Next prepare your environment

Fork the repository by clicking on the ‘Fork’ button on the
repository’s page. This creates a copy of the code under your GitHub user account. Clone your transformers fork to your local disk, and add the base repository as a remote:
  Copied git clone https://github.com/[your Github handle]/transformers.git
cd transformers
git remote add upstream https://github.com/huggingface/transformers.git
Set up a development environment, for instance by running the following command:
  Copied python -m venv .env
source .env/bin/activate
pip install -e ".[dev]"
Depending on your OS, and since the number of optional dependencies of Transformers is growing, you might get a
failure with this command. If that’s the case make sure to install the Deep Learning framewor


2. Next prepare your environment

k you are working with
(PyTorch, TensorFlow and/or Flax) then do:
  Copied pip install -e ".[quality]"
which should be enough for most use cases. You can then return to the parent directory
  Copied cd ..
We recommend adding the PyTorch version of brand_new_bert to Transformers. To install PyTorch, please follow the
instructions on https://pytorch.org/get-started/locally/.
Note: You don’t need to have CUDA installed. Making the new model work on CPU is sufficient.
To port brand_new_bert, you will also need access to its original repository:
  Copied git clone https://github.com/org_that_created_brand_new_bert_org/brand_new_bert.git
cd brand_new_bert
pip install -e .
Now you have set up a development environment to port brand_new_bert to 🤗 Transformers