Did you find a bug?

The 🤗 Transformers library is robust and reliable thanks to users who report the problems they encounter.
Before you report an issue, we would really appreciate it if you could make sure the bug was not
already reported (use the search bar on GitHub under Issues). Your issue should also be related to bugs in the library itself, and not your code. If you’re unsure whether the bug is in your code or the library, please ask on the forum first. This helps us respond quicker to fixing issues related to the library versus general questions.
Once you’ve confirmed the bug hasn’t already been reported, please includ


Did you find a bug?

e the following information in your issue so we can quickly resolve it:
Your OS type and version and Python, PyTorch and
TensorFlow versions when applicable. A short, self-contained, code snippet that allows us to reproduce the bug in
less than 30s. The full traceback if an exception is raised. Attach any other additional information, like screenshots, you think may help.
To get the OS and software versions automatically, run the following command:
  Copied transformers-cli env
You can also run the same command from the root of the repository:
  Copied python src/transformers/commands/transformers_cli.py en


Do you want a new feature?

If there is a new feature you’d like to see in 🤗 Transformers, please open an issue and describe:
What is the motivation behind this feature? Is it related to a problem or frustration with the library? Is it a feature related to something you need for a project? Is it something you worked on and think it could benefit the community? Whatever it is, we’d love to hear about it! Describe your requested feature in as much detail as possible. The more you can tell us about it, the better we’ll be able to help you. Provide a code snippet that demonstrates the features usage. If the feature is related to a paper, please include a link.
If your issue is well written we’re already 80% of the way there by the time you create it.
We have added templates to help you get started with your issue.


Do you want to implement a new model?

New models are constantly released and if you want to implement a new model, please provide the following information
A short description of the model and link to the paper. Link to the implementation if it is open-sourced. Link to the model weights if they are available.
If you are willing to contribute the model yourself, let us know so we can help you add it to 🤗 Transformers!
We have added a detailed guide and templates to help you get started with adding a new model, and we also have a more technical guide for how to add a model to 🤗 Transformers.


Do you want to add documentation?

We’re always looking for improvements to the documentation that make it more clear and accurate. Please let us know how the documentation can be improved such as typos and any content that is missing, unclear or inaccurate. We’ll be happy to make the changes or help you make a contribution if you’re interested!
For more details about how to generate, build, and write the documentation, take a look at the documentation README.


Create a Pull Request

Before writing any code, we strongly advise you to search through the existing PRs or
issues to make sure nobody is already working on the same thing. If you are
unsure, it is always a good idea to open an issue to get some feedback.
You will need basic git proficiency to contribute to
🤗 Transformers. While git is not the easiest tool to use, it has the greatest
manual. Type git --help in a shell and enjoy! If you prefer books, Pro
Git is a very good reference.
You’ll need Python 3.8 or above to contribute to 🤗 Transformers. Follow the steps below to start contributing:
Fork the repository by
clicking on the Fork button on the repository’s page. This creates a copy of the code
under your GitHub user account. Clone your fork to your local disk, and add the base repository as a remote:   Copied git clone git@github.com:<your Github handle>/transformers.git
cd transformers
git remote add upstream https://github.com/huggingface/transformers.git Creat


Create a Pull Request

e a new branch to hold your development changes:   Copied git checkout -b a-descriptive-name-for-my-changes 🚨 Do not work on the main branch! Set up a development environment by running the following command in a virtual environment:   Copied pip install -e ".[dev]" If 🤗 Transformers was already installed in the virtual environment, remove
it with pip uninstall transformers before reinstalling it in editable
mode with the -e flag. Depending on your OS, and since the number of optional dependencies of Transformers is growing, you might get a
failure with this command. If that’s the case make sure to install the Deep Learning framework you are working with
(PyTorch, TensorFlow and/or Flax) then do:   Copied pip install -e ".[quality]" which should be enough for most use cases. Develop the features on your branch. As you work on your code, you should make sure the test suite
passes. Run the tests impacted by your changes like this:   Copied pytest t


Create a Pull Request

ests/<TEST_TO_RUN>.py For more information about tests, check out the
Testing guide. 🤗 Transformers relies on black and ruff to format its source code
consistently. After you make changes, apply automatic style corrections and code verifications
that can’t be automated in one go with:   Copied make fixup This target is also optimized to only work with files modified by the PR you’re working on. If you prefer to run the checks one after the other, the following command applies the
style corrections:   Copied make style 🤗 Transformers also uses ruff and a few custom scripts to check for coding mistakes. Quality
controls are run by the CI, but you can run the same checks with:   Copied make quality Finally, we have a lot of scripts to make sure we didn’t forget to update
some files when adding a new model. You can run these scripts with:   Copied make repo-consistency To learn more about those checks and how to fix any issues with them, check out th


Create a Pull Request

e
Checks on a Pull Request guide. If you’re modifying documents under docs/source directory, make sure the documentation can still be built. This check will also run in the CI when you open a pull request. To run a local check
make sure you install the documentation builder:   Copied pip install ".[docs]" Run the following command from the root of the repository:   Copied doc-builder build transformers docs/source/en --build_dir ~/tmp/test-build This will build the documentation in the ~/tmp/test-build folder where you can inspect the generated
Markdown files with your favorite editor. You can also preview the docs on GitHub when you open a pull request. Once you’re happy with your changes, add changed files with git add and
record your changes locally with git commit:   Copied git add modified_file.py
git commit Please remember to write good commit
messages to clearly communicate the changes you made! To keep your copy of the code up to date wit


Create a Pull Request

h the original
repository, rebase your branch on upstream/branch before you open a pull request or if requested by a maintainer:   Copied git fetch upstream
git rebase upstream/main Push your changes to your branch:   Copied git push -u origin a-descriptive-name-for-my-changes If you’ve already opened a pull request, you’ll need to force push with the --force flag. Otherwise, if the pull request hasn’t been opened yet, you can just push your changes normally. Now you can go to your fork of the repository on GitHub and click on Pull request to open a pull request. Make sure you tick off all the boxes in our checklist below. When you’re ready, you can send your changes to the project maintainers for review. It’s ok if maintainers request changes, it happens to our core contributors
too! So everyone can see the changes in the pull request, work in your local
branch and push the changes to your fork. They will automatically appear in
the pull request


Pull request checklist

☐ The pull request title should summarize your contribution.
☐ If your pull request addresses an issue, please mention the issue number in the pull
request description to make sure they are linked (and people viewing the issue know you
are working on it).
☐ To indicate a work in progress please prefix the title with [WIP]. These are
useful to avoid duplicated work, and to differentiate it from PRs ready to be merged.
☐ Make sure existing tests pass.
☐ If adding a new feature, also add tests for it.
If you are adding a new model, make sure you use
ModelTester.all_model_classes = (MyModel, MyModelWithLMHead,...) to trigger the common tests. If you are adding new @slow tests, make sure they pass using
RUN_SLOW=1 python -m pytest tests/models/my_new_model/test_my_new_model.py. If you are adding a new tokenizer, write tests and make


Pull request checklist

sure
RUN_SLOW=1 python -m pytest tests/models/{your_model_name}/test_tokenization_{your_model_name}.py passes. CircleCI does not run the slow tests, but GitHub Actions does every night!
☐ All public methods must have informative docstrings (see
modeling_bert.py
for an example).
☐ Due to the rapidly growing repository, don’t add any images, videos and other
non-text files that’ll significantly weigh down the repository. Instead, use a Hub
repository such as hf-internal-testing
to host these files and reference them by URL. We recommend placing documentation
related images in the following repository:
huggingface/documentation-images.
You can open a PR on this dataset repostitory and ask a Hugging Face member to merge it.
For more information about the checks run on a pull request, take a look at our Checks on a Pull Request guide


Tests

An extensive test suite is included to test the library behavior and several examples. Library tests can be found in
the tests folder and examples tests in the
examples folder.
We like pytest and pytest-xdist because it’s faster. From the root of the
repository, specify a path to a subfolder or a test file to run the test.
  Copied python -m pytest -n auto --dist=loadfile -s -v ./tests/models/my_new_model
Similarly, for the examples directory, specify a path to a subfolder or test file to run the test. For example, the following command tests the text classification subfolder in the PyTorch examples directory:
  Copied pip install -r examples/xxx/requirements.txt  # only needed the first time
python -m pytest -n auto --dist=loadfile -s -v ./examples/pytorch/text-classif


Tests

ication
In fact, this is actually how our make test and make test-examples commands are implemented (not including the pip install)!
You can also specify a smaller set of tests in order to test only the feature
you’re working on.
By default, slow tests are skipped but you can set the RUN_SLOW environment variable to
yes to run them. This will download many gigabytes of models so make sure you
have enough disk space, a good internet connection or a lot of patience!
Remember to specify a path to a subfolder or a test file to run the test. Otherwise, you’ll run all the tests in the tests or examples folder, which will take a very long time!
  Copied RUN_SLOW=yes python -m pytest -n auto --dist=loadfile -s -v ./tests/models/my_new_model
RUN_SLOW=yes python -m pytest -n auto


Tests

 --dist=loadfile -s -v ./examples/pytorch/text-classification
Like the slow tests, there are other environment variables available which not enabled by default during testing:
RUN_CUSTOM_TOKENIZERS: Enables tests for custom tokenizers. RUN_PT_FLAX_CROSS_TESTS: Enables tests for PyTorch + Flax integration. RUN_PT_TF_CROSS_TESTS: Enables tests for TensorFlow + PyTorch integration.
More environment variables and additional information can be found in the testing_utils.py.
🤗 Transformers uses pytest as a test runner only. It doesn’t use any
pytest-specific features in the test suite itself.
This means unittest is fully supported. Here’s how to run tests with
unittest:
  Copied python -m unittest discover -s tests -t . -v
python -m unittest discover -s examples -t examples -


Style guide

For documentation strings, 🤗 Transformers follows the Google Python Style Guide.
Check our documentation writing guide
for more information.


Develop on Windows

On Windows (unless you’re working in Windows Subsystem for Linux or WSL), you need to configure git to transform Windows CRLF line endings to Linux LF line endings:
  Copied git config core.autocrlf input
One way to run the make command on Windows is with MSYS2:
Download MSYS2, and we assume it’s installed in C:\msys64. Open the command line C:\msys64\msys2.exe (it should be available from the Start menu). Run in the shell: pacman -Syu and install make with pacman -S make. Add C:\msys64\usr\bin to your PATH environment variable.
You can now use make from any terminal (Powershell, cmd.exe, etc.)! 🎉


Sync a forked repository with upstream main (the Hugging Face repository)

When updating the main branch of a forked repository, please follow these steps to avoid pinging the upstream repository which adds reference notes to each upstream PR, and sends unnecessary notifications to the developers involved in these PRs.
When possible, avoid syncing with the upstream using a branch and PR on the forked repository. Instead, merge directly into the forked main. If a PR is absolutely necessary, use the following steps after checking out your branch:
  Copied git checkout -b your-branch-for-syncing
git pull --squash --no-commit upstream main
git commit -m '<your message without GitHub references>'
git push --set-upstream origin your-branch-for-syncing


XLA Integration for TensorFlow Models

Accelerated Linear Algebra, dubbed XLA, is a compiler for accelerating the runtime of TensorFlow Models. From the official documentation:
XLA (Accelerated Linear Algebra) is a domain-specific compiler for linear algebra that can accelerate TensorFlow models with potentially no source code changes.
Using XLA in TensorFlow is simple – it comes packaged inside the tensorflow library, and it can be triggered with the jit_compile argument in any graph-creating function such as tf.function. When using Keras methods like fit() and predict(), you can enable XLA simply by passing the jit_compile argument to model.compile(). However, XLA is not limited to these methods - it can als


XLA Integration for TensorFlow Models

o be used to accelerate any arbitrary tf.function.
Several TensorFlow methods in 🤗 Transformers have been rewritten to be XLA-compatible, including text generation for models such as GPT2, T5 and OPT, as well as speech processing for models such as Whisper.
While the exact amount of speed-up is very much model-dependent, for TensorFlow text generation models inside 🤗 Transformers, we noticed a speed-up of ~100x. This document will explain how you can use XLA for these models to get the maximum amount of performance. We’ll also provide links to additional resources if you’re interested to learn more about the benchmarks and our design philosophy behind the XLA integration.


Running TF functions with XLA

Let us consider the following model in TensorFlow:
  Copied import tensorflow as tf
model = tf.keras.Sequential(
    [tf.keras.layers.Dense(10, input_shape=(10,), activation="relu"), tf.keras.layers.Dense(5, activation="softmax")]
)
The above model accepts inputs having a dimension of (10, ). We can use the model for running a forward pass like so:
  Copied # Generate random inputs for the model.
batch_size = 16
input_vector_dim = 10
random_inputs = tf.random.normal((batch_size, input_vector_dim))
# Run a forward pass.
_ = model(random_inputs)
In order to run the forward pass with an XLA-compiled function, we’d need to do:
  Copied xla_fn = tf.function(model, jit_compile=True)
_ = xla_fn(random_inputs)
The default call() function of the model is used for compiling the XLA graph. But if there’s any other model function you want to compile into XLA that’s also possible with:
  Copied my_xla_fn = tf.function(model.my_xla_fn, jit_compile=True)


Running a TF text generation model with XLA from 🤗 Transformers

To enable XLA-accelerated generation within 🤗 Transformers, you need to have a recent version of transformers installed. You can install it by running:
  Copied pip install transformers --upgrade
And then you can run the following code:
  Copied import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModelForCausalLM
# Will error if the minimal version of Transformers is not installed.
from transformers.utils import check_min_version
check_min_version("4.21.0")
tokenizer = AutoTokenizer.from_pretrained("gpt2", padding_side="left", pad_token="</s>")
model = TFAutoModelForCausalLM.from_pretrained("gpt2")
input_string = ["TensorFlow is"]
# One line to create an XLA generation function
xla_generate = tf.


function(model.generate, jit_compile=True)
tokenized_input = tokenizer(input_string, return_tensors="tf")
generated_tokens = xla_generate(**tokenized_input, num_beams=2)
decoded_text = tokenizer.decode(generated_tokens[0], skip_special_tokens=True)
print(f"Generated -- {decoded_text}")
# Generated -- TensorFlow is an open-source, open-source, distributed-source application # framework for the
As you can notice, enabling XLA on generate() is just a single line of code. The rest of the code remains unchanged. However, there are a couple of gotchas in the above code snippet that are specific to XLA. You need to be aware of those to realize the speed-ups that XLA can bring in. We discuss these in the following section


Gotchas to be aware of

When you are executing an XLA-enabled function (like xla_generate() above) for the first time, it will internally try to infer the computation graph, which is time-consuming.  This process is known as “tracing”.
You might notice that the generation time is not fast. Successive calls of xla_generate() (or any other XLA-enabled function) won’t have to infer the computation graph, given the inputs to the function follow the same shape with which the computation graph was initially built. While this is not a problem for modalities with fixed input shapes (e.g., images), you must pay attention if you are working with variable input shape modalities (e.g., text).
To ensure xla_generate() always operates with the same input shapes, you can specify the padding arguments when calling the tokenizer.
  Copied import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("gpt2", padding_side="left", pad_toke


Gotchas to be aware of

n="</s>")
model = TFAutoModelForCausalLM.from_pretrained("gpt2")
input_string = ["TensorFlow is"]
xla_generate = tf.function(model.generate, jit_compile=True)
# Here, we call the tokenizer with padding options.
tokenized_input = tokenizer(input_string, pad_to_multiple_of=8, padding=True, return_tensors="tf")
generated_tokens = xla_generate(**tokenized_input, num_beams=2)
decoded_text = tokenizer.decode(generated_tokens[0], skip_special_tokens=True)
print(f"Generated -- {decoded_text}")
This way, you can ensure that the inputs to xla_generate() will always receive inputs with the shape it was traced with and thus leading to speed-ups in the generation time. You can verify this with the code below:
  Copied import time
import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("gpt2", padding_side="left", pad_token="</s>")
model = TFAutoModelForCausalLM.from_pretrained("gpt2")
xla_generate = tf.


Gotchas to be aware of

function(model.generate, jit_compile=True)
for input_string in ["TensorFlow is", "TensorFlow is a", "TFLite is a"]:
    tokenized_input = tokenizer(input_string, pad_to_multiple_of=8, padding=True, return_tensors="tf")
    start = time.time_ns()
    generated_tokens = xla_generate(**tokenized_input, num_beams=2)
    end = time.time_ns()
    print(f"Execution time -- {(end - start) / 1e6:.1f} ms\n")
On a Tesla T4 GPU, you can expect the outputs like so:
  Copied Execution time -- 30819.6 ms
Execution time -- 79.0 ms
Execution time -- 78.9 ms
The first call to xla_generate() is time-consuming because of tracing, but the successive calls are orders of magnitude faster. Keep in mind that any change in the generation options at any point with trigger re-tracing and thus leading to slow-downs in the generation time.
We didn’t cover all the text generation options 🤗 Transformers provides in this document. We encourage you to read the documentation for advanced use case


Additional Resources

Here, we leave you with some additional resources if you want to delve deeper into XLA in 🤗 Transformers and in general.
This Colab Notebook provides an interactive demonstration if you want to fiddle with the XLA-compatible encoder-decoder (like T5) and decoder-only (like GPT2) text generation models. This blog post provides an overview of the comparison benchmarks for XLA-compatible models along with a friendly introduction to XLA in TensorFlow. This blog post discusses our design philosophy behind adding XLA support to the TensorFlow models in 🤗 Transformers. Recommended posts for learning more about XLA and TensorFlow graphs in general:XLA: Optimizing Compiler for Machine Learning Introduction to graphs and tf.function Better performance with tf.function


Share a model

The last two tutorials showed how you can fine-tune a model with PyTorch, Keras, and 🤗 Accelerate for distributed setups. The next step is to share your model with the community! At Hugging Face, we believe in openly sharing knowledge and resources to democratize artificial intelligence for everyone. We encourage you to consider sharing your model with the community to help others save time and resources.
In this tutorial, you will learn two methods for sharing a trained or fine-tuned model on the Model Hub:
Programmatically push your files to the Hub. Drag-and-drop your files to the Hub with the web interface.
To share a model with the community, you need an account on huggingface.co. You can also join an existing organization or create a new one.


Repository features

Each repository on the Model Hub behaves like a typical GitHub repository. Our repositories offer versioning, commit history, and the ability to visualize differences.
The Model Hub’s built-in versioning is based on git and git-lfs. In other words, you can treat one model as one repository, enabling greater access control and scalability. Version control allows revisions, a method for pinning a specific version of a model with a commit hash, tag or branch.
As a result, you can load a specific model version with the revision parameter:
  Copied >>> model = AutoModel.from_pretrained(
...     "julien-c/EsperBERTo-small", revision="v2.0.1"  # tag name, or branch name, or commit hash
... )
Files are also easily edited in a repository, and you can view the commit history as well as the difference:


Setup

Before sharing a model to the Hub, you will need your Hugging Face credentials. If you have access to a terminal, run the following command in the virtual environment where 🤗 Transformers is installed. This will store your access token in your Hugging Face cache folder (~/.cache/ by default):
  Copied huggingface-cli login
If you are using a notebook like Jupyter or Colaboratory, make sure you have the huggingface_hub library installed. This library allows you to programmatically interact with the Hub.
  Copied pip install huggingface_hub
Then use notebook_login to sign-in to the Hub, and follow the link here to generate a token to login with:
  Copied >>> from huggingface_hub import notebook_login
>>> notebook_login()


Convert a model for all frameworks

To ensure your model can be used by someone working with a different framework, we recommend you convert and upload your model with both PyTorch and TensorFlow checkpoints. While users are still able to load your model from a different framework if you skip this step, it will be slower because 🤗 Transformers will need to convert the checkpoint on-the-fly.
Converting a checkpoint for another framework is easy. Make sure you have PyTorch and TensorFlow installed (see here for installation instructions), and then find the specific model for your task in the other framework.
  Pytorch  Hide Pytorch content  Specify from_tf=True to convert a checkpoint from TensorFlow to PyTorch:   Copied >>> pt_model = DistilBertForSequenceClassification.from_pretra


Convert a model for all frameworks

ined("path/to/awesome-name-you-picked", from_tf=True)
>>> pt_model.save_pretrained("path/to/awesome-name-you-picked")  TensorFlow  Hide TensorFlow content  Specify from_pt=True to convert a checkpoint from PyTorch to TensorFlow:   Copied >>> tf_model = TFDistilBertForSequenceClassification.from_pretrained("path/to/awesome-name-you-picked", from_pt=True) Then you can save your new TensorFlow model with its new checkpoint:   Copied >>> tf_model.save_pretrained("path/to/awesome-name-you-picked")   JAX  Hide JAX content  If a model is available in Flax, you can also convert a checkpoint from PyTorch to Flax:   Copied >>> flax_model = FlaxDistilBertForSequenceClassification.from_pretrained(
...     "path/to/awesome-name-you-picked", from_pt=True
...


Push a model during training

Pytorch  Hide Pytorch content   Sharing a model to the Hub is as simple as adding an extra parameter or callback. Remember from the fine-tuning tutorial, the TrainingArguments class is where you specify hyperparameters and additional training options. One of these training options includes the ability to push a model directly to the Hub. Set push_to_hub=True in your TrainingArguments:   Copied >>> training_args = TrainingArguments(output_dir="my-awesome-model", push_to_hub=True) Pass your training arguments as usual to Trainer:   Copied >>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=small_train_dataset,
...     eval_dataset=small_eval_dataset,
...     compute_metrics=compute_metrics,
... ) After you fine-tune your model, call push_to_hub() on Trainer to push the trained model to


Push a model during training

 the Hub. 🤗 Transformers will even automatically add training hyperparameters, training results and framework versions to your model card!   Copied >>> trainer.push_to_hub()  TensorFlow  Hide TensorFlow content  Share a model to the Hub with PushToHubCallback. In the PushToHubCallback function, add: An output directory for your model. A tokenizer. The hub_model_id, which is your Hub username and model name.   Copied >>> from transformers import PushToHubCallback
>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="./your_model_save_path", tokenizer=tokenizer, hub_model_id="your-username/my-awesome-model"
... ) Add the callback to fit, and 🤗 Transformers will push the trained model to the Hub:   Copied >>> model.fit(tf_train_dataset, validation_data=tf_validation_dataset, epochs=3, callbacks=push_to_hub_callback)


Use the push_to_hub function

You can also call push_to_hub directly on your model to upload it to the Hub.
Specify your model name in push_to_hub:
  Copied >>> pt_model.push_to_hub("my-awesome-model")
This creates a repository under your username with the model name my-awesome-model. Users can now load your model with the from_pretrained function:
  Copied >>> from transformers import AutoModel
>>> model = AutoModel.from_pretrained("your_username/my-awesome-model")
If you belong to an organization and want to push your model under the organization name instead, just add it to the repo_id:
  Copied >>> pt_model.push_to_hub("my-awesome-org/my-awesome-mod


Use the push_to_hub function

el")
The push_to_hub function can also be used to add other files to a model repository. For example, add a tokenizer to a model repository:
  Copied >>> tokenizer.push_to_hub("my-awesome-model")
Or perhaps you’d like to add the TensorFlow version of your fine-tuned PyTorch model:
  Copied >>> tf_model.push_to_hub("my-awesome-model")
Now when you navigate to your Hugging Face profile, you should see your newly created model repository. Clicking on the Files tab will display all the files you’ve uploaded to the repository.
For more details on how to create and upload files to a repository, refer to the Hub documentation here


Upload with the web interface

Users who prefer a no-code approach are able to upload a model through the Hub’s web interface. Visit huggingface.co/new to create a new repository:
From here, add some information about your model:
Select the owner of the repository. This can be yourself or any of the organizations you belong to. Pick a name for your model, which will also be the repository name. Choose whether your model is public or private. Specify the license usage for your model.
Now click on the Files tab and click on the Add file button to upload a new file to your repository. Then drag-and-drop a file to upload and add a commit message.


Add a model card

To make sure users understand your model’s capabilities, limitations, potential biases and ethical considerations, please add a model card to your repository. The model card is defined in the README.md file. You can add a model card by:
Manually creating and uploading a README.md file. Clicking on the Edit model card button in your model repository.
Take a look at the DistilBert model card for a good example of the type of information a model card should include. For more details about other options you can control in the README.md file such as a model’s carbon footprint or widget examples, refer to the documentation here.


Fine-tune a pretrained model

There are significant benefits to using a pretrained model. It reduces computation costs, your carbon footprint, and allows you to use state-of-the-art models without having to train one from scratch. 🤗 Transformers provides access to thousands of pretrained models for a wide range of tasks. When you use a pretrained model, you train it on a dataset specific to your task. This is known as fine-tuning, an incredibly powerful training technique. In this tutorial, you will fine-tune a pretrained model with a deep learning framework of your choice:
Fine-tune a pretrained model with 🤗 Transformers Trainer. Fine-tune a pretrained model in TensorFlow with Keras. Fine-tune a pretrained model in native PyTorch.


Prepare a dataset

Before you can fine-tune a pretrained model, download a dataset and prepare it for training. The previous tutorial showed you how to process data for training, and now you get an opportunity to put those skills to the test!
Begin by loading the Yelp Reviews dataset:
  Copied >>> from datasets import load_dataset
>>> dataset = load_dataset("yelp_review_full")
>>> dataset["train"][100]
{'label': 0,
 'text': 'My expectations for McDonalds are t rarely high. But for one to still fail so spectacularly...that takes something special!\\nThe cashier took my friends\'s order, then promptly ignored me. I had to force myself in front of a cashier who opened his register to wait on the person BEHIND me. I waited over five minutes for a gigantic order that included precisely one kid\'s meal. After watching two people who ordered after me be handed their food, I asked


Prepare a dataset

where mine was. The manager started yelling at the cashiers for \\"serving off their orders\\" when they didn\'t have their food. But neither cashier was anywhere near those controls, and the manager was the one serving food to customers and clearing the boards.\\nThe manager was rude when giving me my order. She didn\'t make sure that I had everything ON MY RECEIPT, and never even had the decency to apologize that I felt I was getting poor service.\\nI\'ve eaten at various McDonalds restaurants for over 30 years. I\'ve worked at more than one location. I expect bad days, bad moods, and the occasional mistake. But I have yet to have a decent experience at this store. It will remain a place I avoid unless someone in my party needs to avoid illness from low blood sugar. Perhaps I should go back to the racially biased service of Steak n Shake instead!'}
As y


Prepare a dataset

ou now know, you need a tokenizer to process the text and include a padding and truncation strategy to handle any variable sequence lengths. To process your dataset in one step, use 🤗 Datasets map method to apply a preprocessing function over the entire dataset:
  Copied >>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
>>> def tokenize_function(examples):
...     return tokenizer(examples["text"], padding="max_length", truncation=True)
>>> tokenized_datasets = dataset.map(tokenize_function, batched=True)
If you like, you can create a smaller subset of the full dataset to fine-tune on to reduce the time it takes:
  Copied >>> small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
>>> small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))


Train

At this point, you should follow the section corresponding to the framework you want to use. You can use the links
in the right sidebar to jump to the one you want - and if you want to hide all of the content for a given framework,
just use the button at the top-right of that framework’s block!
  Pytorch  Hide Pytorch content    Train with PyTorch Trainer 🤗 Transformers provides a Trainer class optimized for training 🤗 Transformers models, making it easier to start training without manually writing your own training loop. The Trainer API supports a wide range of training options and features such as logging, gradient accumulation, and mixed precision. Start by loading your model and specify the number of expected labels. From the Yelp Review dataset card, you know there are five labels:   Copied >>> from transformers import AutoModelForSequenceClassification
>>> model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=5) You will see a war


Train

ning about some of the pretrained weights not being used and some weights being randomly
initialized. Don’t worry, this is completely normal! The pretrained head of the BERT model is discarded, and replaced with a randomly initialized classification head. You will fine-tune this new model head on your sequence classification task, transferring the knowledge of the pretrained model to it.  Training hyperparameters Next, create a TrainingArguments class which contains all the hyperparameters you can tune as well as flags for activating different training options. For this tutorial you can start with the default training hyperparameters, but feel free to experiment with these to find your optimal settings. Specify where to save the checkpoints from your training:   Copied >>> from transformers import TrainingArguments
>>> training_args = TrainingArguments(output_dir="test_trainer")  Evaluate Trainer does not automatically evaluate model performance during training. You’ll n


Train

eed to pass Trainer a function to compute and report metrics. The 🤗 Evaluate library provides a simple accuracy function you can load with the evaluate.load (see this quicktour for more information) function:   Copied >>> import numpy as np
>>> import evaluate
>>> metric = evaluate.load("accuracy") Call compute on metric to calculate the accuracy of your predictions. Before passing your predictions to compute, you need to convert the predictions to logits (remember all 🤗 Transformers models return logits):   Copied >>> def compute_metrics(eval_pred):
...     logits, labels = eval_pred
...     predictions = np.argmax(logits, axis=-1)
...     return metric.compute(predictions=predictions, references=labels) If you’d like to monitor your evaluation metrics during fine-tuning, specify the evaluation_strategy parameter in your training arguments to report the evaluation metric at the end of each epoch:   Copied >>> from transformers import TrainingArguments, Trainer
>>> train


Train

ing_args = TrainingArguments(output_dir="test_trainer", evaluation_strategy="epoch")  Trainer Create a Trainer object with your model, training arguments, training and test datasets, and evaluation function:   Copied >>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=small_train_dataset,
...     eval_dataset=small_eval_dataset,
...     compute_metrics=compute_metrics,
... ) Then fine-tune your model by calling train():   Copied >>> trainer.train()  TensorFlow  Hide TensorFlow content     Train a TensorFlow model with Keras You can also train 🤗 Transformers models in TensorFlow with the Keras API!  Loading data for Keras When you want to train a 🤗 Transformers model with the Keras API, you need to convert your dataset to a format that
Keras understands. If your dataset is small, you can just convert the whole thing to NumPy arrays and pass it to Keras.
Let’s try that first before we do anything more complicated. First, load a dat


Train

aset. We’ll use the CoLA dataset from the GLUE benchmark,
since it’s a simple binary text classification task, and just take the training split for now.   Copied from datasets import load_dataset
dataset = load_dataset("glue", "cola")
dataset = dataset["train"]  # Just take the training split for now Next, load a tokenizer and tokenize the data as NumPy arrays. Note that the labels are already a list of 0 and 1s,
so we can just convert that directly to a NumPy array without tokenization!   Copied from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
tokenized_data = tokenizer(dataset["sentence"], return_tensors="np", padding=True)
# Tokenizer returns a BatchEncoding, but we convert that to a dict for Keras
tokenized_data = dict(tokenized_data)
labels = np.array(dataset["label"])  # Label is already an array of 0 and 1 Finally, load, compile, and fit the model. Note that Transformers models all have a default task-relevant los


Train

s function, so you don’t need to specify one unless you want to:   Copied from transformers import TFAutoModelForSequenceClassification
from tensorflow.keras.optimizers import Adam
# Load and compile our model
model = TFAutoModelForSequenceClassification.from_pretrained("bert-base-cased")
# Lower learning rates are often better for fine-tuning transformers
model.compile(optimizer=Adam(3e-5))  # No loss argument!
model.fit(tokenized_data, labels) You don’t have to pass a loss argument to your models when you compile() them! Hugging Face models automatically
choose a loss that is appropriate for their task and model architecture if this argument is left blank. You can always
override this by specifying a loss yourself if you want to! This approach works great for smaller datasets, but for larger datasets, you might find it starts to become a problem. Why?
Because the tokenized array and labels would have to be fully loaded into memory, and because NumPy doesn’t handle
“jag


Train

ged” arrays, so every tokenized sample would have to be padded to the length of the longest sample in the whole
dataset. That’s going to make your array even bigger, and all those padding tokens will slow down training too!  Loading data as a tf.data.Dataset If you want to avoid slowing down training, you can load your data as a tf.data.Dataset instead. Although you can write your own
tf.data pipeline if you want, we have two convenience methods for doing this: prepare_tf_dataset(): This is the method we recommend in most cases. Because it is a method
on your model, it can inspect the model to automatically figure out which columns are usable as model inputs, and
discard the others to make a simpler, more performant dataset. to_tf_dataset: This method is more low-level, and is useful when you want to exactly control how
your dataset is created, by specifying exactly which columns and label_cols to include. Before you can use prepare_tf_dataset(), you will need to add the


Train

 tokenizer outputs to your dataset as columns, as shown in
the following code sample:   Copied def tokenize_dataset(data):
    # Keys of the returned dictionary will be added to the dataset as columns
    return tokenizer(data["text"])
dataset = dataset.map(tokenize_dataset) Remember that Hugging Face datasets are stored on disk by default, so this will not inflate your memory usage! Once the
columns have been added, you can stream batches from the dataset and add padding to each batch, which greatly
reduces the number of padding tokens compared to padding the entire dataset.   Copied >>> tf_dataset = model.prepare_tf_dataset(dataset["train"], batch_size=16, shuffle=True, tokenizer=tokenizer) Note that in the code sample above, you need to pass the tokenizer to prepare_tf_dataset so it can correctly pad batches as they’re loaded.
If all the samples in your dataset are the same length and no padding is necessary, you can skip this argument.
If you need to do something mor


Train

e complex than just padding samples (e.g. corrupting tokens for masked language
modelling), you can use the collate_fn argument instead to pass a function that will be called to transform the
list of samples into a batch and apply any preprocessing you want. See our
examples or
notebooks to see this approach in action. Once you’ve created a tf.data.Dataset, you can compile and fit the model as before:   Copied model.compile(optimizer=Adam(3e-5))  # No loss argument!
model.fit(tf_dataset)
 Train in native PyTorch
  Pytorch  Hide Pytorch content   Trainer takes care of the training loop and allows you to fine-tune a model in a single line of code. For users who prefer to write their own training loop, you can also fine-tune a 🤗 Transformers model in native PyTorch. At this point, you may need to restart your notebook or execute the following code to free some memory:   Copied del model
del trainer
torch.cuda.empty_cache() Next, manually postprocess tokenized_dataset to p


Train

repare it for training. Remove the text column because the model does not accept raw text as an input:   Copied >>> tokenized_datasets = tokenized_datasets.remove_columns(["text"]) Rename the label column to labels because the model expects the argument to be named labels:   Copied >>> tokenized_datasets = tokenized_datasets.rename_column("label", "labels") Set the format of the dataset to return PyTorch tensors instead of lists:   Copied >>> tokenized_datasets.set_format("torch") Then create a smaller subset of the dataset as previously shown to speed up the fine-tuning:   Copied >>> small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
>>> small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))  DataLoader Create a DataLoader for your training and test datasets so you can iterate over batches of data:   Copied >>> from torch.utils.data import DataLoader
>>> train_dataloader = DataLoader(small_train_dataset,


Train

shuffle=True, batch_size=8)
>>> eval_dataloader = DataLoader(small_eval_dataset, batch_size=8) Load your model with the number of expected labels:   Copied >>> from transformers import AutoModelForSequenceClassification
>>> model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=5)  Optimizer and learning rate scheduler Create an optimizer and learning rate scheduler to fine-tune the model. Let’s use the AdamW optimizer from PyTorch:   Copied >>> from torch.optim import AdamW
>>> optimizer = AdamW(model.parameters(), lr=5e-5) Create the default learning rate scheduler from Trainer:   Copied >>> from transformers import get_scheduler
>>> num_epochs = 3
>>> num_training_steps = num_epochs * len(train_dataloader)
>>> lr_scheduler = get_scheduler(
...     name="linear", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps
... ) Lastly, specify device to use a GPU if you have access to one. Otherwise, training on a CP


Train

U may take several hours instead of a couple of minutes.   Copied >>> import torch
>>> device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
>>> model.to(device) Get free access to a cloud GPU if you don’t have one with a hosted notebook like Colaboratory or SageMaker StudioLab. Great, now you are ready to train! 🥳  Training loop To keep track of your training progress, use the tqdm library to add a progress bar over the number of training steps:   Copied >>> from tqdm.auto import tqdm
>>> progress_bar = tqdm(range(num_training_steps))
>>> model.train()
>>> for epoch in range(num_epochs):
...     for batch in train_dataloader:
...         batch = {k: v.to(device) for k, v in batch.items()}
...         outputs = model(**batch)
...         loss = outputs.loss
...         loss.backward()
...         optimizer.step()
...         lr_scheduler.step()
...         optimizer.zero_grad()
...         progress_bar.update(1)  Evaluate Just like how you a


Train

dded an evaluation function to Trainer, you need to do the same when you write your own training loop. But instead of calculating and reporting the metric at the end of each epoch, this time you’ll accumulate all the batches with add_batch and calculate the metric at the very end.   Copied >>> import evaluate
>>> metric = evaluate.load("accuracy")
>>> model.eval()
>>> for batch in eval_dataloader:
...     batch = {k: v.to(device) for k, v in batch.items()}
...     with torch.no_grad():
...         outputs = model(**batch)
...     logits = outputs.logits
...     predictions = torch.argmax(logits, dim=-1)
...     metric.add_batch(predictions=predictions, references=batch["labels"])
>>> metric.compute()
 Additional resources
For more fine-tuning examples, refer to:
🤗 Transformers Examples includes scripts
to train common NLP tasks in PyTorch and TensorFlow. 🤗 Transformers Notebooks contains various notebooks on how to fine-tune a model for specific tasks in PyTorch and Ten


Train with PyTorch Trainer

🤗 Transformers provides a Trainer class optimized for training 🤗 Transformers models, making it easier to start training without manually writing your own training loop. The Trainer API supports a wide range of training options and features such as logging, gradient accumulation, and mixed precision.
Start by loading your model and specify the number of expected labels. From the Yelp Review dataset card, you know there are five labels:
  Copied >>> from transformers import AutoModelForSequenceClassifica


Train with PyTorch Trainer

tion
>>> model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=5)
You will see a warning about some of the pretrained weights not being used and some weights being randomly
initialized. Don’t worry, this is completely normal! The pretrained head of the BERT model is discarded, and replaced with a randomly initialized classification head. You will fine-tune this new model head on your sequence classification task, transferring the knowledge of the pretrained model to it


Training hyperparameters

Next, create a TrainingArguments class which contains all the hyperparameters you can tune as well as flags for activating different training options. For this tutorial you can start with the default training hyperparameters, but feel free to experiment with these to find your optimal settings.
Specify where to save the checkpoints from your training:
  Copied >>> from transformers import TrainingArguments
>>> training_args = TrainingArguments(output_dir="test_trainer")


Evaluate

Trainer does not automatically evaluate model performance during training. You’ll need to pass Trainer a function to compute and report metrics. The 🤗 Evaluate library provides a simple accuracy function you can load with the evaluate.load (see this quicktour for more information) function:
  Copied >>> import numpy as np
>>> import evaluate
>>> metric = evaluate.load("accuracy")
Call compute on metric to calculate the accuracy of your predictions. Before passing your predictions to compute, you need to convert the predictions to logits (remember all 🤗 Transformers mode


Evaluate

ls return logits):
  Copied >>> def compute_metrics(eval_pred):
...     logits, labels = eval_pred
...     predictions = np.argmax(logits, axis=-1)
...     return metric.compute(predictions=predictions, references=labels)
If you’d like to monitor your evaluation metrics during fine-tuning, specify the evaluation_strategy parameter in your training arguments to report the evaluation metric at the end of each epoch:
  Copied >>> from transformers import TrainingArguments, Trainer
>>> training_args = TrainingArguments(output_dir="test_trainer", evaluation_strategy="epoch")


Trainer

Create a Trainer object with your model, training arguments, training and test datasets, and evaluation function:
  Copied >>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=small_train_dataset,
...     eval_dataset=small_eval_dataset,
...     compute_metrics=compute_metrics,
... )
Then fine-tune your model by calling train():
  Copied >>> trainer.train()


Loading data for Keras

When you want to train a 🤗 Transformers model with the Keras API, you need to convert your dataset to a format that
Keras understands. If your dataset is small, you can just convert the whole thing to NumPy arrays and pass it to Keras.
Let’s try that first before we do anything more complicated.
First, load a dataset. We’ll use the CoLA dataset from the GLUE benchmark,
since it’s a simple binary text classification task, and just take the training split for now.
  Copied from datasets import load_dataset
dataset = load_dataset("glue", "cola")
dataset = dataset["train"]  # Just take the training split for now
Next, load a tokenizer and tokenize the data as NumPy arrays. Note that the labels are already a list of 0 and 1s,
so we can just convert that directly to a NumPy array without tokenization!
  Copied from transformers i


Loading data for Keras

mport AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
tokenized_data = tokenizer(dataset["sentence"], return_tensors="np", padding=True)
# Tokenizer returns a BatchEncoding, but we convert that to a dict for Keras
tokenized_data = dict(tokenized_data)
labels = np.array(dataset["label"])  # Label is already an array of 0 and 1
Finally, load, compile, and fit the model. Note that Transformers models all have a default task-relevant loss function, so you don’t need to specify one unless you want to:
  Copied from transformers import TFAutoModelForSequenceClassification
from tensorflow.keras.optimizers import Adam
# Load and compile our model
model = TFAutoModelForSequenceClassification.from_pretrained("bert-base-cased")
# Lower learning rates are often better for fine-tuning transformers
model.compil


Loading data for Keras

e(optimizer=Adam(3e-5))  # No loss argument!
model.fit(tokenized_data, labels)
You don’t have to pass a loss argument to your models when you compile() them! Hugging Face models automatically
choose a loss that is appropriate for their task and model architecture if this argument is left blank. You can always
override this by specifying a loss yourself if you want to!
This approach works great for smaller datasets, but for larger datasets, you might find it starts to become a problem. Why?
Because the tokenized array and labels would have to be fully loaded into memory, and because NumPy doesn’t handle
“jagged” arrays, so every tokenized sample would have to be padded to the length of the longest sample in the whole
dataset. That’s going to make your array even bigger, and all those padding tokens will slow down training to


Loading data as a tf.data.Dataset

If you want to avoid slowing down training, you can load your data as a tf.data.Dataset instead. Although you can write your own
tf.data pipeline if you want, we have two convenience methods for doing this:
prepare_tf_dataset(): This is the method we recommend in most cases. Because it is a method
on your model, it can inspect the model to automatically figure out which columns are usable as model inputs, and
discard the others to make a simpler, more performant dataset. to_tf_dataset: This method is more low-level, and is useful when you want to exactly control how
your dataset is created, by specifying exactly which columns and label_cols to include.
Before you can use prepare_tf_dataset(), you will need to add the tokeniz


Loading data as a tf.data.Dataset

er outputs to your dataset as columns, as shown in
the following code sample:
  Copied def tokenize_dataset(data):
    # Keys of the returned dictionary will be added to the dataset as columns
    return tokenizer(data["text"])
dataset = dataset.map(tokenize_dataset)
Remember that Hugging Face datasets are stored on disk by default, so this will not inflate your memory usage! Once the
columns have been added, you can stream batches from the dataset and add padding to each batch, which greatly
reduces the number of padding tokens compared to padding the entire dataset.
  Copied >>> tf_dataset = model.prepare_tf_dataset(dataset["train"], batch_size=16, shuffle=True, tokenizer=tokenizer)
Note that in the code sample above, you


Loading data as a tf.data.Dataset

need to pass the tokenizer to prepare_tf_dataset so it can correctly pad batches as they’re loaded.
If all the samples in your dataset are the same length and no padding is necessary, you can skip this argument.
If you need to do something more complex than just padding samples (e.g. corrupting tokens for masked language
modelling), you can use the collate_fn argument instead to pass a function that will be called to transform the
list of samples into a batch and apply any preprocessing you want. See our
examples or
notebooks to see this approach in action.
Once you’ve created a tf.data.Dataset, you can compile and fit the model as before:
  Copied model.compile(optimizer=Adam(3e-5))  # No loss argument!
model.fit(tf_dataset


Train in native PyTorch

Pytorch  Hide Pytorch content   Trainer takes care of the training loop and allows you to fine-tune a model in a single line of code. For users who prefer to write their own training loop, you can also fine-tune a 🤗 Transformers model in native PyTorch. At this point, you may need to restart your notebook or execute the following code to free some memory:   Copied del model
del trainer
torch.cuda.empty_cache() Next, manually postprocess tokenized_dataset to prepare it for training. Remove the text column because the model does not accept raw text as an input:   Copied >>> tokenized_datasets = tokenized_datasets.remove_columns(["text"]) Rename the label column to labels because the model expects the argument to be named labels:   Copied >>> tokenized_datasets = tokenized_datasets.rename_column("label", "labels") Set the format of the dataset to return PyTorch tensors ins


Train in native PyTorch

tead of lists:   Copied >>> tokenized_datasets.set_format("torch") Then create a smaller subset of the dataset as previously shown to speed up the fine-tuning:   Copied >>> small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
>>> small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))  DataLoader Create a DataLoader for your training and test datasets so you can iterate over batches of data:   Copied >>> from torch.utils.data import DataLoader
>>> train_dataloader = DataLoader(small_train_dataset, shuffle=True, batch_size=8)
>>> eval_dataloader = DataLoader(small_eval_dataset, batch_size=8) Load your model with the number of expected labels:   Copied >>> from transformers import AutoModelForSequenceClassification
>>> model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=5)


Train in native PyTorch

 Optimizer and learning rate scheduler Create an optimizer and learning rate scheduler to fine-tune the model. Let’s use the AdamW optimizer from PyTorch:   Copied >>> from torch.optim import AdamW
>>> optimizer = AdamW(model.parameters(), lr=5e-5) Create the default learning rate scheduler from Trainer:   Copied >>> from transformers import get_scheduler
>>> num_epochs = 3
>>> num_training_steps = num_epochs * len(train_dataloader)
>>> lr_scheduler = get_scheduler(
...     name="linear", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps
... ) Lastly, specify device to use a GPU if you have access to one. Otherwise, training on a CPU may take several hours instead of a couple of minutes.   Copied >>> import torch
>>> device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
>>> model.to(device) Get free access to a


Train in native PyTorch

cloud GPU if you don’t have one with a hosted notebook like Colaboratory or SageMaker StudioLab. Great, now you are ready to train! 🥳  Training loop To keep track of your training progress, use the tqdm library to add a progress bar over the number of training steps:   Copied >>> from tqdm.auto import tqdm
>>> progress_bar = tqdm(range(num_training_steps))
>>> model.train()
>>> for epoch in range(num_epochs):
...     for batch in train_dataloader:
...         batch = {k: v.to(device) for k, v in batch.items()}
...         outputs = model(**batch)
...         loss = outputs.loss
...         loss.backward()
...         optimizer.step()
...         lr_scheduler.step()
...         optimizer.zero_grad()
...         progress_bar.update(1)  Evaluate Just like how you added an evaluation function to Trainer, you need to do the same when you write your own training loop. But ins


Train in native PyTorch

tead of calculating and reporting the metric at the end of each epoch, this time you’ll accumulate all the batches with add_batch and calculate the metric at the very end.   Copied >>> import evaluate
>>> metric = evaluate.load("accuracy")
>>> model.eval()
>>> for batch in eval_dataloader:
...     batch = {k: v.to(device) for k, v in batch.items()}
...     with torch.no_grad():
...         outputs = model(**batch)
...     logits = outputs.logits
...     predictions = torch.argmax(logits, dim=-1)
...     metric.add_batch(predictions=predictions, references=batch["labels"])
>>> metric.compute()
 Additional resources
For more fine-tuning examples, refer to:
🤗 Transformers Examples includes scripts
to train common NLP tasks in PyTorch and TensorFlow. 🤗 Transformers Notebooks contains various notebooks on how to fine-tune a model for specific tasks in PyTorch and TensorFlow


DataLoader

Create a DataLoader for your training and test datasets so you can iterate over batches of data:
  Copied >>> from torch.utils.data import DataLoader
>>> train_dataloader = DataLoader(small_train_dataset, shuffle=True, batch_size=8)
>>> eval_dataloader = DataLoader(small_eval_dataset, batch_size=8)
Load your model with the number of expected labels:
  Copied >>> from transformers import AutoModelForSequenceClassification
>>> model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=5)


Optimizer and learning rate scheduler

Create an optimizer and learning rate scheduler to fine-tune the model. Let’s use the AdamW optimizer from PyTorch:
  Copied >>> from torch.optim import AdamW
>>> optimizer = AdamW(model.parameters(), lr=5e-5)
Create the default learning rate scheduler from Trainer:
  Copied >>> from transformers import get_scheduler
>>> num_epochs = 3
>>> num_training_steps = num_epochs * len(train_dataloader)
>>> lr_scheduler = get_scheduler(
...     name="linear", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps
... )
Lastly, specify device to use a GPU if you have access to one. Otherwise, training on a CPU may take several hours instead of a couple of minutes.
  Copied >>> import torch
>>> device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
>>> model.to(device)
Get free access to a cloud GPU if you don’t have one with a hosted notebook like Colaboratory or SageMaker StudioLab.
Great, now you are ready to train! 🥳


Training loop

To keep track of your training progress, use the tqdm library to add a progress bar over the number of training steps:
  Copied >>> from tqdm.auto import tqdm
>>> progress_bar = tqdm(range(num_training_steps))
>>> model.train()
>>> for epoch in range(num_epochs):
...     for batch in train_dataloader:
...         batch = {k: v.to(device) for k, v in batch.items()}
...         outputs = model(**batch)
...         loss = outputs.loss
...         loss.backward()
...         optimizer.step()
...         lr_scheduler.step()
...         optimizer.zero_grad()
...         progress_bar.update(1)


Evaluate

Just like how you added an evaluation function to Trainer, you need to do the same when you write your own training loop. But instead of calculating and reporting the metric at the end of each epoch, this time you’ll accumulate all the batches with add_batch and calculate the metric at the very end.
  Copied >>> import evaluate
>>> metric = evaluate.load("accuracy")
>>> model.eval()
>>> for batch in eval_dataloader:
...     batch = {k: v.to(device) for k, v in batch.items()}
...     with torch.no_grad():
...         outputs = model(**batch)
...     logits = outputs.logits
...     predictions = torch.argmax(logits, dim=-1)
...     metric.add_batch(predictions=predictions, references=batch["labels"])
>>> metric.compute()


Additional resources

For more fine-tuning examples, refer to:
🤗 Transformers Examples includes scripts
to train common NLP tasks in PyTorch and TensorFlow. 🤗 Transformers Notebooks contains various notebooks on how to fine-tune a model for specific tasks in PyTorch and TensorFlow.


Using pipelines for a webserver

Creating an inference engine is a complex topic, and the "best" solution
will most likely depend on your problem space. Are you on CPU or GPU? Do
you want the lowest latency, the highest throughput, support for
many models, or just highly optimize 1 specific model?
There are many ways to tackle this topic, so what we are going to present is a good default
to get started which may not necessarily be the most optimal solution for you.
The key thing to understand is that we can use an iterator, just like you would on a
dataset, since a webserver is basically a system that waits for requests and
treats them as they come in.
Usually webservers are multiplexed (multithreaded, async, etc..) to handle various
requests concurrently. Pipelines on the other hand (and mostly the underlying models)
are not really great for parallelism; they take up a lot of RAM, so it’s best to give them all the available resources when they are running or it’s a compute-intensive job.
We are going to solve that


Using pipelines for a webserver

by having the webserver handle the light load of receiving
and sending requests, and having a single thread handling the actual work.
This example is going to use starlette. The actual framework is not really
important, but you might have to tune or change the code if you are using another
one to achieve the same effect.
Create server.py:
  Copied from starlette.applications import Starlette
from starlette.responses import JSONResponse
from starlette.routing import Route
from transformers import pipeline
import asyncio
async def homepage(request):
    payload = await request.body()
    string = payload.decode("utf-8")
    response_q = asyncio.Queue()
    await request.app.model_queue.put((string, response_q))
    output = await response_q.get()
    return JSONResponse(output)
async def server_loop(q):
    pipe = pipeline(model="bert-base-uncased")
    while True:
        (string, response_q) = await q.get()
        out = pipe(string)
        await response_q.put(out)
app = Starlette(


Using pipelines for a webserver

    routes=[
        Route("/", homepage, methods=["POST"]),
    ],
)
@app.on_event("startup")
async def startup_event():
    q = asyncio.Queue()
    app.model_queue = q
    asyncio.create_task(server_loop(q))
Now you can start it with:
  Copied uvicorn server:app
And you can query it:
  Copied curl -X POST -d "test [MASK]" http://localhost:8000/
#[{"score":0.7742936015129089,"token":1012,"token_str":".","sequence":"test."},...]
And there you go, now you have a good idea of how to create a webserver!
What is really important is that we load the model only once, so there are no copies
of the model on the webserver. This way, no unnecessary RAM is being used.
Then the queuing mechanism allows you to do fancy stuff like maybe accumulating a few
items before inferring to use dynamic batching:
The code sample below is intentionally written like pseudo-code for readability.
Do not run this without checking if it makes sense for your system resources!
  Copied (string, rq) = await q.get()
st


Using pipelines for a webserver

rings = []
queues = []
while True:
    try:
        (string, rq) = await asyncio.wait_for(q.get(), timeout=0.001)  # 1ms
    except asyncio.exceptions.TimeoutError:
        break
    strings.append(string)
    queues.append(rq)
strings
outs = pipe(strings, batch_size=len(strings))
for rq, out in zip(queues, outs):
    await rq.put(out)
Again, the proposed code is optimized for readability, not for being the best code.
First of all, there’s no batch size limit which is usually not a
great idea. Next, the timeout is reset on every queue fetch, meaning you could
wait much more than 1ms before running the inference (delaying the first request
by that much).
It would be better to have a single 1ms deadline.
This will always wait for 1ms even if the queue is empty, which might not be the
best since you probably want to start doing inference if there’s nothing in the queue.
But maybe it does make sense if batching is really crucial for your use case.
Again, there’s really no one best solutio


Error checking

There’s a lot that can go wrong in production: out of memory, out of space,
loading the model might fail, the query might be wrong, the query might be
correct but still fail to run because of a model misconfiguration, and so on.
Generally, it’s good if the server outputs the errors to the user, so
adding a lot of try..except statements to show those errors is a good
idea. But keep in mind it may also be a security risk to reveal all those errors depending
on your security context.


Circuit breaking

Webservers usually look better when they do circuit breaking. It means they
return proper errors when they’re overloaded instead of just waiting for the query indefinitely. Return a 503 error instead of waiting for a super long time or a 504 after a long time.
This is relatively easy to implement in the proposed code since there is a single queue.
Looking at the queue size is a basic way to start returning errors before your
webserver fails under load.


Blocking the main thread

Currently PyTorch is not async aware, and computation will block the main
thread while running. That means it would be better if PyTorch was forced to run
on its own thread/process. This wasn’t done here because the code is a lot more
complex (mostly because threads and async and queues don’t play nice together).
But ultimately it does the same thing.
This would be important if the inference of single items were long (> 1s) because
in this case, it means every query during inference would have to wait for 1s before
even receiving an error.


Dynamic batching

In general, batching is not necessarily an improvement over passing 1 item at
a time (see batching details for more information). But it can be very effective
when used in the correct setting. In the API, there is no dynamic
batching by default (too much opportunity for a slowdown). But for BLOOM inference -
which is a very large model - dynamic batching is essential to provide a decent experience for everyone.


Testing

Let’s take a look at how 🤗 Transformers models are tested and how you can write new tests and improve the existing ones.
There are 2 test suites in the repository:
tests — tests for the general API examples — tests primarily for various applications that aren’t part of the API


How transformers are tested

Once a PR is submitted it gets tested with 9 CircleCi jobs. Every new commit to that PR gets retested. These jobs
are defined in this config file, so that if needed you can reproduce the same
environment on your machine. These CI jobs don’t run @slow tests. There are 3 jobs run by github actions: torch hub integration: checks whether torch hub
integration works. self-hosted (push): runs fast tests on GPU only on commits on
main. It only runs if a commit on main has updated the code in one of the following folders: src,
tests, .github (to prevent running on added model cards, notebooks, etc.) self-hosted runner: runs normal and slow tests on GPU in
tests and examples:
  Copied RUN_SLOW=1 pytest tests/
RUN_SLOW=1 pytest examples/
The results can be observed here.


Choosing which tests to run

This document goes into many details of how tests can be run. If after reading everything, you need even more details
you will find them here.
Here are some most useful ways of running tests.
Run all:
  Copied pytest
or:
  Copied make test
Note that the latter is defined as:
  Copied python -m pytest -n auto --dist=loadfile -s -v ./tests/
which tells pytest to:
run as many test processes as they are CPU cores (which could be too many if you don’t have a ton of RAM!) ensure that all tests from the same file will be run by the same test process do not capture output run in verbose mode


Getting the list of all tests

All tests of the test suite:
  Copied pytest --collect-only -q
All tests of a given test file:
  Copied pytest tests/test_optimization.py --collect-only -q


Run specific tests

Since unittest is used inside most of the tests, to run specific subtests you need to know the name of the unittest
class containing those tests. For example, it could be:
  Copied pytest tests/test_optimization.py::OptimizationTest::test_adam_w
Here:
tests/test_optimization.py - the file with tests OptimizationTest - the name of the class test_adam_w - the name of the specific test function
If the file contains multiple classes, you can choose to run only tests of a given class. For example:
  Copied pytest tests/test_optimization.py::OptimizationTest
will run all the tests inside that class.
As mentioned earlier you can see what tests are contained inside the OptimizationTest class by running:
  Copied pytest tests/test_optimization.py::OptimizationTest --collect-only -q
You can run tests by keyword exp


Run specific tests

ressions.
To run only tests whose name contains adam:
  Copied pytest -k adam tests/test_optimization.py
Logical and and or can be used to indicate whether all keywords should match or either. not can be used to
negate.
To run all tests except those whose name contains adam:
  Copied pytest -k "not adam" tests/test_optimization.py
And you can combine the two patterns in one:
  Copied pytest -k "ada and not adam" tests/test_optimization.py
For example to run both test_adafactor and test_adam_w you can use:
  Copied pytest -k "test_adam_w or test_adam_w" tests/test_optimization.py
Note that we use or here, since we want either of the keywords to match to include both.
If you want to include only tests that include both patterns, and is to be used:
  Copied pytest -k "test and ada" tests/test_optimization.py


Run accelerate tests

Sometimes you need to run accelerate tests on your models. For that you can just add -m accelerate_tests to your command, if let’s say you want to run these tests on OPT run:
  Copied RUN_SLOW=1 pytest -m accelerate_tests tests/models/opt/test_modeling_opt.py


Run documentation tests

In order to test whether the documentation examples are correct, you should check that the doctests are passing.
As an example, let’s use WhisperModel.forward’s docstring:
  Copied r"""
Returns:
Example:
    ```python
    >>> import torch
    >>> from transformers import WhisperModel, WhisperFeatureExtractor
    >>> from datasets import load_dataset
    >>> model = WhisperModel.from_pretrained("openai/whisper-base")
    >>> feature_extractor = WhisperFeatureExtractor.from_pretrained("openai/whisper-base")
    >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
    >>>


Run documentation tests

 inputs = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt")
    >>> input_features = inputs.input_features
    >>> decoder_input_ids = torch.tensor([[1, 1]]) * model.config.decoder_start_token_id
    >>> last_hidden_state = model(input_features, decoder_input_ids=decoder_input_ids).last_hidden_state
    >>> list(last_hidden_state.shape)
    [1, 2, 512]
    ```"""
Just run the following line to automatically test every docstring example in the desired file:
  Copied pytest --doctest-modules <path_to_file_or_dir>
If the file has a markdown extention, you should add the --doctest-glob="*.md" argument


Run only modified tests

You can run the tests related to the unstaged files or the current branch (according to Git) by using pytest-picked. This is a great way of quickly testing your changes didn’t break
anything, since it won’t run the tests related to files you didn’t touch.
  Copied pip install pytest-picked
  Copied pytest --picked
All tests will be run from files and folders which are modified, but not yet committed.


Automatically rerun failed tests on source modification

pytest-xdist provides a very useful feature of detecting all failed
tests, and then waiting for you to modify files and continuously re-rerun those failing tests until they pass while you
fix them. So that you don’t need to re start pytest after you made the fix. This is repeated until all tests pass after
which again a full run is performed.
  Copied pip install pytest-xdist
To enter the mode: pytest -f or pytest --looponfail
File changes are detected by looking at looponfailroots root directories and all


 of their contents (recursively).
If the default for this value does not work for you, you can change it in your project by setting a configuration
option in setup.cfg:
  Copied [tool:pytest]
looponfailroots = transformers tests
or pytest.ini/tox.ini files:
  Copied [pytest]
looponfailroots = transformers tests
This would lead to only looking for file changes in the respective directories, specified relatively to the ini-file’s
directory.
pytest-watch is an alternative implementation of this functionality.


Skip a test module

If you want to run all test modules, except a few you can exclude them by giving an explicit list of tests to run. For
example, to run all except test_modeling_*.py tests:
  Copied pytest *ls -1 tests/*py | grep -v test_modeling*


Clearing state

CI builds and when isolation is important (against speed), cache should be cleared:
  Copied pytest --cache-clear tests


Running tests in parallel

As mentioned earlier make test runs tests in parallel via pytest-xdist plugin (-n X argument, e.g. -n 2
to run 2 parallel jobs).
pytest-xdist’s --dist= option allows one to control how the tests are grouped. --dist=loadfile puts the
tests located in one file onto the same process.
Since the order of executed tests is different and unpredictable, if running the test suite with pytest-xdist
produces failures (meaning we have some undetected coupled tests), use pytest-replay to replay the tests in the same order, which should help with then somehow
reducing that failing sequence to a minimum.


Test order and repetition

It’s good to repeat the tests several times, in sequence, randomly, or in sets, to detect any potential
inter-dependency and state-related bugs (tear down). And the straightforward multiple repetition is just good to detect
some problems that get uncovered by randomness of DL.
 Repeat tests
pytest-flakefinder:
  Copied pip install pytest-flakefinder
And then run every test multiple times (50 by default):
  Copied pytest --flake-finder --flake-runs=5 tests/test_failing_test.py
This plugin doesn’t work with -n flag from pytest-xdist.
There is another plugin pytest-repeat, but it doesn’t work with unittest.
 Run tests in a random order
  Copied pip install pytest-random-order
Important: the presence of pytest-random-order will automatically randomize tests, no configu


Test order and repetition

ration change or
command line options is required.
As explained earlier this allows detection of coupled tests - where one test’s state affects the state of another. When
pytest-random-order is installed it will print the random seed it used for that session, e.g:
  Copied pytest tests
[...]
Using --random-order-bucket=module
Using --random-order-seed=573663
So that if the given particular sequence fails, you can reproduce it by adding that exact seed, e.g.:
  Copied pytest --random-order-seed=573663
[...]
Using --random-order-bucket=module
Using --random-order-seed=573663
It will only reproduce the exact order if you use the exact same list of tests (or no list at all). Once you start to
manually narrowing down the list you can no longer rely on the seed, but have


Test order and repetition

 to list them manually in the exact order
they failed and tell pytest to not randomize them instead using --random-order-bucket=none, e.g.:
  Copied pytest --random-order-bucket=none tests/test_a.py tests/test_c.py tests/test_b.py
To disable the shuffling for all tests:
  Copied pytest --random-order-bucket=none
By default --random-order-bucket=module is implied, which will shuffle the files on the module levels. It can also
shuffle on class, package, global and none levels. For the complete details please see its
documentation.
Another randomization alternative is: pytest-randomly. This
module has a very similar functionality/interface, but it doesn’t have the bucket modes available in
pytest-random-order. It has the same problem of imposing itself once installed.


Look and feel variations

pytest-sugar
pytest-sugar is a plugin that improves the look-n-feel, adds a
progressbar, and show tests that fail and the assert instantly. It gets activated automatically upon installation.
  Copied pip install pytest-sugar
To run tests without it, run:
  Copied pytest -p no:sugar
or uninstall it.
 Report each sub-test name and its progress
For a single or a group of tests via pytest (after pip install pytest-pspec):
  Copied pytest --pspec tests/test_optimization.py
 Instantly shows failed tests
pytest-instafail shows failures and errors instantly instead of
waiting until the end of test session.
  Copied pip install pytest-instafail
  Copied pytest --instafail


To GPU or not to GPU

On a GPU-enabled setup, to test in CPU-only mode add CUDA_VISIBLE_DEVICES="":
  Copied CUDA_VISIBLE_DEVICES="" pytest tests/utils/test_logging.py
or if you have multiple gpus, you can specify which one is to be used by pytest. For example, to use only the
second gpu if you have gpus 0 and 1, you can run:
  Copied CUDA_VISIBLE_DEVICES="1" pytest tests/utils/test_logging.py
This is handy when you want to run different tasks on different GPUs.
Some tests must be run on CPU-only, others on either CPU or GPU or TPU, yet others on multiple-GPUs. The following skip
decorators are used to set the requirements of tests CPU/GPU/TPU-wise:
require_torch - this test will run only under torch require_torch_gpu - as require_torch plus requires at least 1 GPU require_torch_multi_gpu - as require_torch plus requires at least 2 GPUs re


To GPU or not to GPU

quire_torch_non_multi_gpu - as require_torch plus requires 0 or 1 GPUs require_torch_up_to_2_gpus - as require_torch plus requires 0 or 1 or 2 GPUs require_torch_tpu - as require_torch plus requires at least 1 TPU
Let’s depict the GPU requirements in the following table:
| n gpus | decorator                      |
|--------+--------------------------------|
| >= 0 | @require_torch               |
| >= 1 | @require_torch_gpu           |
| >= 2 | @require_torch_multi_gpu     |
| < 2  | @require_torch_non_multi_gpu |
| < 3  | @require_torch_up_to_2_gpus  |
For example, here is a test that must be run only when there are 2 or more GPUs available and pytorch is installed:
  Copied @require_torch_multi_gpu
def test_example_with_multi_gpu():
If a test requires tensorflow use the require_tf decorator. For example:
  Copied @r


To GPU or not to GPU

equire_tf
def test_tf_thing_with_tensorflow():
These decorators can be stacked. For example, if a test is slow and requires at least one GPU under pytorch, here is
how to set it up:
  Copied @require_torch_gpu
@slow
def test_example_slow_on_gpu():
Some decorators like @parametrized rewrite test names, therefore @require_* skip decorators have to be listed
last for them to work correctly. Here is an example of the correct usage:
  Copied @parameterized.expand(...)
@require_torch_multi_gpu
def test_integration_foo():
This order problem doesn’t exist with @pytest.mark.parametrize, you can put it first or last and it will still
work. But it only works with non-unittests.
Inside tests:
How many GPUs are available:
  Copied from transformers.testing_utils import get_gpu_count
n_gpu = get_gpu_count()  # works with torch and


Testing with a specific PyTorch backend or device

To run the test suite on a specific torch device add TRANSFORMERS_TEST_DEVICE="$device" where $device is the target backend. For example, to test on CPU only:
  Copied TRANSFORMERS_TEST_DEVICE="cpu" pytest tests/utils/test_logging.py
This variable is useful for testing custom or less common PyTorch backends such as mps. It can also be used to achieve the same effect as CUDA_VISIBLE_DEVICES by targeting specific GPUs or testing in CPU-only mode.
Certain devices will require an additional import after importing torch for the first time. This can be specified using the environment variable TRANSFORMERS_TEST_BACKEND:
  Copied TRANSFORMERS_TEST_BACKEND="torch_npu" pytest tests/utils/test_logging.py


Distributed training

pytest can’t deal with distributed training directly. If this is attempted - the sub-processes don’t do the right
thing and end up thinking they are pytest and start running the test suite in loops. It works, however, if one
spawns a normal process that then spawns off multiple workers and manages the IO pipes.
Here are some tests that use it:
test_trainer_distributed.py test_deepspeed.py
To jump right into the execution point, search for the execute_subprocess_async call in those tests.
You will need at least 2 GPUs to see these tests in action:
  Copied CUDA_VISIBLE_DEVICES=0,1 RUN_SLOW=1 pytest -sv tests/test_trainer_distributed.py


Output capture

During test execution any output sent to stdout and stderr is captured. If a test or a setup method fails, its
according captured output will usually be shown along with the failure traceback.
To disable output capturing and to get the stdout and stderr normally, use -s or --capture=no:
  Copied pytest -s tests/utils/test_logging.py
To send test results to JUnit format output:
  Copied py.test tests --junitxml=result.xml


Color control

To have no color (e.g., yellow on white background is not readable):
  Copied pytest --color=no tests/utils/test_logging.py


Sending test report to online pastebin service

Creating a URL for each test failure:
  Copied pytest --pastebin=failed tests/utils/test_logging.py
This will submit test run information to a remote Paste service and provide a URL for each failure. You may select
tests as usual or add for example -x if you only want to send one particular failure.
Creating a URL for a whole test session log:
  Copied pytest --pastebin=all tests/utils/test_logging.py


Writing tests

🤗 transformers tests are based on unittest, but run by pytest, so most of the time features from both systems
can be used.
You can read here which features are supported, but the important
thing to remember is that most pytest fixtures don’t work. Neither parametrization, but we use the module
parameterized that works in a similar way.


Parametrization

Often, there is a need to run the same test multiple times, but with different arguments. It could be done from within
the test, but then there is no way of running that test for just one set of arguments.
  Copied # test_this1.py
import unittest
from parameterized import parameterized
class TestMathUnitTest(unittest.TestCase):
    @parameterized.expand(
        [
            ("negative", -1.5, -2.0),
            ("integer", 1, 1.0),
            ("large fraction", 1.6, 1),
        ]
    )
    def test_floor(self, name, input, expected):
        assert_equal(math.floor(input), expected)
Now, by default this test will be run 3 times, each time with the last 3 arguments of test_floor being assigned the
corresponding arguments in the parameter list.
and you could run just the negative and integer sets of params with:
  Copied pytest -k "negative and integer" tests/test_mytest.py
or all but negative sub-tests, with:
  Copied pytest -k "not negati


Parametrization

ve" tests/test_mytest.py
Besides using the -k filter that was just mentioned, you can find out the exact name of each sub-test and run any
or all of them using their exact names.
  Copied pytest test_this1.py --collect-only -q
and it will list:
  Copied test_this1.py::TestMathUnitTest::test_floor_0_negative
test_this1.py::TestMathUnitTest::test_floor_1_integer
test_this1.py::TestMathUnitTest::test_floor_2_large_fraction
So now you can run just 2 specific sub-tests:
  Copied pytest test_this1.py::TestMathUnitTest::test_floor_0_negative  test_this1.py::TestMathUnitTest::test_floor_1_integer
The module parameterized which is already in the developer dependencies
of transformers works for both: unittests and pytest tests.
If, however, the test is not a unittest, you may use pytest.mark.parametrize (or you may see it being used in
some existing tests, mostly under examples).
Here is the same example, this time using pytest’s parametrize marker:


Parametrization

 Copied # test_this2.py
import pytest
@pytest.mark.parametrize(
    "name, input, expected",
    [
        ("negative", -1.5, -2.0),
        ("integer", 1, 1.0),
        ("large fraction", 1.6, 1),
    ],
)
def test_floor(name, input, expected):
    assert_equal(math.floor(input), expected)
Same as with parameterized, with pytest.mark.parametrize you can have a fine control over which sub-tests are
run, if the -k filter doesn’t do the job. Except, this parametrization function creates a slightly different set of
names for the sub-tests. Here is what they look like:
  Copied pytest test_this2.py --collect-only -q
and it will list:
  Copied test_this2.py::test_floor[integer-1-1.0]
test_this2.py::test_floor[negative--1.5--2.0]
test_this2.py::test_floor[large fraction-1.6-1]
So now you can run just the specific test:
  Copied pytest test_this2.py::test_floor[negative--1.5--2.0] test_this2.py::test_floor[integer-1-1.0]
as in the previous example.


Files and directories

In tests often we need to know where things are relative to the current test file, and it’s not trivial since the test
could be invoked from more than one directory or could reside in sub-directories with different depths. A helper class
transformers.test_utils.TestCasePlus solves this problem by sorting out all the basic paths and provides easy
accessors to them:
pathlib objects (all fully resolved): test_file_path - the current test file path, i.e. __file__ test_file_dir - the directory containing the current test file tests_dir - the directory of the tests test suite examples_dir - the directory of the examples test suite repo_root_dir - the directory of the repository src_dir - the directory of src (i.e. where the transformers sub-dir resides) stringified paths---same as above but these return paths as strings, rather than pathlib objects:


Files and directories

test_file_path_str test_file_dir_str tests_dir_str examples_dir_str repo_root_dir_str src_dir_str
To start using those all you need is to make sure that the test resides in a subclass of
transformers.test_utils.TestCasePlus. For example:
  Copied from transformers.testing_utils import TestCasePlus
class PathExampleTest(TestCasePlus):
    def test_something_involving_local_locations(self):
        data_dir = self.tests_dir / "fixtures/tests_samples/wmt_en_ro"
If you don’t need to manipulate paths via pathlib or you just need a path as a string, you can always invoked
str() on the pathlib object or use the accessors ending with _str. For example:
  Copied from transformers.testing_utils import TestCasePlus
class PathExampleTest(TestCasePlus):
    def test_something_involving_stringified_locations(self):
        examples_dir = self.examples_dir_st


Temporary files and directories

Using unique temporary files and directories are essential for parallel test running, so that the tests won’t overwrite
each other’s data. Also we want to get the temporary files and directories removed at the end of each test that created
them. Therefore, using packages like tempfile, which address these needs is essential.
However, when debugging tests, you need to be able to see what goes into the temporary file or directory and you want
to know it’s exact path and not having it randomized on every test re-run.
A helper class transformers.test_utils.TestCasePlus is best used for such purposes. It’s a sub-class of
unittest.TestCase, so we can easily inherit from it in the test modules.
Here is an example of its usage:
  Copied from transformers.testing_utils import TestCasePlus
class Example


Temporary files and directories

sTests(TestCasePlus):
    def test_whatever(self):
        tmp_dir = self.get_auto_remove_tmp_dir()
This code creates a unique temporary directory, and sets tmp_dir to its location.
Create a unique temporary dir:
  Copied def test_whatever(self):
    tmp_dir = self.get_auto_remove_tmp_dir()
tmp_dir will contain the path to the created temporary dir. It will be automatically removed at the end of the
test.
Create a temporary dir of my choice, ensure it’s empty before the test starts and don’t empty it after the test.
  Copied def test_whatever(self):
    tmp_dir = self.get_auto_remove_tmp_dir("./xxx")
This is useful for debug when you want to monitor a specific directory and want to make sure the previous tests didn’t
leave any data in there.
You can override the default behavior by directly ov


Temporary files and directories

erriding the before and after args, leading to one of the
following behaviors: before=True: the temporary dir will always be cleared at the beginning of the test. before=False: if the temporary dir already existed, any existing files will remain there. after=True: the temporary dir will always be deleted at the end of the test. after=False: the temporary dir will always be left intact at the end of the test.
In order to run the equivalent of rm -r safely, only subdirs of the project repository checkout are allowed if
an explicit tmp_dir is used, so that by mistake no /tmp or similar important part of the filesystem will
get nuked. i.e. please always pass paths that start with ./.
Each test can register multiple temporary directories and they all will get auto-removed, unless requested
otherwis


Temporary sys.path override

If you need to temporary override sys.path to import from another test for example, you can use the
ExtendSysPath context manager. Example:
  Copied import os
from transformers.testing_utils import ExtendSysPath
bindir = os.path.abspath(os.path.dirname(__file__))
with ExtendSysPath(f"{bindir}/.."):
    from test_trainer import TrainerIntegrationCommon  # noqa


Skipping tests

This is useful when a bug is found and a new test is written, yet the bug is not fixed yet. In order to be able to
commit it to the main repository we need make sure it’s skipped during make test.
Methods:
A skip means that you expect your test to pass only if some conditions are met, otherwise pytest should skip
running the test altogether. Common examples are skipping windows-only tests on non-windows platforms, or skipping
tests that depend on an external resource which is not available at the moment (for example a database). A xfail means that you expect a test to fail for some reason. A common example is a test for a feature not yet
implemented, or a bug not yet fixed. When a test passes despite being expected to fail (marked with
pytest.mark.xfail), it’s an xpass and will


Skipping tests

be reported in the test summary.
One of the important differences between the two is that skip doesn’t run the test, and xfail does. So if the
code that’s buggy causes some bad state that will affect other tests, do not use xfail.
 Implementation
Here is how to skip whole test unconditionally:
  Copied @unittest.skip("this bug needs to be fixed")
def test_feature_x():
or via pytest:
  Copied @pytest.mark.skip(reason="this bug needs to be fixed")
or the xfail way:
  Copied @pytest.mark.xfail
def test_feature_x():
Here is how to skip a test based on some internal check inside the test:
  Copied def test_feature_x():
    if not has_something():
        pytest.skip("unsupported configuration")
or the whole module:
  Copied import pytest
if not pytest.config.getoption("--custom-flag"


Skipping tests

):
    pytest.skip("--custom-flag is missing, skipping tests", allow_module_level=True)
or the xfail way:
  Copied def test_feature_x():
    pytest.xfail("expected to fail until bug XYZ is fixed")
Here is how to skip all tests in a module if some import is missing:
  Copied docutils = pytest.importorskip("docutils", minversion="0.3")
Skip a test based on a condition:
  Copied @pytest.mark.skipif(sys.version_info < (3,6), reason="requires python3.6 or higher")
def test_feature_x():
or:
  Copied @unittest.skipIf(torch_device == "cpu", "Can't do half precision")
def test_feature_x():
or skip the whole module:
  Copied @pytest.mark.skipif(sys.platform == 'win32', reason="does not run on windows")
class TestClass():
    def test_feature_x(self):
More details, example and ways are her


Slow tests

The library of tests is ever-growing, and some of the tests take minutes to run, therefore we can’t afford waiting for
an hour for the test suite to complete on CI. Therefore, with some exceptions for essential tests, slow tests should be
marked as in the example below:
  Copied from transformers.testing_utils import slow
@slow
def test_integration_foo():
Once a test is marked as @slow, to run such tests set RUN_SLOW=1 env var, e.g.:
  Copied RUN_SLOW=1 pytest tests
Some decorators like @parameterized rewrite test names, therefore @slow and the rest of the skip decorators
@require_* have to be listed last for them to work correctly. Here is an example of the correct usage:
  Copied @parameteriz ed.expand(...)
@slow
def test_integration_foo():
As explained at the beginning of this document, slow tests get to run on a scheduled basis, rather than in PRs CI
checks. So it’s possible that some pro


Slow tests

blems will be missed during a PR submission and get merged. Such problems will
get caught during the next scheduled CI job. But it also means that it’s important to run the slow tests on your
machine before submitting the PR.
Here is a rough decision making mechanism for choosing which tests should be marked as slow:
If the test is focused on one of the library’s internal components (e.g., modeling files, tokenization files,
pipelines), then we should run that test in the non-slow test suite. If it’s focused on an other aspect of the library,
such as the documentation or the examples, then we should run these tests in the slow test suite. And then, to refine
this approach we should have exceptions:
All tests that need to download a heavy set of weights or a dataset that is larger than ~50MB (e.g., model or
tokenizer integration tests, pipeline integration tests) should be set to slow. If you’


Slow tests

re adding a new model, you
should create and upload to the hub a tiny version of it (with random weights) for integration tests. This is
discussed in the following paragraphs. All tests that need to do a training not specifically optimized to be fast should be set to slow. We can introduce exceptions if some of these should-be-non-slow tests are excruciatingly slow, and set them to
@slow. Auto-modeling tests, which save and load large files to disk, are a good example of tests that are marked
as @slow. If a test completes under 1 second on CI (including downloads if any) then it should be a normal test regardless.
Collectively, all the non-slow tests need to cover entirely the different internals, while remaining fast. For example,
a significant coverage can be achieved by testing with specially created tiny models with random weights. Such models
have the very minimal number of layers (e.g.,


Slow tests

 2), vocab size (e.g., 1000), etc. Then the @slow tests can use large
slow models to do qualitative testing. To see the use of these simply look for tiny models with:
  Copied grep tiny tests examples
Here is a an example of a script that created the tiny model
stas/tiny-wmt19-en-de. You can easily adjust it to your specific
model’s architecture.
It’s easy to measure the run-time incorrectly if for example there is an overheard of downloading a huge model, but if
you test it locally the downloaded files would be cached and thus the download time not measured. Hence check the
execution speed report in CI logs instead (the output of pytest --durations=0 tests).
That report is also useful to find slow outliers that aren’t marked as such, or which need to be re-written to be fast.
If you notice that the test suite starts getting slow on CI, the top listing of this report will show the slowest
tes


Testing the stdout/stderr output

In order to test functions that write to stdout and/or stderr, the test can access those streams using the
pytest’s capsys system. Here is how this is accomplished:
  Copied import sys
def print_to_stdout(s):
    print(s)
def print_to_stderr(s):
    sys.stderr.write(s)
def test_result_and_stdout(capsys):
    msg = "Hello"
    print_to_stdout(msg)
    print_to_stderr(msg)
    out, err = capsys.readouterr()  # consume the captured output streams
    # optional: if you want to replay the consumed streams:
    sys.stdout.write(out)
    sys.stderr.write(err)
    # test:
    assert msg in out
    assert msg in err
And, of course, most of the time, stderr will come as a part of an exception, so try/except has to be used in such
a case:
  Copied def raise_exc


Testing the stdout/stderr output

eption(msg):
    raise ValueError(msg)
def test_something_exception():
    msg = "Not a good value"
    error = ""
    try:
        raise_exception(msg)
    except Exception as e:
        error = str(e)
        assert msg in error, f"{msg} is in the exception:\n{error}"
Another approach to capturing stdout is via contextlib.redirect_stdout:
  Copied from io import StringIO
from contextlib import redirect_stdout
def print_to_stdout(s):
    print(s)
def test_result_and_stdout():
    msg = "Hello"
    buffer = StringIO()
    with redirect_stdout(buffer):
        print_to_stdout(msg)
    out = buffer.getvalue()
    # optional: if you want to replay the consumed streams:
    sys.stdout.write(out)
    # test:
    assert msg in out
An important potential iss