3.-4. Run a pretrained checkpoint using the original repository

At first, you will work on the original brand_new_bert repository. Often, the original implementation is very
“researchy”. Meaning that documentation might be lacking and the code can be difficult to understand. But this should
be exactly your motivation to reimplement brand_new_bert. At Hugging Face, one of our main goals is to make people
stand on the shoulders of giants which translates here very well into taking a working model and rewriting it to make
it as accessible, user-friendly, and beautiful as possible. This is the number-one motivation to re-implement
models into 🤗 Transformers - trying to make complex new NLP technology accessible to everybody.
You should start thereby by diving into the original repository.
Successfully running the official pretrained model in the original repository is often the most difficult step.
From our experience, it is very important to spend some time getting familiar with the original code-base. You


need to
figure out the following:
Where to find the pretrained weights? How to load the pretrained weights into the corresponding model? How to run the tokenizer independently from the model? Trace one forward pass so that you know which classes and functions are required for a simple forward pass. Usually,
you only have to reimplement those functions. Be able to locate the important components of the model: Where is the model’s class? Are there model sub-classes,
e.g. EncoderModel, DecoderModel? Where is the self-attention layer? Are there multiple different attention layers,
e.g. self-attention, cross-attention…? How can you debug the model in the original environment of the repo? Do you have to add print statements, can you
work with an interactive debugger like ipdb, or should you use an efficient IDE to debug the model, like PyCharm?
It is very important that before you start the porting process, you can efficiently debug code in the or


iginal
repository! Also, remember that you are working with an open-source library, so do not hesitate to open an issue, or
even a pull request in the original repository. The maintainers of this repository are most likely very happy about
someone looking into their code!
At this point, it is really up to you which debugging environment and strategy you prefer to use to debug the original
model. We strongly advise against setting up a costly GPU environment, but simply work on a CPU both when starting to
dive into the original repository and also when starting to write the 🤗 Transformers implementation of the model. Only
at the very end, when the model has already been successfully ported to 🤗 Transformers, one should verify that the
model also works as expected on GPU.
In general, there are two possible debugging environments for running the original model
Jupyter notebooks / google colab Local python scripts.
Jupyter notebooks have the adv


antage that they allow for cell-by-cell execution which can be helpful to better split
logical components from one another and to have faster debugging cycles as intermediate results can be stored. Also,
notebooks are often easier to share with other contributors, which might be very helpful if you want to ask the Hugging
Face team for help. If you are familiar with Jupyter notebooks, we strongly recommend you work with them.
The obvious disadvantage of Jupyter notebooks is that if you are not used to working with them you will have to spend
some time adjusting to the new programming environment and you might not be able to use your known debugging tools
anymore, like ipdb.
For each code-base, a good first step is always to load a small pretrained checkpoint and to be able to reproduce a
single forward pass using a dummy integer vector of input IDs as an input. Such a script could look like this (in
pseudocode):
  Copied model = BrandNewBert


Model.load_pretrained_checkpoint("/path/to/checkpoint/")
input_ids = [0, 4, 5, 2, 3, 7, 9]  # vector of input ids
original_output = model.predict(input_ids)
Next, regarding the debugging strategy, there are generally a few from which to choose from:
Decompose the original model into many small testable components and run a forward pass on each of those for
verification Decompose the original model only into the original tokenizer and the original model, run a forward pass on
those, and use intermediate print statements or breakpoints for verification
Again, it is up to you which strategy to choose. Often, one or the other is advantageous depending on the original code
base.
If the original code-base allows you to decompose the model into smaller sub-components, e.g. if the original
code-base can easily be run in eager mode, it is usually worth the effort to do so. There are some important advantages
to taking the more difficult road in the b


eginning:
at a later stage when comparing the original model to the Hugging Face implementation, you can verify automatically
for each component individually that the corresponding component of the 🤗 Transformers implementation matches instead
of relying on visual comparison via print statements it can give you some rope to decompose the big problem of porting a model into smaller problems of just porting
individual components and thus structure your work better separating the model into logical meaningful components will help you to get a better overview of the model’s design
and thus to better understand the model at a later stage those component-by-component tests help you to ensure that no regression occurs as you continue
changing your code
Lysandre’s integration checks for ELECTRA
gives a nice example of how this can be done.
However, if the original code-base is very complex or only allows intermediate components to be run in a compil


ed mode,
it might be too time-consuming or even impossible to separate the model into smaller testable sub-components. A good
example is T5’s MeshTensorFlow library which is
very complex and does not offer a simple way to decompose the model into its sub-components. For such libraries, one
often relies on verifying print statements.
No matter which strategy you choose, the recommended procedure is often the same that you should start to debug the
starting layers first and the ending layers last.
It is recommended that you retrieve the output, either by print statements or sub-component functions, of the following
layers in the following order:
Retrieve the input IDs passed to the model Retrieve the word embeddings Retrieve the input of the first Transformer layer Retrieve the output of the first Transformer layer Retrieve the output of the following n - 1 Transformer layers Retrieve the output of the whole BrandNewBert Model
Input IDs should


 thereby consists of an array of integers, e.g. input_ids = [0, 4, 4, 3, 2, 4, 1, 7, 19]
The outputs of the following layers often consist of multi-dimensional float arrays and can look like this:
  Copied [[
 [-0.1465, -0.6501,  0.1993,  ...,  0.1451,  0.3430,  0.6024],
 [-0.4417, -0.5920,  0.3450,  ..., -0.3062,  0.6182,  0.7132],
 [-0.5009, -0.7122,  0.4548,  ..., -0.3662,  0.6091,  0.7648],
 ...,
 [-0.5613, -0.6332,  0.4324,  ..., -0.3792,  0.7372,  0.9288],
 [-0.5416, -0.6345,  0.4180,  ..., -0.3564,  0.6992,  0.9191],
 [-0.5334, -0.6403,  0.4271,  ..., -0.3339,  0.6533,  0.8694]]],
We expect that every model added to 🤗 Transformers passes a couple of integration tests, meaning that the original
model and the reimplemented version in 🤗 Transformers have to give the exact same output up to a precision of 0.001!
Since it is normal that the exact same model written in different libraries can give a slightly different output
depending on th


e library framework, we accept an error tolerance of 1e-3 (0.001). It is not enough if the model gives
nearly the same output, they have to be almost identical. Therefore, you will certainly compare the intermediate
outputs of the 🤗 Transformers version multiple times against the intermediate outputs of the original implementation of
brand_new_bert in which case an efficient debugging environment of the original repository is absolutely
important. Here is some advice to make your debugging environment as efficient as possible.
Find the best way of debugging intermediate results. Is the original repository written in PyTorch? Then you should
probably take the time to write a longer script that decomposes the original model into smaller sub-components to
retrieve intermediate values. Is the original repository written in Tensorflow 1? Then you might have to rely on
TensorFlow print operations like tf.print to output
intermediate values. Is the


 original repository written in Jax? Then make sure that the model is not jitted when
running the forward pass, e.g. check-out this link. Use the smallest pretrained checkpoint you can find. The smaller the checkpoint, the faster your debug cycle
becomes. It is not efficient if your pretrained model is so big that your forward pass takes more than 10 seconds.
In case only very large checkpoints are available, it might make more sense to create a dummy model in the new
environment with randomly initialized weights and save those weights for comparison with the 🤗 Transformers version
of your model Make sure you are using the easiest way of calling a forward pass in the original repository. Ideally, you want to
find the function in the original repository that only calls a single forward pass, i.e. that is often called
predict, evaluate, forward or __call__. You don’t want to debug a function that calls forward
multiple times, e.g. to generate


text, like autoregressive_sample, generate. Try to separate the tokenization from the model’s forward pass. If the original repository shows examples where
you have to input a string, then try to find out where in the forward call the string input is changed to input ids
and start from this point. This might mean that you have to possibly write a small script yourself or change the
original code so that you can directly input the ids instead of an input string. Make sure that the model in your debugging setup is not in training mode, which often causes the model to yield
random outputs due to multiple dropout layers in the model. Make sure that the forward pass in your debugging
environment is deterministic so that the dropout layers are not used. Or use transformers.utils.set_seed
if the old and new implementations are in the same framework.
The following section gives you more specific details/tips on how you can do this for brand_new_bert


5.-14. Port BrandNewBert to 🤗 Transformers

Next, you can finally start adding new code to 🤗 Transformers. Go into the clone of your 🤗 Transformers’ fork:
  Copied cd transformers
In the special case that you are adding a model whose architecture exactly matches the model architecture of an
existing model you only have to add a conversion script as described in this section.
In this case, you can just re-use the whole model architecture of the already existing model.
Otherwise, let’s start generating a new model. You have two choices here:
transformers-cli add-new-model-like to add a new model like an existing one transformers-cli add-new-model to add a new model from our template (will look like BERT or Bart depending on the type of model you select)
In both cases, you will be prompted with a questionnaire to fill in the basic information of your model. The second command requires to install cookiecutter, you can find more information on it here.
Open a Pull Request on the main huggingface/transformers rep


o
Before starting to adapt the automatically generated code, now is the time to open a “Work in progress (WIP)” pull
request, e.g. “[WIP] Add brand_new_bert”, in 🤗 Transformers so that you and the Hugging Face team can work
side-by-side on integrating the model into 🤗 Transformers.
You should do the following:
Create a branch with a descriptive name from your main branch
  Copied git checkout -b add_brand_new_bert
Commit the automatically generated code:
  Copied git add .
git commit
Fetch and rebase to current main
  Copied git fetch upstream
git rebase upstream/main
Push the changes to your account using:
  Copied git push -u origin a-descriptive-name-for-my-changes
Once you are satisfied, go to the webpage of your fork on GitHub. Click on “Pull request”. Make sure to add the
GitHub handle of some members of the Hugging Face team as reviewers, so that the Hugging Face team gets notified for
future changes. Change the PR into a draft by clicking on “Convert to dr


aft” on the right of the GitHub pull request web page.
In the following, whenever you have made some progress, don’t forget to commit your work and push it to your account so
that it shows in the pull request. Additionally, you should make sure to update your work with the current main from
time to time by doing:
  Copied git fetch upstream
git merge upstream/main
In general, all questions you might have regarding the model or your implementation should be asked in your PR and
discussed/solved in the PR. This way, the Hugging Face team will always be notified when you are committing new code or
if you have a question. It is often very helpful to point the Hugging Face team to your added code so that the Hugging
Face team can efficiently understand your problem or question.
To do so, you can go to the “Files changed” tab where you see all of your changes, go to a line regarding which you
want to ask a question, and click on the “+” symbol to add a comment. Whenever


 a question or problem has been solved,
you can click on the “Resolve” button of the created comment.
In the same way, the Hugging Face team will open comments when reviewing your code. We recommend asking most questions
on GitHub on your PR. For some very general questions that are not very useful for the public, feel free to ping the
Hugging Face team by Slack or email.
5. Adapt the generated models code for brand_new_bert
At first, we will focus only on the model itself and not care about the tokenizer. All the relevant code should be
found in the generated files src/transformers/models/brand_new_bert/modeling_brand_new_bert.py and
src/transformers/models/brand_new_bert/configuration_brand_new_bert.py.
Now you can finally start coding :). The generated code in
src/transformers/models/brand_new_bert/modeling_brand_new_bert.py will either have the same architecture as BERT if
it’s an encoder-only model or BART if it’s an encoder-decoder model. At this point, you


should remind yourself what
you’ve learned in the beginning about the theoretical aspects of the model: How is the model different from BERT or
BART?”. Implement those changes which often means changing the self-attention layer, the order of the normalization
layer, etc… Again, it is often useful to look at the similar architecture of already existing models in Transformers to
get a better feeling of how your model should be implemented.
Note that at this point, you don’t have to be very sure that your code is fully correct or clean. Rather, it is
advised to add a first unclean, copy-pasted version of the original code to
src/transformers/models/brand_new_bert/modeling_brand_new_bert.py until you feel like all the necessary code is
added. From our experience, it is much more efficient to quickly add a first version of the required code and
improve/correct the code iteratively with the conversion script as described in the next section. The only thing that
has to w


ork at this point is that you can instantiate the 🤗 Transformers implementation of brand_new_bert, i.e. the
following command should work:
  Copied from transformers import BrandNewBertModel, BrandNewBertConfig
model = BrandNewBertModel(BrandNewBertConfig())
The above command will create a model according to the default parameters as defined in BrandNewBertConfig() with
random weights, thus making sure that the init() methods of all components works.
Note that all random initialization should happen in the _init_weights method of your BrandnewBertPreTrainedModel
class. It should initialize all leaf modules depending on the variables of the config. Here is an example with the
BERT _init_weights method:
  Copied def _init_weights(self, module):
    """Initialize the weights"""
    if isinstance(module, nn.Linear):
        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        if module.bias is not None:
            module.bias.data.zero_()


   elif isinstance(module, nn.Embedding):
        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        if module.padding_idx is not None:
            module.weight.data[module.padding_idx].zero_()
    elif isinstance(module, nn.LayerNorm):
        module.bias.data.zero_()
        module.weight.data.fill_(1.0)
You can have some more custom schemes if you need a special initialization for some modules. For instance, in
Wav2Vec2ForPreTraining, the last two linear layers need to have the initialization of the regular PyTorch nn.Linear
but all the other ones should use an initialization as above. This is coded like this:
  Copied def _init_weights(self, module):
    """Initialize the weights"""
    if isinstnace(module, Wav2Vec2ForPreTraining):
        module.project_hid.reset_parameters()
        module.project_q.reset_parameters()
        module.project_hid._is_hf_initialized = True
        module.project_q._is_hf_initialized = True
    eli


f isinstance(module, nn.Linear):
        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        if module.bias is not None:
            module.bias.data.zero_()
The _is_hf_initialized flag is internally used to make sure we only initialize a submodule once. By setting it to
True for module.project_q and module.project_hid, we make sure the custom initialization we did is not overridden later on,
the _init_weights function won’t be applied to them.
6. Write a conversion script
Next, you should write a conversion script that lets you convert the checkpoint you used to debug brand_new_bert in
the original repository to a checkpoint compatible with your just created 🤗 Transformers implementation of
brand_new_bert. It is not advised to write the conversion script from scratch, but rather to look through already
existing conversion scripts in 🤗 Transformers for one that has been used to convert a similar model that was written in
the same framew


ork as brand_new_bert. Usually, it is enough to copy an already existing conversion script and
slightly adapt it for your use case. Don’t hesitate to ask the Hugging Face team to point you to a similar already
existing conversion script for your model.
If you are porting a model from TensorFlow to PyTorch, a good starting point might be BERT’s conversion script here If you are porting a model from PyTorch to PyTorch, a good starting point might be BART’s conversion script here
In the following, we’ll quickly explain how PyTorch models store layer weights and define layer names. In PyTorch, the
name of a layer is defined by the name of the class attribute you give the layer. Let’s define a dummy model in
PyTorch, called SimpleModel as follows:
  Copied from torch import nn
class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.dense = nn.Linear(10, 10)
        self.intermediate = nn.Linear(10, 10)
        self.layer_norm = nn.


LayerNorm(10)
Now we can create an instance of this model definition which will fill all weights: dense, intermediate,
layer_norm with random weights. We can print the model to see its architecture
  Copied model = SimpleModel()
print(model)
This will print out the following:
  Copied SimpleModel(
  (dense): Linear(in_features=10, out_features=10, bias=True)
  (intermediate): Linear(in_features=10, out_features=10, bias=True)
  (layer_norm): LayerNorm((10,), eps=1e-05, elementwise_affine=True)
)
We can see that the layer names are defined by the name of the class attribute in PyTorch. You can print out the weight
values of a specific layer:
  Copied print(model.dense.weight.data)
to see that the weights were randomly initialized
  Copied tensor([[-0.0818,  0.2207, -0.0749, -0.0030,  0.0045, -0.1569, -0.1598,  0.0212,
         -0.2077,  0.2157],
        [ 0.1044,  0.0201,  0.0990,  0.2482,  0.3116,  0.2509,  0.2866, -0.2190,
          0.2166, -0.0212],
        [-0.


2000,  0.1107, -0.1999, -0.3119,  0.1559,  0.0993,  0.1776, -0.1950,
         -0.1023, -0.0447],
        [-0.0888, -0.1092,  0.2281,  0.0336,  0.1817, -0.0115,  0.2096,  0.1415,
         -0.1876, -0.2467],
        [ 0.2208, -0.2352, -0.1426, -0.2636, -0.2889, -0.2061, -0.2849, -0.0465,
          0.2577,  0.0402],
        [ 0.1502,  0.2465,  0.2566,  0.0693,  0.2352, -0.0530,  0.1859, -0.0604,
          0.2132,  0.1680],
        [ 0.1733, -0.2407, -0.1721,  0.1484,  0.0358, -0.0633, -0.0721, -0.0090,
          0.2707, -0.2509],
        [-0.1173,  0.1561,  0.2945,  0.0595, -0.1996,  0.2988, -0.0802,  0.0407,
          0.1829, -0.1568],
        [-0.1164, -0.2228, -0.0403,  0.0428,  0.1339,  0.0047,  0.1967,  0.2923,
          0.0333, -0.0536],
        [-0.1492, -0.1616,  0.1057,  0.1950, -0.2807, -0.2710, -0.1586,  0.0739,
          0.2220,  0.2358]]).
In the conversion script, you should fill those randomly initialized weights with the exact weights of the
correspon


ding layer in the checkpoint. E.g.
  Copied # retrieve matching layer weights, e.g. by
# recursive algorithm
layer_name = "dense"
pretrained_weight = array_of_dense_layer
model_pointer = getattr(model, "dense")
model_pointer.weight.data = torch.from_numpy(pretrained_weight)
While doing so, you must verify that each randomly initialized weight of your PyTorch model and its corresponding
pretrained checkpoint weight exactly match in both shape and name. To do so, it is necessary to add assert
statements for the shape and print out the names of the checkpoints weights. E.g. you should add statements like:
  Copied assert (
    model_pointer.weight.shape == pretrained_weight.shape
), f"Pointer shape of random weight {model_pointer.shape} and array shape of checkpoint weight {pretrained_weight.shape} mismatched"
Besides, you should also print out the names of both weights to make sure they match, e.g.
  Copied logger.info(f"Initialize PyTorch weight {layer_name} from {


pretrained_weight.name}")
If either the shape or the name doesn’t match, you probably assigned the wrong checkpoint weight to a randomly
initialized layer of the 🤗 Transformers implementation.
An incorrect shape is most likely due to an incorrect setting of the config parameters in BrandNewBertConfig() that
do not exactly match those that were used for the checkpoint you want to convert. However, it could also be that
PyTorch’s implementation of a layer requires the weight to be transposed beforehand.
Finally, you should also check that all required weights are initialized and print out all checkpoint weights that
were not used for initialization to make sure the model is correctly converted. It is completely normal, that the
conversion trials fail with either a wrong shape statement or a wrong name assignment. This is most likely because either
you used incorrect parameters in BrandNewBertConfig(), have a wrong architecture in the 🤗 Transformers
implementation, y


ou have a bug in the init() functions of one of the components of the 🤗 Transformers
implementation or you need to transpose one of the checkpoint weights.
This step should be iterated with the previous step until all weights of the checkpoint are correctly loaded in the
Transformers model. Having correctly loaded the checkpoint into the 🤗 Transformers implementation, you can then save
the model under a folder of your choice /path/to/converted/checkpoint/folder that should then contain both a
pytorch_model.bin file and a config.json file:
  Copied model.save_pretrained("/path/to/converted/checkpoint/folder")
7. Implement the forward pass
Having managed to correctly load the pretrained weights into the 🤗 Transformers implementation, you should now make
sure that the forward pass is correctly implemented. In Get familiar with the original repository, you have already created a script that runs a forward
pass of the model using the original repository. Now you should


 write an analogous script using the 🤗 Transformers
implementation instead of the original one. It should look as follows:
  Copied model = BrandNewBertModel.from_pretrained("/path/to/converted/checkpoint/folder")
input_ids = [0, 4, 4, 3, 2, 4, 1, 7, 19]
output = model(input_ids).last_hidden_states
It is very likely that the 🤗 Transformers implementation and the original model implementation don’t give the exact
same output the very first time or that the forward pass throws an error. Don’t be disappointed - it’s expected! First,
you should make sure that the forward pass doesn’t throw any errors. It often happens that the wrong dimensions are
used leading to a Dimensionality mismatch error or that the wrong data type object is used, e.g. torch.long
instead of torch.float32. Don’t hesitate to ask the Hugging Face team for help, if you don’t manage to solve
certain errors.
The final part to make sure the 🤗 Transformers implementation works correctly is to ensure th


at the outputs are
equivalent to a precision of 1e-3. First, you should ensure that the output shapes are identical, i.e. outputs.shape should yield the same value for the script of the 🤗 Transformers implementation and the original
implementation. Next, you should make sure that the output values are identical as well. This one of the most difficult
parts of adding a new model. Common mistakes why the outputs are not identical are:
Some layers were not added, i.e. an activation layer was not added, or the residual connection was forgotten The word embedding matrix was not tied The wrong positional embeddings are used because the original implementation uses on offset Dropout is applied during the forward pass. To fix this make sure model.training is False and that no dropout
layer is falsely activated during the forward pass, i.e. pass self.training to PyTorch’s functional dropout
The best way to fix the problem is usually to look at the forward pass of the origi


nal implementation and the 🤗
Transformers implementation side-by-side and check if there are any differences. Ideally, you should debug/print out
intermediate outputs of both implementations of the forward pass to find the exact position in the network where the 🤗
Transformers implementation shows a different output than the original implementation. First, make sure that the
hard-coded input_ids in both scripts are identical. Next, verify that the outputs of the first transformation of
the input_ids (usually the word embeddings) are identical. And then work your way up to the very last layer of the
network. At some point, you will notice a difference between the two implementations, which should point you to the bug
in the 🤗 Transformers implementation. From our experience, a simple and efficient way is to add many print statements
in both the original implementation and 🤗 Transformers implementation, at the same positions in the network
respectively, and to succe


ssively remove print statements showing the same values for intermediate presentations.
When you’re confident that both implementations yield the same output, verify the outputs with
torch.allclose(original_output, output, atol=1e-3), you’re done with the most difficult part! Congratulations - the
work left to be done should be a cakewalk 😊.
8. Adding all necessary model tests
At this point, you have successfully added a new model. However, it is very much possible that the model does not yet
fully comply with the required design. To make sure, the implementation is fully compatible with 🤗 Transformers, all
common tests should pass. The Cookiecutter should have automatically added a test file for your model, probably under
the same tests/models/brand_new_bert/test_modeling_brand_new_bert.py. Run this test file to verify that all common
tests pass:
  Copied pytest tests/models/brand_new_bert/test_modeling_brand_new_bert.py
Having fixed all common tests, it is now c


rucial to ensure that all the nice work you have done is well tested, so that
a) The community can easily understand your work by looking at specific tests of brand_new_bert b) Future changes to your model will not break any important feature of the model.
At first, integration tests should be added. Those integration tests essentially do the same as the debugging scripts
you used earlier to implement the model to 🤗 Transformers. A template of those model tests has already added by the
Cookiecutter, called BrandNewBertModelIntegrationTests and only has to be filled out by you. To ensure that those
tests are passing, run
  Copied RUN_SLOW=1 pytest -sv tests/models/brand_new_bert/test_modeling_brand_new_bert.py::BrandNewBertModelIntegrationTests
In case you are using Windows, you should replace RUN_SLOW=1 with SET RUN_SLOW=1
Second, all features that are special to brand_new_bert should be tested additionally in a separate test under
BrandNewBertModelTester/`BrandNe


wBertModelTest. This part is often forgotten but is extremely useful in two
ways:
It helps to transfer the knowledge you have acquired during the model addition to the community by showing how the
special features of brand_new_bert should work. Future contributors can quickly test changes to the model by running those special tests.
9. Implement the tokenizer
Next, we should add the tokenizer of brand_new_bert. Usually, the tokenizer is equivalent to or very similar to an
already existing tokenizer of 🤗 Transformers.
It is very important to find/extract the original tokenizer file and to manage to load this file into the 🤗
Transformers’ implementation of the tokenizer.
To ensure that the tokenizer works correctly, it is recommended to first create a script in the original repository
that inputs a string and returns the `input_ids“. It could look similar to this (in pseudo-code):
  Copied input_str = "This is a long example input string containing special character


s .$?-, numbers 2872 234 12 and words."
model = BrandNewBertModel.load_pretrained_checkpoint("/path/to/checkpoint/")
input_ids = model.tokenize(input_str)
You might have to take a deeper look again into the original repository to find the correct tokenizer function or you
might even have to do changes to your clone of the original repository to only output the input_ids. Having written
a functional tokenization script that uses the original repository, an analogous script for 🤗 Transformers should be
created. It should look similar to this:
  Copied from transformers import BrandNewBertTokenizer
input_str = "This is a long example input string containing special characters .$?-, numbers 2872 234 12 and words."
tokenizer = BrandNewBertTokenizer.from_pretrained("/path/to/tokenizer/folder/")
input_ids = tokenizer(input_str).input_ids
When both input_ids yield the same values, as a final step a tokenizer test file should also be added.
Analogous to the modeling test f


iles of brand_new_bert, the tokenization test files of brand_new_bert should
contain a couple of hard-coded integration tests.
10. Run End-to-end integration tests
Having added the tokenizer, you should also add a couple of end-to-end integration tests using both the model and the
tokenizer to tests/models/brand_new_bert/test_modeling_brand_new_bert.py in 🤗 Transformers.
Such a test should show on a meaningful
text-to-text sample that the 🤗 Transformers implementation works as expected. A meaningful text-to-text sample can
include e.g. a source-to-target-translation pair, an article-to-summary pair, a question-to-answer pair, etc… If none
of the ported checkpoints has been fine-tuned on a downstream task it is enough to simply rely on the model tests. In a
final step to ensure that the model is fully functional, it is advised that you also run all tests on GPU. It can
happen that you forgot to add some .to(self.device) statements to internal tensors of the model,


which in such a
test would show in an error. In case you have no access to a GPU, the Hugging Face team can take care of running those
tests for you.
11. Add Docstring
Now, all the necessary functionality for brand_new_bert is added - you’re almost done! The only thing left to add is
a nice docstring and a doc page. The Cookiecutter should have added a template file called
docs/source/model_doc/brand_new_bert.md that you should fill out. Users of your model will usually first look at
this page before using your model. Hence, the documentation must be understandable and concise. It is very useful for
the community to add some Tips to show how the model should be used. Don’t hesitate to ping the Hugging Face team
regarding the docstrings.
Next, make sure that the docstring added to src/transformers/models/brand_new_bert/modeling_brand_new_bert.py is
correct and included all necessary inputs and outputs. We have a detailed guide about writing documentation and our do


cstring format here. It is always to good to remind oneself that documentation should
be treated at least as carefully as the code in 🤗 Transformers since the documentation is usually the first contact
point of the community with the model.
Code refactor
Great, now you have added all the necessary code for brand_new_bert. At this point, you should correct some potential
incorrect code style by running:
  Copied make style
and verify that your coding style passes the quality check:
  Copied make quality
There are a couple of other very strict design tests in 🤗 Transformers that might still be failing, which shows up in
the tests of your pull request. This is often because of some missing information in the docstring or some incorrect
naming. The Hugging Face team will surely help you if you’re stuck here.
Lastly, it is always a good idea to refactor one’s code after having ensured that the code works correctly. With all
tests passing, now it’s a good time to go ove


r the added code again and do some refactoring.
You have now finished the coding part, congratulation! 🎉 You are Awesome! 😎
12. Upload the models to the model hub
In this final part, you should convert and upload all checkpoints to the model hub and add a model card for each
uploaded model checkpoint. You can get familiar with the hub functionalities by reading our Model sharing and uploading Page. You should work alongside the Hugging Face team here to decide on a fitting name for each
checkpoint and to get the required access rights to be able to upload the model under the author’s organization of
brand_new_bert. The push_to_hub method, present in all models in transformers, is a quick and efficient way to push your checkpoint to the hub. A little snippet is pasted below:
  Copied brand_new_bert.push_to_hub("brand_new_bert")
# Uncomment the following line to push to an organization.
# brand_new_bert.push_to_hub("<organization>/brand_new_bert")
It is worth spendi


ng some time to create fitting model cards for each checkpoint. The model cards should highlight the
specific characteristics of this particular checkpoint, e.g. On which dataset was the checkpoint
pretrained/fine-tuned on? On what down-stream task should the model be used? And also include some code on how to
correctly use the model.
13. (Optional) Add notebook
It is very helpful to add a notebook that showcases in-detail how brand_new_bert can be used for inference and/or
fine-tuned on a downstream task. This is not mandatory to merge your PR, but very useful for the community.
14. Submit your finished PR
You’re done programming now and can move to the last step, which is getting your PR merged into main. Usually, the
Hugging Face team should have helped you already at this point, but it is worth taking some time to give your finished
PR a nice description and eventually add comments to your code, if you want to point out certain design choices to your
reviewer.


Share your work!!

Now, it’s time to get some credit from the community for your work! Having completed a model addition is a major
contribution to Transformers and the whole NLP community. Your code and the ported pre-trained models will certainly be
used by hundreds and possibly even thousands of developers and researchers. You should be proud of your work and share
your achievements with the community.
You have made another model that is super easy to access for everyone in the community! 🤯


Training on TPU with TensorFlow

If you don’t need long explanations and just want TPU code samples to get started with, check out our TPU example notebook!


What is a TPU?

A TPU is a Tensor Processing Unit. They are hardware designed by Google, which are used to greatly speed up the tensor computations within neural networks, much like GPUs. They can be used for both network training and inference. They are generally accessed through Google’s cloud services, but small TPUs can also be accessed directly for free through Google Colab and Kaggle Kernels.
Because all TensorFlow models in 🤗 Transformers are Keras models, most of the methods in this document are generally applicable to TPU training for any Keras model! However, there are a few points that are specific to the HuggingFace ecosystem (hug-o-system?) of Transformers and Datasets, and we’ll make sure to flag them up when we get to them.


What kinds of TPU are available?

New users are often very confused by the range of TPUs, and the different ways to access them. The first key distinction to understand is the difference between TPU Nodes and TPU VMs.
When you use a TPU Node, you are effectively indirectly accessing a remote TPU. You will need a separate VM, which will initialize your network and data pipeline and then forward them to the remote node. When you use a TPU on Google Colab, you are accessing it in the TPU Node style.
Using TPU Nodes can have some quite unexpected behaviour for people who aren’t used to them! In particular, because the TPU is located on a physically different system to the machine you’re running your Python code on, your data cannot be local to your machine - any data pipeline that loads from your machine’s internal storage will totally fail! Instead, data must be stored in Google Cloud Storage wh


What kinds of TPU are available?

ere your data pipeline can still access it, even when the pipeline is running on the remote TPU node.
If you can fit all your data in memory as np.ndarray or tf.Tensor, then you can fit() on that data even when using Colab or a TPU Node, without needing to upload it to Google Cloud Storage.
🤗Specific Hugging Face Tip🤗: The methods Dataset.to_tf_dataset() and its higher-level wrapper model.prepare_tf_dataset() , which you will see throughout our TF code examples, will both fail on a TPU Node. The reason for this is that even though they create a tf.data.Dataset it is not a “pure” tf.data pipeline and uses tf.numpy_function or Dataset.from_generator() to stream data from the underlying HuggingFace Dataset. This HuggingFace Dataset is backed by data that is on a local disc and which the remote TPU Node will not be able to read.
The second way to access a TPU is


What kinds of TPU are available?

via a TPU VM. When using a TPU VM, you connect directly to the machine that the TPU is attached to, much like training on a GPU VM. TPU VMs are generally easier to work with, particularly when it comes to your data pipeline. All of the above warnings do not apply to TPU VMs!
This is an opinionated document, so here’s our opinion: Avoid using TPU Node if possible. It is more confusing and more difficult to debug than TPU VMs. It is also likely to be unsupported in future - Google’s latest TPU, TPUv4, can only be accessed as a TPU VM, which suggests that TPU Nodes are increasingly going to become a “legacy” access method. However, we understand that the only free TPU access is on Colab and Kaggle Kernels, which uses TPU Node - so we’ll try to explain how to handle it if you have to! Check the TPU example notebook for code samples that explain this in more detai


What sizes of TPU are available?

A single TPU (a v2-8/v3-8/v4-8) runs 8 replicas. TPUs exist in pods that can run hundreds or thousands of replicas simultaneously. When you use more than a single TPU but less than a whole pod (for example, a v3-32), your TPU fleet is referred to as a pod slice.
When you access a free TPU via Colab, you generally get a single v2-8 TPU.


I keep hearing about this XLA thing. What’s XLA, and how does it relate to TPUs?

XLA is an optimizing compiler, used by both TensorFlow and JAX. In JAX it is the only compiler, whereas in TensorFlow it is optional (but mandatory on TPU!). The easiest way to enable it when training a Keras model is to pass the argument jit_compile=True to model.compile(). If you don’t get any errors and performance is good, that’s a great sign that you’re ready to move to TPU!
Debugging on TPU is generally a bit harder than on CPU/GPU, so we recommend getting your code running on CPU/GPU with XLA first before trying it on TPU. You don’t have to train for long, of course - just for a few steps to make sure that your


model and data pipeline are working like you expect them to.
XLA compiled code is usually faster - so even if you’re not planning to run on TPU, adding jit_compile=True can improve your performance. Be sure to note the caveats below about XLA compatibility, though!
Tip born of painful experience: Although using jit_compile=True is a good way to get a speed boost and test if your CPU/GPU code is XLA-compatible, it can actually cause a lot of problems if you leave it in when actually training on TPU. XLA compilation will happen implicitly on TPU, so remember to remove that line before actually running your code on a TPU!


How do I make my model XLA compatible?

In many cases, your code is probably XLA-compatible already! However, there are a few things that work in normal TensorFlow that don’t work in XLA. We’ve distilled them into three core rules below:
🤗Specific HuggingFace Tip🤗: We’ve put a lot of effort into rewriting our TensorFlow models and loss functions to be XLA-compatible. Our models and loss functions generally obey rule #1 and #2 by default, so you can skip over them if you’re using transformers models. Don’t forget about these rules when writing your own models and loss functions, though!
 XLA Rule #1: Your code cannot have “data-dependent conditionals”
What that means is that any if statement cannot depend on values inside a tf.Tensor. For example, this code block cannot be compiled with XLA!
  Copied if tf.reduce_sum(tensor) > 10:
    tensor = tensor / 2.0
This might seem very restrictive at f


How do I make my model XLA compatible?

irst, but most neural net code doesn’t need to do this. You can often get around this restriction by using tf.cond (see the documentation here) or by removing the conditional and finding a clever math trick with indicator variables instead, like so:
  Copied sum_over_10 = tf.cast(tf.reduce_sum(tensor) > 10, tf.float32)
tensor = tensor / (1.0 + sum_over_10)
This code has exactly the same effect as the code above, but by avoiding a conditional, we ensure it will compile with XLA without problems!
 XLA Rule #2: Your code cannot have “data-dependent shapes”
What this means is that the shape of all of the tf.Tensor objects in your code cannot depend on their values. For example, the function tf.unique cannot be compiled with XLA, because it returns a tensor containing one instance of each unique value in the input. The shape of this output will obviously be


How do I make my model XLA compatible?

different depending on how repetitive the input Tensor was, and so XLA refuses to handle it!
In general, most neural network code obeys rule #2 by default. However, there are a few common cases where it becomes a problem. One very common one is when you use label masking, setting your labels to a negative value to indicate that those positions should be ignored when computing the loss. If you look at NumPy or PyTorch loss functions that support label masking, you will often see code like this that uses boolean indexing:
  Copied label_mask = labels >= 0
masked_outputs = outputs[label_mask]
masked_labels = labels[label_mask]
loss = compute_loss(masked_outputs, masked_labels)
mean_loss = torch.mean(loss)
This code is totally fine in NumPy or PyTorch, but it breaks in XLA! Why? Because the shape of masked_outputs and masked_labels depends on how many posit


How do I make my model XLA compatible?

ions are masked - that makes it a data-dependent shape. However, just like for rule #1, we can often rewrite this code to yield exactly the same output without any data-dependent shapes.
  Copied label_mask = tf.cast(labels >= 0, tf.float32)
loss = compute_loss(outputs, labels)
loss = loss * label_mask  # Set negative label positions to 0
mean_loss = tf.reduce_sum(loss) / tf.reduce_sum(label_mask)
Here, we avoid data-dependent shapes by computing the loss for every position, but zeroing out the masked positions in both the numerator and denominator when we calculate the mean, which yields exactly the same result as the first block while maintaining XLA compatibility. Note that we use the same trick as in rule #1 - converting a tf.bool to tf.float32 and using it as an indicator variable. This is a really useful trick, so remember it if you need to conver


How do I make my model XLA compatible?

t your own code to XLA!
 XLA Rule #3: XLA will need to recompile your model for every different input shape it sees
This is the big one. What this means is that if your input shapes are very variable, XLA will have to recompile your model over and over, which will create huge performance problems. This commonly arises in NLP models, where input texts have variable lengths after tokenization. In other modalities, static shapes are more common and this rule is much less of a problem.
How can you get around rule #3? The key is padding - if you pad all your inputs to the same length, and then use an attention_mask, you can get the same results as you’d get from variable shapes, but without any XLA issues. However, excessive padding can cause severe slowdown too - if you pad all your samples to the maximum length in the whole dataset, you might end up with b


How do I make my model XLA compatible?

atches consisting endless padding tokens, which will waste a lot of compute and memory!
There isn’t a perfect solution to this problem. However, you can try some tricks. One very useful trick is to pad batches of samples up to a multiple of a number like 32 or 64 tokens. This often only increases the number of tokens by a small amount, but it hugely reduces the number of unique input shapes, because every input shape now has to be a multiple of 32 or 64. Fewer unique input shapes means fewer XLA compilations!
🤗Specific HuggingFace Tip🤗: Our tokenizers and data collators have methods that can help you here. You can use padding="max_length" or padding="longest" when calling tokenizers to get them to output padded data. Our tokenizers and data collators also have a pad_to_multiple_of argument that you can use to reduce the number of unique input shapes you


How do I actually train my model on TPU?

Once your training is XLA-compatible and (if you’re using TPU Node / Colab) your dataset has been prepared appropriately, running on TPU is surprisingly easy! All you really need to change in your code is to add a few lines to initialize your TPU, and to ensure that your model and dataset are created inside a TPUStrategy scope. Take a look at our TPU example notebook to see this in action!


Summary

There was a lot in here, so let’s summarize with a quick checklist you can follow when you want to get your model ready for TPU training:
Make sure your code follows the three rules of XLA Compile your model with jit_compile=True on CPU/GPU and confirm that you can train it with XLA Either load your dataset into memory or use a TPU-compatible dataset loading approach (see notebook) Migrate your code either to Colab (with accelerator set to “TPU”) or a TPU VM on Google Cloud Add TPU initializer code (see notebook) Create your TPUStrategy and make sure dataset loading and model creation are inside the strategy.scope() (see notebook) Don’t forget to take jit_compile=True out again when you move to TPU! 🙏🙏🙏🥺🥺🥺 Call model.fit() You did it!


Custom Tools and Prompts

If you are not aware of what tools and agents are in the context of transformers, we recommend you read the
Transformers Agents page first.
Transformers Agents is an experimental API that is subject to change at any time. Results returned by the agents
can vary as the APIs or underlying models are prone to change.
Creating and using custom tools and prompts is paramount to empowering the agent and having it perform new tasks.
In this guide we’ll take a look at:
How to customize the prompt How to use custom tools How to create custom tools


Customizing the prompt

As explained in Transformers Agents agents can run in run() and chat() mode.
Both the run and chat modes underlie the same logic. The language model powering the agent is conditioned on a long
prompt and completes the prompt by generating the next tokens until the stop token is reached.
The only difference between the two modes is that during the chat mode the prompt is extended with
previous user inputs and model generations. This allows the agent to have access to past interactions,
seemingly giving the agent some kind of memory.


Structure of the prompt

Let’s take a closer look at how the prompt is structured to understand how it can be best customized.
The prompt is structured broadly into four parts.
Introduction: how the agent should behave, explanation of the concept of tools. Description of all the tools. This is defined by a <<all_tools>> token that is dynamically replaced at runtime with the tools defined/chosen by the user. A set of examples of tasks and their solution Current example, and request for solution.
To better understand each part, let’s look at a shortened version of how the run prompt can look like:
  Copied I will ask you to perform a task, your job is to come up with a series of simple commands in Python that will perform the task.
[...]
You can print intermediate results if it makes sense to do so.
Tools:
- document_qa: This is a tool that answers a question about a document (pdf). It takes an input named `document` which should be the docume


Structure of the prompt

nt containing the information, as well as a `question` that is the question about the document. It returns a text that contains the answer to the question.
- image_captioner: This is a tool that generates a description of an image. It takes an input named `image` which should be the image to the caption and returns a text that contains the description in English.
[...]
Task: "Answer the question in the variable `question` about the image stored in the variable `image`. The question is in French."
I will use the following tools: `translator` to translate the question into English and then `image_qa` to answer the question on the input image.
Answer:
```py
translated_question = translator(question=question, src_lang="French", tgt_lang="English")
print(f"The translated question is {translated_question}.")
answer = image_qa(image=image, question=translated_question)
print(f"The answer is {answer}")
```
Task: "Identify th


Structure of the prompt

e oldest person in the `document` and create an image showcasing the result as a banner."
I will use the following tools: `document_qa` to find the oldest person in the document, then `image_generator` to generate an image according to the answer.
Answer:
```py
answer = document_qa(document, question="What is the oldest person?")
print(f"The answer is {answer}.")
image = image_generator("A banner showing " + answer)
```
[...]
Task: "Draw me a picture of rivers and lakes"
I will use the following
The introduction (the text before “Tools:”) explains precisely how the model shall behave and what it should do.
This part most likely does not need to be customized as the agent shall always behave the same way.
The second part (the bullet points below “Tools”) is dynamically added upon calling run or chat. There are
exactly as many bullet points as there are tools in agent.toolbox and each bullet point consists of the name


Structure of the prompt

and description of the tool:
  Copied - <tool.name>: <tool.description>
Let’s verify this quickly by loading the document_qa tool and printing out the name and description.
  Copied from transformers import load_tool
document_qa = load_tool("document-question-answering")
print(f"- {document_qa.name}: {document_qa.description}")
which gives:
  Copied - document_qa: This is a tool that answers a question about a document (pdf). It takes an input named `document` which should be the document containing the information, as well as a `question` that is the question about the document. It returns a text that contains the answer to the question.
We can see that the tool name is short and precise. The description includes two parts, the first explaining
what the tool does and the second states what input arguments and return values are expected.
A good tool name and tool description are very important for the agent to correc


Structure of the prompt

tly use it. Note that the only
information the agent has about the tool is its name and description, so one should make sure that both
are precisely written and match the style of the existing tools in the toolbox. In particular make sure the description
mentions all the arguments expected by name in code-style, along with the expected type and a description of what they
are.
Check the naming and description of the curated Transformers tools to better understand what name and
description a tool is expected to have. You can see all tools with the Agent.toolbox property.
The third part includes a set of curated examples that show the agent exactly what code it should produce
for what kind of user request. The large language models empowering the agent are extremely good at
recognizing patterns in a prompt and repeating the pattern with new data. Therefore, it is very important
that the examples are written in a way tha


Structure of the prompt

t maximizes the likelihood of the agent to generating correct,
executable code in practice.
Let’s have a look at one example:
  Copied Task: "Identify the oldest person in the `document` and create an image showcasing the result as a banner."
I will use the following tools: `document_qa` to find the oldest person in the document, then `image_generator` to generate an image according to the answer.
Answer:
```py
answer = document_qa(document, question="What is the oldest person?")
print(f"The answer is {answer}.")
image = image_generator("A banner showing " + answer)
```
The pattern the model is prompted to repeat has three parts: The task statement, the agent’s explanation of
what it intends to do, and finally the generated code. Every example that is part of the prompt has this exact
pattern, thus making sure that the agent will reproduce exactly the same pattern when generating new tokens.
The prompt examples are c


Structure of the prompt

urated by the Transformers team and rigorously evaluated on a set of
problem statements
to ensure that the agent’s prompt is as good as possible to solve real use cases of the agent.
The final part of the prompt corresponds to:
  Copied Task: "Draw me a picture of rivers and lakes"
I will use the following
is a final and unfinished example that the agent is tasked to complete. The unfinished example
is dynamically created based on the actual user input. For the above example, the user ran:
  Copied agent.run("Draw me a picture of rivers and lakes")
The user input - a.k.a the task: “Draw me a picture of rivers and lakes” is cast into the
prompt template: “Task: <task> \n\n I will use the following”. This sentence makes up the final lines of the
prompt the agent is conditioned on, therefore strongly influencing the agent to finish the example
exactly in the same way it was previously done in the examples.
Without going


Structure of the prompt

 into too much detail, the chat template has the same prompt structure with the
examples having a slightly different style, e.g.:
  Copied [...]
=====
Human: Answer the question in the variable `question` about the image stored in the variable `image`.
Assistant: I will use the tool `image_qa` to answer the question on the input image.
```py
answer = image_qa(text=question, image=image)
print(f"The answer is {answer}")
```
Human: I tried this code, it worked but didn't give me a good result. The question is in French
Assistant: In this case, the question needs to be translated first. I will use the tool `translator` to do this.
```py
translated_question = translator(question=question, src_lang="French", tgt_lang="English")
print(f"The translated question is {translated_question}.")
answer = image_qa(text=translated_question, image=image)
print(f"The answer is {answer}")
```
=====
[...]
Contrary, to the examples of th


Structure of the prompt

e run prompt, each chat prompt example has one or more exchanges between the
Human and the Assistant. Every exchange is structured similarly to the example of the run prompt.
The user’s input is appended to behind Human: and the agent is prompted to first generate what needs to be done
before generating code. An exchange can be based on previous exchanges, therefore allowing the user to refer
to past exchanges as is done e.g. above by the user’s input of “I tried this code” refers to the
previously generated code of the agent.
Upon running .chat, the user’s input or task is cast into an unfinished example of the form:
  Copied Human: <user-input>\n\nAssistant:
which the agent completes. Contrary to the run command, the chat command then appends the completed example
to the prompt, thus giving the agent more context for the next chat turn.
Great now that we know how the prompt is structured, let’s see how we can custo


Writing good user inputs

While large language models are getting better and better at understanding users’ intentions, it helps
enormously to be as precise as possible to help the agent pick the correct task. What does it mean to be
as precise as possible?
The agent sees a list of tool names and their description in its prompt. The more tools are added the
more difficult it becomes for the agent to choose the correct tool and it’s even more difficult to choose
the correct sequences of tools to run. Let’s look at a common failure case, here we will only return
the code to analyze it.
  Copied from transformers import HfAgent
agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder")
agent.run("Show me a tree", return_code=True)
g


Writing good user inputs

ives:
  Copied ==Explanation from the agent==
I will use the following tool: `image_segmenter` to create a segmentation mask for the image.
==Code generated by the agent==
mask = image_segmenter(image, prompt="tree")
which is probably not what we wanted. Instead, it is more likely that we want an image of a tree to be generated.
To steer the agent more towards using a specific tool it can therefore be very helpful to use important keywords that
are present in the tool’s name and description. Let’s have a look.
  Copied agent.toolbox["image_generator"].description
  Copied 'This is a tool that creates an image according to a prompt, which is a text description. It takes an input named `prompt` which contains the image descript


Writing good user inputs

ion and outputs an image.
The name and description make use of the keywords “image”, “prompt”, “create” and “generate”. Using these words will most likely work better here. Let’s refine our prompt a bit.
  Copied agent.run("Create an image of a tree", return_code=True)
gives:
  Copied ==Explanation from the agent==
I will use the following tool `image_generator` to generate an image of a tree.
==Code generated by the agent==
image = image_generator(prompt="tree")
Much better! That looks more like what we want. In short, when you notice that the agent struggles to
correctly map your task to the correct tools, try looking up the most pertinent keywords of the tool’s name
and description and try refining your task request with i


Customizing the tool descriptions

As we’ve seen before the agent has access to each of the tools’ names and descriptions. The base tools
should have very precise names and descriptions, however, you might find that it could help to change the
the description or name of a tool for your specific use case. This might become especially important
when you’ve added multiple tools that are very similar or if you want to use your agent only for a certain
domain, e.g. image generation and transformations.
A common problem is that the agent confuses image generation with image transformation/modification when
used a lot for image generation tasks, e.g.
  Copied agent.run("Make an image of a house and a car", return_code=True)
returns
  Copied ==Explanation from the agent==
I will use the following tools `image_generator` to generate an image of a house and `image_transformer` to transform the image of a car into the image of a house.
==Code generated by the agent==
house_image = image_generator(prompt="A house")
car_image =


Customizing the tool descriptions

image_generator(prompt="A car")
house_car_image = image_transformer(image=car_image, prompt="A house")
which is probably not exactly what we want here. It seems like the agent has a difficult time
to understand the difference between image_generator and image_transformer and often uses the two together.
We can help the agent here by changing the tool name and description of image_transformer. Let’s instead call it modifier
to disassociate it a bit from “image” and “prompt”:
  Copied agent.toolbox["modifier"] = agent.toolbox.pop("image_transformer")
agent.toolbox["modifier"].description = agent.toolbox["modifier"].description.replace(
    "transforms an image according to a prompt", "modifies an image"
)
Now “modify” is a strong cue to use the new image processor which should help with the above prompt. Let’s run it again.
  Copied agent.run("Make an image of a house and a car", return_code=True)
Now we’re getting:
  Copied ==Explanation from the agent==
I will use the following tool


Customizing the tool descriptions

s: `image_generator` to generate an image of a house, then `image_generator` to generate an image of a car.
==Code generated by the agent==
house_image = image_generator(prompt="A house")
car_image = image_generator(prompt="A car")
which is definitely closer to what we had in mind! However, we want to have both the house and car in the same image. Steering the task more toward single image generation should help:
  Copied agent.run("Create image: 'A house and car'", return_code=True)
  Copied ==Explanation from the agent==
I will use the following tool: `image_generator` to generate an image.
==Code generated by the agent==
image = image_generator(prompt="A house and car")
Agents are still brittle for many use cases, especially when it comes to
slightly more complex use cases like generating an image of multiple objects.
Both the agent itself and the underlying prompt will be further improved in the coming
months making sure that agents become more robust to a variety of user inputs


Customizing the whole prompt

To give the user maximum flexibility, the whole prompt template as explained in above
can be overwritten by the user. In this case make sure that your custom prompt includes an introduction section,
a tool section, an example section, and an unfinished example section. If you want to overwrite the run prompt template,
you can do as follows:
  Copied template = """ [...] """
agent = HfAgent(your_endpoint, run_prompt_template=template)
Please make sure to have the <<all_tools>> string and the <<prompt>> defined somewhere in the template so that the agent can be aware
of the tools, it has available to it as well as correctly insert the user’s prompt.
Similarly, one can overwrite the chat prompt template. Note that the chat mode always uses the following format for the exchanges:
  Copied Human: <<task>>
Assistant:
Therefore it is important that the examples of the cus


Customizing the whole prompt

tom chat prompt template also make use of this format.
You can overwrite the chat template at instantiation as follows.
  Copied template = """ [...] """
agent = HfAgent(url_endpoint=your_endpoint, chat_prompt_template=template)
Please make sure to have the <<all_tools>> string defined somewhere in the template so that the agent can be aware
of the tools, it has available to it.
In both cases, you can pass a repo ID instead of the prompt template if you would like to use a template hosted by someone in the community. The default prompts live in this repo as an example.
To upload your custom prompt on a repo on the Hub and share it with the community just make sure:
to use a dataset repository to put the prompt template for the run command in a file named run_prompt_template.txt to put the prompt template for the chat command in a file named chat_prompt_template.txt


Using custom tools

In this section, we’ll be leveraging two existing custom tools that are specific to image generation:
We replace huggingface-tools/image-transformation,
with diffusers/controlnet-canny-tool
to allow for more image modifications. We add a new tool for image upscaling to the default toolbox:
diffusers/latent-upscaler-tool replace the existing image-transformation tool.
We’ll start by loading the custom tools with the convenient load_tool() function:
  Copied from transformers import load_tool
controlnet_transformer = load_tool("diffusers/controlnet-canny-tool")
upscaler = load_tool("diffusers/latent-upscaler-tool")
Upon adding custom tools to an agent, the tools’ descriptions and names are automatically
included in the agents’ prompts. Thus, it is imperative that custom tools have
a well-written description and name in order for the agent to understand how to use them.
Let’s take a look a


Using custom tools

t the description and name of controlnet_transformer:
  Copied print(f"Description: '{controlnet_transformer.description}'")
print(f"Name: '{controlnet_transformer.name}'")
gives
  Copied Description: 'This is a tool that transforms an image with ControlNet according to a prompt.
It takes two inputs: `image`, which should be the image to transform, and `prompt`, which should be the prompt to use to change it. It returns the modified image.'
Name: 'image_transformer'
The name and description are accurate and fit the style of the curated set of tools.
Next, let’s instantiate an agent with controlnet_transformer and upscaler:
  Copied tools = [controlnet_transformer, upscaler]
agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder", additional_tools=tools)
This command should give you the following info:
  Copied image_transformer has been replaced by <transformers


Using custom tools

_modules.diffusers.controlnet-canny-tool.bd76182c7777eba9612fc03c0
8718a60c0aa6312.image_transformation.ControlNetTransformationTool object at 0x7f1d3bfa3a00> as provided in `additional_tools`
The set of curated tools already has an image_transformer tool which is hereby replaced with our custom tool.
Overwriting existing tools can be beneficial if we want to use a custom tool exactly for the same task as an existing tool
because the agent is well-versed in using the specific task. Beware that the custom tool should follow the exact same API
as the overwritten tool in this case, or you should adapt the prompt template to make sure all examples using that
tool are updated.
The upscaler tool was given the name image_upscaler which is not yet present in the default toolbox and is therefore simply added to the list of tools.
You can always have a look at the toolbox that is currently availa


Using custom tools

ble to the agent via the agent.toolbox attribute:
  Copied print("\n".join([f"- {a}" for a in agent.toolbox.keys()]))
  Copied - document_qa
- image_captioner
- image_qa
- image_segmenter
- transcriber
- summarizer
- text_classifier
- text_qa
- text_reader
- translator
- image_transformer
- text_downloader
- image_generator
- video_generator
- image_upscaler
Note how image_upscaler is now part of the agents’ toolbox.
Let’s now try out the new tools! We will re-use the image we generated in Transformers Agents Quickstart.
  Copied from diffusers.utils import load_image
image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rivers_and_lakes.png"
)
Let’s transform the image into a beautiful winter landscape:
  Copied image = agent.run("Transform the image: 'A frozen lake and snowy forest'", image=image)
  Copied ==Explanation fro


Using custom tools

m the agent==
I will use the following tool: `image_transformer` to transform the image.
==Code generated by the agent==
image = image_transformer(image, prompt="A frozen lake and snowy forest")
The new image processing tool is based on ControlNet which can make very strong modifications to the image.
By default the image processing tool returns an image of size 512x512 pixels. Let’s see if we can upscale it.
  Copied image = agent.run("Upscale the image", image)
  Copied ==Explanation from the agent==
I will use the following tool: `image_upscaler` to upscale the image.
==Code generated by the agent==
upscaled_image = image_upscaler(image)
The agent automatically mapped our prompt “Upscale the image” to the just added upscaler tool purely based on the description and name of the upscaler tool
and was able to correctly run it.
Next, let’s have a look at how you can create a new custom t


Adding new tools

In this section, we show how to create a new tool that can be added to the agent.
 Creating a new tool
We’ll first start by creating a tool. We’ll add the not-so-useful yet fun task of fetching the model on the Hugging Face
Hub with the most downloads for a given task.
We can do that with the following code:
  Copied from huggingface_hub import list_models
task = "text-classification"
model = next(iter(list_models(filter=task, sort="downloads", direction=-1)))
print(model.id)
For the task text-classification, this returns 'facebook/bart-large-mnli', for translation it returns 't5-base.
How do we convert this to a tool that the agent can leverage? All tools depend on the superclass Tool that holds the
main attributes necessary. We’ll create a class that inherits from it:
  Copied from transformers import Tool
class HFModelDownloadsTool(Tool):
    pass
This class has a few needs:
An attribute name, which corresponds to the name of the tool itself. To be in tune with other tools


Adding new tools

which have a
performative name, we’ll name it model_download_counter. An attribute description, which will be used to populate the prompt of the agent. inputs and outputs attributes. Defining this will help the python interpreter make educated choices about types,
and will allow for a gradio-demo to be spawned when we push our tool to the Hub. They’re both a list of expected
values, which can be text, image, or audio. A __call__ method which contains the inference code. This is the code we’ve played with above!
Here’s what our class looks like now:
  Copied from transformers import Tool
from huggingface_hub import list_models
class HFModelDownloadsTool(Tool):
    name = "model_download_counter"
    description = (
        "This is a tool that returns the most downloaded model of a given task on the Hugging Face Hub. "
        "It takes the name of the category (such as text-classification, depth-estimation, etc), and "
        "returns the name of the checkpoint."
    )
    in


Adding new tools

puts = ["text"]
    outputs = ["text"]
    def __call__(self, task: str):
        model = next(iter(list_models(filter=task, sort="downloads", direction=-1)))
        return model.id
We now have our tool handy. Save it in a file and import it from your main script. Let’s name this file
model_downloads.py, so the resulting import code looks like this:
  Copied from model_downloads import HFModelDownloadsTool
tool = HFModelDownloadsTool()
In order to let others benefit from it and for simpler initialization, we recommend pushing it to the Hub under your
namespace. To do so, just call push_to_hub on the tool variable:
  Copied tool.push_to_hub("hf-model-downloads")
You now have your code on the Hub! Let’s take a look at the final step, which is to have the agent use it.
 Having the agent use the tool
We now have our tool that lives on the Hub which can be instantiated as such (change the user name for your tool):
  Copied from transformers import load_tool
tool = load_tool("lysan


Adding new tools

dre/hf-model-downloads")
In order to use it in the agent, simply pass it in the additional_tools parameter of the agent initialization method:
  Copied from transformers import HfAgent
agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder", additional_tools=[tool])
agent.run(
    "Can you read out loud the name of the model that has the most downloads in the 'text-to-video' task on the Hugging Face Hub?"
)
which outputs the following:
  Copied ==Code generated by the agent==
model = model_download_counter(task="text-to-video")
print(f"The model with the most downloads is {model}.")
audio_model = text_reader(model)
==Result==
The model with the most downloads is damo-vilab/text-to-video-ms-1.7b.
and generates the following audio.
| Audio   |
|---------|
Depending on the LLM, some are quite brittle and require very exact prompts in order to work well. Having a well-defined
name and description of the tool is paramount to having it be leveraged by the age


Replacing existing tools

Replacing existing tools can be done simply by assigning a new item to the agent’s toolbox. Here’s how one would do so:
  Copied from transformers import HfAgent, load_tool
agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder")
agent.toolbox["image-transformation"] = load_tool("diffusers/controlnet-canny-tool")
Beware when replacing tools with others! This will also adjust the agent’s prompt. This can be good if you have a better
prompt suited for the task, but it can also result in your tool being selected way more than others or for other
tools to be selected instead of the one you have defined.


Leveraging gradio-tools

gradio-tools is a powerful library that allows using Hugging
Face Spaces as tools. It supports many existing Spaces as well as custom Spaces to be designed with it.
We offer support for gradio_tools by using the Tool.from_gradio method. For example, we want to take
advantage of the StableDiffusionPromptGeneratorTool tool offered in the gradio-tools toolkit so as to
improve our prompts and generate better images.
We first import the tool from gradio_tools and instantiate it:
  Copied from gradio_tools import StableDiffusionPromptGeneratorTool
gradio_tool = StableDiffusionPromptGeneratorTool()
We pass that instance to the Tool.from_gradio method:
  Copied from transformers import Tool
tool = Tool.from_gradio(gradio_tool)
Now we can manage it exactly as we would a usual custom tool. We leverage it to improve our prompt
a rabbit wearing a space suit:
  Copied from transformers import HfAgent
agent = HfAgent("


Leveraging gradio-tools

https://api-inference.huggingface.co/models/bigcode/starcoder", additional_tools=[tool])
agent.run("Generate an image of the `prompt` after improving it.", prompt="A rabbit wearing a space suit")
The model adequately leverages the tool:
  Copied ==Explanation from the agent==
I will use the following  tools: `StableDiffusionPromptGenerator` to improve the prompt, then `image_generator` to generate an image according to the improved prompt.
==Code generated by the agent==
improved_prompt = StableDiffusionPromptGenerator(prompt)
print(f"The improved prompt is {improved_prompt}.")
image = image_generator(improved_prompt)
Before finally generating the image:
gradio-tools requires textual inputs and outputs, even when working with different modalities. This implementation
works with image and audio objects. The two are currently incompatible, but will rapidly become compatible as we
work to improve the support


Future compatibility with Langchain

We love Langchain and think it has a very compelling suite of tools. In order to handle these tools,
Langchain requires textual inputs and outputs, even when working with different modalities.
This is often the serialized version (i.e., saved to disk) of the objects.
This difference means that multi-modality isn’t handled between transformers-agents and langchain.
We aim for this limitation to be resolved in future versions, and welcome any help from avid langchain
users to help us achieve this compatibility.
We would love to have better support. If you would like to help, please
open an issue and share what you have in mind.


Sharing custom models

The 🤗 Transformers library is designed to be easily extensible. Every model is fully coded in a given subfolder
of the repository with no abstraction, so you can easily copy a modeling file and tweak it to your needs.
If you are writing a brand new model, it might be easier to start from scratch. In this tutorial, we will show you
how to write a custom model and its configuration so it can be used inside Transformers, and how you can share it
with the community (with the code it relies on) so that anyone can use it, even if it’s not present in the 🤗
Transformers library.
We will illustrate all of this on a ResNet model, by wrapping the ResNet class of the
timm library into a PreTrainedModel.


Writing a custom configuration

Before we dive into the model, let’s first write its configuration. The configuration of a model is an object that
will contain all the necessary information to build the model. As we will see in the next section, the model can only
take a config to be initialized, so we really need that object to be as complete as possible.
In our example, we will take a couple of arguments of the ResNet class that we might want to tweak. Different
configurations will then give us the different types of ResNets that are possible. We then just store those arguments,
after checking the validity of a few of them.
  Copied from transformers import PretrainedConfig
from typing import List
class ResnetConfig(PretrainedConfig):
    model_type = "resnet"
    def __init__(
        self,
        block_type="bottleneck",


Writing a custom configuration

       layers: List[int] = [3, 4, 6, 3],
        num_classes: int = 1000,
        input_channels: int = 3,
        cardinality: int = 1,
        base_width: int = 64,
        stem_width: int = 64,
        stem_type: str = "",
        avg_down: bool = False,
        **kwargs,
    ):
        if block_type not in ["basic", "bottleneck"]:
            raise ValueError(f"`block_type` must be 'basic' or bottleneck', got {block_type}.")
        if stem_type not in ["", "deep", "deep-tiered"]:
            raise ValueError(f"`stem_type` must be '', 'deep' or 'deep-tiered', got {stem_type}.")
        self.block_type = block_type
        self.layers = layers
        self.num_classes = num_classes
        self.input_channels = input_channels
        self.cardinality = cardinality
        self.base_width = bas


Writing a custom configuration

e_width
        self.stem_width = stem_width
        self.stem_type = stem_type
        self.avg_down = avg_down
        super().__init__(**kwargs)
The three important things to remember when writing you own configuration are the following:
you have to inherit from PretrainedConfig, the __init__ of your PretrainedConfig must accept any kwargs, those kwargs need to be passed to the superclass __init__.
The inheritance is to make sure you get all the functionality from the 🤗 Transformers library, while the two other
constraints come from the fact a PretrainedConfig has more fields than the ones you are setting. When reloading a
config with the from_pretrained method, those fields need to be accepted by your config and then sent to the
superclass.
Defining a model_type for your configuration (here m


Writing a custom configuration

odel_type="resnet") is not mandatory, unless you want to
register your model with the auto classes (see last section).
With this done, you can easily create and save your configuration like you would do with any other model config of the
library. Here is how we can create a resnet50d config and save it:
  Copied resnet50d_config = ResnetConfig(block_type="bottleneck", stem_width=32, stem_type="deep", avg_down=True)
resnet50d_config.save_pretrained("custom-resnet")
This will save a file named config.json inside the folder custom-resnet. You can then reload your config with the
from_pretrained method:
  Copied resnet50d_config = ResnetConfig.from_pretrained("custom-resnet")
You can also use any other method of the PretrainedConfig class, like push_to_hub() to
directly upload your config to the Hub.


Writing a custom model

Now that we have our ResNet configuration, we can go on writing the model. We will actually write two: one that
extracts the hidden features from a batch of images (like BertModel) and one that is suitable for image
classification (like BertForSequenceClassification).
As we mentioned before, we’ll only write a loose wrapper of the model to keep it simple for this example. The only
thing we need to do before writing this class is a map between the block types and actual block classes. Then the
model is defined from the configuration by passing everything to the ResNet class:
  Copied from transformers import PreTrainedModel
from timm.models.resnet import BasicBlock, Bottleneck, ResNet
from .configuration_resnet import ResnetConfig
BLOCK_MAPPING = {"basic": BasicBlock, "bottleneck": Bottleneck}
class ResnetModel(P


Writing a custom model

reTrainedModel):
    config_class = ResnetConfig
    def __init__(self, config):
        super().__init__(config)
        block_layer = BLOCK_MAPPING[config.block_type]
        self.model = ResNet(
            block_layer,
            config.layers,
            num_classes=config.num_classes,
            in_chans=config.input_channels,
            cardinality=config.cardinality,
            base_width=config.base_width,
            stem_width=config.stem_width,
            stem_type=config.stem_type,
            avg_down=config.avg_down,
        )
    def forward(self, tensor):
        return self.model.forward_features(tensor)
For the model that will classify images, we just change the forward method:
  Copied import torch
class ResnetModelForImageClassification(PreTrainedModel):
    config_class = ResnetConfig


Writing a custom model

    def __init__(self, config):
        super().__init__(config)
        block_layer = BLOCK_MAPPING[config.block_type]
        self.model = ResNet(
            block_layer,
            config.layers,
            num_classes=config.num_classes,
            in_chans=config.input_channels,
            cardinality=config.cardinality,
            base_width=config.base_width,
            stem_width=config.stem_width,
            stem_type=config.stem_type,
            avg_down=config.avg_down,
        )
    def forward(self, tensor, labels=None):
        logits = self.model(tensor)
        if labels is not None:
            loss = torch.nn.cross_entropy(logits, labels)
            return {"loss": loss, "logits": logits}
        return {"logits": logits}
In both cases, notice how we inherit from PreTrainedModel and


Writing a custom model

call the superclass initialization with the config
(a bit like when you write a regular torch.nn.Module). The line that sets the config_class is not mandatory, unless
you want to register your model with the auto classes (see last section).
If your model is very similar to a model inside the library, you can re-use the same configuration as this model.
You can have your model return anything you want, but returning a dictionary like we did for
ResnetModelForImageClassification, with the loss included when labels are passed, will make your model directly
usable inside the Trainer class. Using another output format is fine as long as you are planning on using your own
training loop or another library for training.
Now that we have our model class, let’s create one:
  Copied resnet50d = ResnetModelForImageClassific


Writing a custom model

ation(resnet50d_config)
Again, you can use any of the methods of PreTrainedModel, like save_pretrained() or
push_to_hub(). We will use the second in the next section, and see how to push the model weights
with the code of our model. But first, let’s load some pretrained weights inside our model.
In your own use case, you will probably be training your custom model on your own data. To go fast for this tutorial,
we will use the pretrained version of the resnet50d. Since our model is just a wrapper around it, it’s going to be
easy to transfer those weights:
  Copied import timm
pretrained_model = timm.create_model("resnet50d", pretrained=True)
resnet50d.model.load_state_dict(pretrained_model.state_dict())
Now let’s see how to make sure that when we do save_pretrained() or push_to_hub(), the
code of the model is sa


Sending the code to the Hub

This API is experimental and may have some slight breaking changes in the next releases.
First, make sure your model is fully defined in a .py file. It can rely on relative imports to some other files as
long as all the files are in the same directory (we don’t support submodules for this feature yet). For our example,
we’ll define a modeling_resnet.py file and a configuration_resnet.py file in a folder of the current working
directory named resnet_model. The configuration file contains the code for ResnetConfig and the modeling file
contains the code of ResnetModel and ResnetModelForImageClassification.
  Copied .
└── resnet_model
    ├── __init__.py
    ├── configuration_resnet.py
    └── modeling_resnet.py
The __init__.py can be empty, it’s just there so th


Sending the code to the Hub

at Python detects resnet_model can be use as a module.
If copying a modeling files from the library, you will need to replace all the relative imports at the top of the file
to import from the transformers package.
Note that you can re-use (or subclass) an existing configuration/model.
To share your model with the community, follow those steps: first import the ResNet model and config from the newly
created files:
  Copied from resnet_model.configuration_resnet import ResnetConfig
from resnet_model.modeling_resnet import ResnetModel, ResnetModelForImageClassification
Then you have to tell the library you want to copy the code files of those objects when using the save_pretrained
method and properly register them with a given Auto class (especially for models),


Sending the code to the Hub

 just run:
  Copied ResnetConfig.register_for_auto_class()
ResnetModel.register_for_auto_class("AutoModel")
ResnetModelForImageClassification.register_for_auto_class("AutoModelForImageClassification")
Note that there is no need to specify an auto class for the configuration (there is only one auto class for them,
AutoConfig) but it’s different for models. Your custom model could be suitable for many different tasks, so you
have to specify which one of the auto classes is the correct one for your model.
Next, let’s create the config and models as we did before:
  Copied resnet50d_config = ResnetConfig(block_type="bottleneck", stem_width=32, stem_type="deep", avg_down=True)
resnet50d = ResnetModelForImageClassification(resnet50d_config)
pretrained_model = timm.c


Sending the code to the Hub

reate_model("resnet50d", pretrained=True)
resnet50d.model.load_state_dict(pretrained_model.state_dict())
Now to send the model to the Hub, make sure you are logged in. Either run in your terminal:
  Copied huggingface-cli login
or from a notebook:
  Copied from huggingface_hub import notebook_login
notebook_login()
You can then push to your own namespace (or an organization you are a member of) like this:
  Copied resnet50d.push_to_hub("custom-resnet50d")
On top of the modeling weights and the configuration in json format, this also copied the modeling and
configuration .py files in the folder custom-resnet50d and uploaded the result to the Hub. You can check the result
in this model repo.
See the sharing tutorial for more information on the push to Hub method


Using a model with custom code

You can use any configuration, model or tokenizer with custom code files in its repository with the auto-classes and
the from_pretrained method. All files and code uploaded to the Hub are scanned for malware (refer to the Hub security documentation for more information), but you should still
review the model code and author to avoid executing malicious code on your machine. Set trust_remote_code=True to use
a model with custom code:
  Copied from transformers import AutoModelForImageClassification
model = AutoModelForImageClassification.from_pretrained("sgugger/custom-resne


Using a model with custom code

t50d", trust_remote_code=True)
It is also strongly encouraged to pass a commit hash as a revision to make sure the author of the models did not
update the code with some malicious new lines (unless you fully trust the authors of the models).
  Copied commit_hash = "ed94a7c6247d8aedce4647f00f20de6875b5b292"
model = AutoModelForImageClassification.from_pretrained(
    "sgugger/custom-resnet50d", trust_remote_code=True, revision=commit_hash
)
Note that when browsing the commit history of the model repo on the Hub, there is a button to easily copy the commit
hash of any commit.


Registering a model with custom code to the auto classes

If you are writing a library that extends 🤗 Transformers, you may want to extend the auto classes to include your own
model. This is different from pushing the code to the Hub in the sense that users will need to import your library to
get the custom models (contrarily to automatically downloading the model code from the Hub).
As long as your config has a model_type attribute that is different from existing model types, and that your model
classes have the right config_class attributes, you can just add them to the auto classes like this


:
  Copied from transformers import AutoConfig, AutoModel, AutoModelForImageClassification
AutoConfig.register("resnet", ResnetConfig)
AutoModel.register(ResnetConfig, ResnetModel)
AutoModelForImageClassification.register(ResnetConfig, ResnetModelForImageClassification)
Note that the first argument used when registering your custom config to AutoConfig needs to match the model_type
of your custom config, and the first argument used when registering your custom models to any auto model class needs
to match the config_class of those models


How to convert a 🤗 Transformers model to TensorFlow?

Having multiple frameworks available to use with 🤗 Transformers gives you flexibility to play their strengths when
designing your application, but it implies that compatibility must be added on a per-model basis. The good news is that
adding TensorFlow compatibility to an existing model is simpler than adding a new model from scratch!
Whether you wish to have a deeper understanding of large TensorFlow models, make a major open-source contribution, or
enable TensorFlow for your model of choice, this guide is for you.
This guide empowers you, a member of our community, to contribute TensorFlow model weights and/or
architectures to be used in 🤗 Transformers, with minimal supervision from the Hugging Face team. Writing a new model
is no small feat, but hopefully this guide will make it less of a rollercoaster 🎢 and more of a walk in the park 🚶.
Harnessing our collective experiences is absolutely critic


al to make this process increasingly easier, and thus we
highly encourage that you suggest improvements to this guide!
Before you dive deeper, it is recommended that you check the following resources if you’re new to 🤗 Transformers:
General overview of 🤗 Transformers Hugging Face’s TensorFlow Philosophy
In the remainder of this guide, you will learn what’s needed to add a new TensorFlow model architecture, the
procedure to convert PyTorch into TensorFlow model weights, and how to efficiently debug mismatches across ML
frameworks. Let’s get started!
Are you unsure whether the model you wish to use already has a corresponding TensorFlow architecture?   Check the model_type field of the config.json of your model of choice
(example). If the corresponding model folder in
🤗 Transformers has a file whose name starts with “modeling_tf”, it means that it has a corresponding TensorFlow
architecture (example)


Step-by-step guide to add TensorFlow model architecture code

There are many ways to design a large model architecture, and multiple ways of implementing said design. However,
you might recall from our general overview of 🤗 Transformers
that we are an opinionated bunch - the ease of use of 🤗 Transformers relies on consistent design choices. From
experience, we can tell you a few important things about adding TensorFlow models:
Don’t reinvent the wheel! More often than not, there are at least two reference implementations you should check: the
PyTorch equivalent of the model you are implementing and other TensorFlow models for the same class of problems. Great model implementations survive the test of time. This doesn’t happen because the code is pretty, but rather
because the code is clear, easy to debug and bu


ild upon. If you make the life of the maintainers easy with your
TensorFlow implementation, by replicating the same patterns as in other TensorFlow models and minimizing the mismatch
to the PyTorch implementation, you ensure your contribution will be long lived. Ask for help when you’re stuck! The 🤗 Transformers team is here to help, and we’ve probably found solutions to the same
problems you’re facing.
Here’s an overview of the steps needed to add a TensorFlow model architecture:
Select the model you wish to convert Prepare transformers dev environment (Optional) Understand theoretical aspects and the existing implementation Implement the model architecture Implement model tests Submit the pull request (Optional) Build demos and share with the world


1.-3. Prepare your model contribution

1. Select the model you wish to convert
Let’s start off with the basics: the first thing you need to know is the architecture you want to convert. If you
don’t have your eyes set on a specific architecture, asking the 🤗 Transformers team for suggestions is a great way to
maximize your impact - we will guide you towards the most prominent architectures that are missing on the TensorFlow
side. If the specific model you want to use with TensorFlow already has a TensorFlow architecture implementation in
🤗 Transformers but is lacking weights, feel free to jump straight into the
weight conversion section
of this page.
For simplicity, the remainder of this guide assumes you’ve decided to contribute with the TensorFlow version of
BrandNewBert (the same example as in the guide to add a new model from scratch).
Before starting the work on a TensorFlow model architec


1.-3. Prepare your model contribution

ture, double-check that there is no ongoing effort to do so.
You can search for BrandNewBert on the
pull request GitHub page to confirm that there is no
TensorFlow-related pull request.
2. Prepare transformers dev environment
Having selected the model architecture, open a draft PR to signal your intention to work on it. Follow the
instructions below to set up your environment and open a draft PR.
Fork the repository by clicking on the ‘Fork’ button on the
repository’s page. This creates a copy of the code under your GitHub user account. Clone your transformers fork to your local disk, and add the base repository as a remote:
  Copied git clone https://github.com/[your Github handle]/transformers.git
cd transformers
git remote add upstream https://github.com/huggingface/transformers.git
Set up a development environment, for instance by running the following


1.-3. Prepare your model contribution

 command:
  Copied python -m venv .env
source .env/bin/activate
pip install -e ".[dev]"
Depending on your OS, and since the number of optional dependencies of Transformers is growing, you might get a
failure with this command. If that’s the case make sure to install TensorFlow then do:
  Copied pip install -e ".[quality]"
Note: You don’t need to have CUDA installed. Making the new model work on CPU is sufficient.
Create a branch with a descriptive name from your main branch
  Copied git checkout -b add_tf_brand_new_bert
Fetch and rebase to current main
  Copied git fetch upstream
git rebase upstream/main
Add an empty .py file in transformers/src/models/brandnewbert/ named modeling_tf_brandnewbert.py. This will
be your TensorFlow model file. Push the changes to your account using:
  Copied git add .
git commit -m "initial commit"
git push -u origin add_tf_b


1.-3. Prepare your model contribution

rand_new_bert
Once you are satisfied, go to the webpage of your fork on GitHub. Click on “Pull request”. Make sure to add the
GitHub handle of some members of the Hugging Face team as reviewers, so that the Hugging Face team gets notified for
future changes. Change the PR into a draft by clicking on “Convert to draft” on the right of the GitHub pull request web page.
Now you have set up a development environment to port BrandNewBert to TensorFlow in 🤗 Transformers.
3. (Optional) Understand theoretical aspects and the existing implementation
You should take some time to read BrandNewBert’s paper, if such descriptive work exists. There might be large
sections of the paper that are difficult to understand. If this is the case, this is fine - don’t worry! The goal is
not to get a deep theoretical understanding of the paper, but to extract the necessary informa


1.-3. Prepare your model contribution

tion required to
effectively re-implement the model in 🤗 Transformers using TensorFlow. That being said, you don’t have to spend too
much time on the theoretical aspects, but rather focus on the practical ones, namely the existing model documentation
page (e.g. model docs for BERT).
After you’ve grasped the basics of the models you are about to implement, it’s important to understand the existing
implementation. This is a great chance to confirm that a working implementation matches your expectations for the
model, as well as to foresee technical challenges on the TensorFlow side.
It’s perfectly natural that you feel overwhelmed with the amount of information that you’ve just absorbed. It is
definitely not a requirement that you understand all facets of the model at this stage. Nevertheless, we highly
encourage you to clear any pressing questions in our fo