Randomly initializing EncoderDecoderModel from model configurations.

EncoderDecoderModel can be randomly initialized from an encoder and a decoder config. In the following example, we show how to do this using the default BertModel configuration for the encoder and the default BertForCausalLM configuration for the decoder.
  Copied >>> from transformers import BertConfig, EncoderDecoderConfig, EncoderDecoderModel
>>> config_encoder = BertConfig()
>>> config_decoder = BertConfig()
>>> config = EncoderDecoderConfig.from_encoder_decoder_configs(config_encoder, config_decoder)
>>> model = EncoderDecoderModel(config=config)


Initialising EncoderDecoderModel from a pretrained encoder and a pretrained decoder.

EncoderDecoderModel can be initialized from a pretrained encoder checkpoint and a pretrained decoder checkpoint. Note that any pretrained auto-encoding model, e.g. BERT, can serve as the encoder and both pretrained auto-encoding models, e.g. BERT, pretrained causal language models, e.g. GPT2, as well as the pretrained decoder part of sequence-to-sequence models, e.g. decoder of BART, can be used as the decoder.
Depending on which architecture you choose as the decoder, the cross-attention layers might be randomly initialized.
Initializing Enc


oderDecoderModel from a pretrained encoder and decoder checkpoint requires the model to be fine-tuned on a downstream task, as has been shown in the Warm-starting-encoder-decoder blog post.
To do so, the EncoderDecoderModel class provides a EncoderDecoderModel.from_encoder_decoder_pretrained() method.
  Copied >>> from transformers import EncoderDecoderModel, BertTokenizer
>>> tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained("bert-base-uncased", "bert-base-uncased")


Loading an existing EncoderDecoderModel checkpoint and perform inference.

To load fine-tuned checkpoints of the EncoderDecoderModel class, EncoderDecoderModel provides the from_pretrained(...) method just like any other model architecture in Transformers.
To perform inference, one uses the generate method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling.
  Copied >>> from transformers import AutoTokenizer, EncoderDecoderModel
>>> # load a fine-tuned seq2seq model and corresponding tokenizer
>>> model = EncoderDecoderModel.from_pretrained("patrickvonplaten/bert2bert_cnn_daily_mail")
>>> tokenizer = AutoTokenizer.from_pretrained("patrickvonplaten/bert2bert_cnn_daily_mail")
>>> # let's perform inference on a long piece of text
>>> ARTICLE_TO_SUMMARIZE = (
...     "PG&E stated it scheduled the blackouts in response


 to forecasts for high winds "
...     "amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were "
...     "scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow."
... )
>>> input_ids = tokenizer(ARTICLE_TO_SUMMARIZE, return_tensors="pt").input_ids
>>> # autoregressively generate summary (uses greedy decoding by default)
>>> generated_ids = model.generate(input_ids)
>>> generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
>>> print(generated_text)
nearly 800 thousand customers were affected by the shutoffs. the aim is to reduce the risk of wildfires. nearly 800, 000 customers were expected to be affected by high winds amid dry conditions. pg & e said it scheduled the blackouts to last through at least midday tomorrow.


Loading a PyTorch checkpoint into TFEncoderDecoderModel .

TFEncoderDecoderModel.from_pretrained() currently doesn’t support initializing the model from a
pytorch checkpoint. Passing from_pt=True to this method will throw an exception. If there are only pytorch
checkpoints for a particular encoder-decoder model, a workaround is:
  Copied >>> # a workaround to load from pytorch checkpoint
>>> from transformers import EncoderDecoderModel, TFEncoderDecoderModel
>>> _model = EncoderDecoderModel.from_pretrained("patrickvonplaten/bert2bert-cnn_dailymail-fp16")
>>> _model.encoder.save_pretrained("./encoder")
>>> _model.decoder.save_pretrained("./decoder")
>>> model = TFEncoderDecoderModel.from_encoder_decoder_pretrained(
...     "./encoder", "./decoder", encoder_from_pt=True, decoder_from_pt=True
... )
>>> # This is only for copying some specific attributes of this particular model.
>>> model.config = _model.config


Training

Once the model is created, it can be fine-tuned similar to BART, T5 or any other encoder-decoder model.
As you can see, only 2 inputs are required for the model in order to compute a loss: input_ids (which are the
input_ids of the encoded input sequence) and labels (which are the input_ids of the encoded
target sequence).
  Copied >>> from transformers import BertTokenizer, EncoderDecoderModel
>>> tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained("bert-base-uncased", "bert-base-uncased")
>>> model.config.decoder_start_token_id = tokenizer.cls_token_id
>>> model.config.pad_token_id = tokenizer.pad_token_id
>>> input_ids = tokenizer(
...


Training

"The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side.During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was  finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft).Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after t


Training

he Millau Viaduct.",
...     return_tensors="pt",
... ).input_ids
>>> labels = tokenizer(
...     "the eiffel tower surpassed the washington monument to become the tallest structure in the world. it was the first structure to reach a height of 300 metres in paris in 1930. it is now taller than the chrysler building by 5. 2 metres ( 17 ft ) and is the second tallest free - standing structure in paris.",
...     return_tensors="pt",
... ).input_ids
>>> # the forward function automatically creates the correct decoder_input_ids
>>> loss = model(input_ids=input_ids, labels=labels).loss
Detailed colab for training.
This model was contributed by thomwolf. This model’s TensorFlow and Flax versions
were contributed by ydshie


EncoderDecoderConfig

class transformers.EncoderDecoderConfig  < source > ( **kwargs  )   Parameters   kwargs (optional) —
Dictionary of keyword arguments. Notably:
encoder (PretrainedConfig, optional) — An instance of a configuration object that defines
the encoder config.
decoder (PretrainedConfig, optional) — An instance of a configuration object that defines
the decoder config.
    EncoderDecoderConfig is the configuration class to store the configuration of a EncoderDecoderModel. It is
used to instantiate an Encoder Decoder model according to the specified arguments, defining the encoder and decoder
configs. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation


EncoderDecoderConfig

from PretrainedConfig for more information.  Examples:   Copied >>> from transformers import BertConfig, EncoderDecoderConfig, EncoderDecoderModel
>>> # Initializing a BERT bert-base-uncased style configuration
>>> config_encoder = BertConfig()
>>> config_decoder = BertConfig()
>>> config = EncoderDecoderConfig.from_encoder_decoder_configs(config_encoder, config_decoder)
>>> # Initializing a Bert2Bert model (with random weights) from the bert-base-uncased style configurations
>>> model = EncoderDecoderModel(config=config)
>>> # Accessing the model configuration
>>> config_encoder = model.config.encoder
>>> config_decoder = model.config.decoder
>>> # set decoder config to causal lm
>>> config_decoder.is_decoder


EncoderDecoderConfig

= True
>>> config_decoder.add_cross_attention = True
>>> # Saving the model, including its configuration
>>> model.save_pretrained("my-model")
>>> # loading model and config from pretrained folder
>>> encoder_decoder_config = EncoderDecoderConfig.from_pretrained("my-model")
>>> model = EncoderDecoderModel.from_pretrained("my-model", config=encoder_decoder_config)  from_encoder_decoder_configs  < source > ( encoder_config: PretrainedConfig decoder_config: PretrainedConfig **kwargs  ) → EncoderDecoderConfig  Returns
EncoderDecoderConfig

An instance of a configuration object
  Instantiate a EncoderDecoderConfig (or a derived class) from a pre-trained encoder model configuration and
decoder model configuration.


EncoderDecoderModel

class transformers.EncoderDecoderModel  < source > ( config: typing.Optional[transformers.configuration_utils.PretrainedConfig] = None encoder: typing.Optional[transformers.modeling_utils.PreTrainedModel] = None decoder: typing.Optional[transformers.modeling_utils.PreTrainedModel] = None  )   Parameters   config (EncoderDecoderConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    This class can be used to initialize a sequence-to-sequence model with any pretrained autoencoding model as the
encoder and any pretrained autoregressive model as the decoder. The encoder is loaded via
from_pretrained() function and the decoder is loaded via from_pretrained()
function. Cross-attention layers are automatically added to the decoder and should be fine-tuned on a downstream
gener


EncoderDecoderModel

ative task, like summarization. The effectiveness of initializing sequence-to-sequence models with pretrained checkpoints for sequence generation
tasks was shown in Leveraging Pre-trained Checkpoints for Sequence Generation
Tasks by Sascha Rothe, Shashi Narayan, Aliaksei Severyn. Michael Matena, Yanqi
Zhou, Wei Li, Peter J. Liu. After such an Encoder Decoder model has been trained/fine-tuned, it can be saved/loaded just like any other models
(see the examples for more information). This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior. EncoderDecoderModel is a generic model class that will be instantiated as a trans


EncoderDecoderModel

former architecture with one
of the base model classes of the library as encoder and another one as decoder when created with the
:meth~transformers.AutoModel.from_pretrained class method for the encoder and
:meth~transformers.AutoModelForCausalLM.from_pretrained class method for the decoder.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None decoder_input_ids: typing.Optional[torch.LongTensor] = None decoder_attention_mask: typing.Optional[torch.BoolTensor] = None encoder_outputs: typing.Optional[typing.Tuple[torch.FloatTensor]] = None past_key_values: typing.Tuple[typing.Tuple[torch.FloatTensor]] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None decoder_inputs_embeds: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.LongTensor] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states:


EncoderDecoderModel

typing.Optional[bool] = None return_dict: typing.Optional[bool] = None **kwargs  ) → transformers.modeling_outputs.Seq2SeqLMOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using PreTrainedTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   decoder_input_ids (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Indices of decoder input sequence tokens in the vocabulary.
Indices can be obtained using PreTrainedTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokeniz


EncoderDecoderModel

er.call() for details.
What are input IDs?
If past_key_values is used, optionally only the last decoder_input_ids have to be input (see
past_key_values).
For training, decoder_input_ids are automatically created by the model by shifting the labels to the
right, replacing -100 by the pad_token_id and prepending them with the decoder_start_token_id.   decoder_attention_mask (torch.BoolTensor of shape (batch_size, target_sequence_length), optional) —
Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also
be used by default.   encoder_outputs (tuple(torch.FloatTensor), optional) —
This tuple must consist of (last_hidden_state, optional: hidden_states, optional: attentions)
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) is a tensor
of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the
decoder.   past_key_values (tuple(tuple(torch.F


EncoderDecoderModel

loatTensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)) —
Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   decoder_inputs_embeds (torch.FloatTensor of shape (batch_size, target_sequence_length, hidden_size),


EncoderDecoderModel

optional) —
Optionally, instead of passing decoder_input_ids you can choose to directly pass an embedded
representation. This is useful if you want more control over how to convert decoder_input_ids indices
into associated vectors than the model’s internal embedding lookup matrix.   labels (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Labels for computing the masked language modeling loss for the decoder. Indices should be in [-100, 0, ..., config.vocab_size] (see input_ids docstring) Tokens with indices set to -100 are ignored
(masked), the loss is only computed for the tokens with labels in [0, ..., config.vocab_size]   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   ou


EncoderDecoderModel

tput_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
If set to True, the model will return a ~utils.Seq2SeqLMOutput instead of a plain tuple.   kwargs (optional) — Remaining dictionary of keyword arguments. Keyword arguments come in two flavors:
Without a prefix which will be input as **encoder_kwargs for the encoder forward function.
With a decoder_ prefix which will be input as **decoder_kwargs for the decoder forward function.
   Returns
transformers.modeling_outputs.Seq2SeqLMOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.Seq2SeqLMOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (EncoderDecoderConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provid


EncoderDecoderModel

ed) — Language modeling loss.
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of t


EncoderDecoderModel

orch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the decoder at the output of each layer plus the initial embedding outputs.
decoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention l


EncoderDecoderModel

ayer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the initial embedding outputs.
encoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_lengt


EncoderDecoderModel

h, sequence_length).
Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
  The EncoderDecoderModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import EncoderDecoderModel, BertTokenizer
>>> import torch
>>> tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained(
...     "bert-base-uncased", "bert-base-uncased"
... )  # initialize Bert2Bert from pre-trained checkpoints
>>> # training
>>> model.config.decoder_start_token_id = tokenizer.cls_token_id
>>> model.config.pad_token_id = tokenizer.pad_token_id
>>> model.con


EncoderDecoderModel

fig.vocab_size = model.config.decoder.vocab_size
>>> input_ids = tokenizer("This is a really long text", return_tensors="pt").input_ids
>>> labels = tokenizer("This is the corresponding summary", return_tensors="pt").input_ids
>>> outputs = model(input_ids=input_ids, labels=labels)
>>> loss, logits = outputs.loss, outputs.logits
>>> # save and load from pretrained
>>> model.save_pretrained("bert2bert")
>>> model = EncoderDecoderModel.from_pretrained("bert2bert")
>>> # generation
>>> generated = model.generate(input_ids)  from_encoder_decoder_pretrained  < source > ( encoder_pretrained_model_name_or_path: str = None decoder_pretrained_model_name_or_path: str = None *model_args **kwargs  )   Parameters   encoder_pretrained_model_name_or_path (str, optional) —
Information necessary to initiate the encoder. Can be either:
A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like


EncoderDecoderModel

bert-base-uncased, or namespaced under a
user or organization name, like dbmdz/bert-base-german-cased.
A path to a directory containing model weights saved using
save_pretrained(), e.g., ./my_model_directory/.
A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In
this case, from_tf should be set to True and a configuration object should be provided as
config argument. This loading path is slower than converting the TensorFlow checkpoint in a
PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
   decoder_pretrained_model_name_or_path (str, optional, defaults to None) —
Information necessary to initiate the decoder. Can be either:
A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like bert-base-uncased, or namespaced under a
user or organization name, like dbmdz/bert-base-german-cased.
A path to a


EncoderDecoderModel

 directory containing model weights saved using
save_pretrained(), e.g., ./my_model_directory/.
A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In
this case, from_tf should be set to True and a configuration object should be provided as
config argument. This loading path is slower than converting the TensorFlow checkpoint in a
PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
   model_args (remaining positional arguments, optional) —
All remaining positional arguments will be passed to the underlying model’s __init__ method.   kwargs (remaining dictionary of keyword arguments, optional) —
Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
output_attentions=True).
To update the encoder configuration, use the prefix encoder_ for each configuration parameter.
To update the decoder configuration, use the prefix decoder_ for each confi


EncoderDecoderModel

guration parameter.
To update the parent model configuration, do not use a prefix for each configuration parameter.
Behaves differently depending on whether a config is provided or automatically loaded.    Instantiate an encoder and a decoder from one or two base classes of the library from pretrained model
checkpoints. The model is set in evaluation mode by default using model.eval() (Dropout modules are deactivated). To train
the model, you need to first set it back in training mode with model.train().  Example:   Copied >>> from transformers import EncoderDecoderModel
>>> # initialize a bert2bert from two pretrained BERT models. Note that the cross-attention layers will be randomly initialized
>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained("bert-base-uncased", "bert-base-uncased")
>>> # saving model after fine-tuning
>>> model.save_pretrained("./bert2bert")
>>> # load fine-tuned model
>>> model = EncoderDecoderModel.from_pretrained("./bert2be


TFEncoderDecoderModel

class transformers.TFEncoderDecoderModel  < source > ( *args **kwargs  )   Parameters   config (EncoderDecoderConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    This class can be used to initialize a sequence-to-sequence model with any pretrained autoencoding model as the
encoder and any pretrained autoregressive model as the decoder. The encoder is loaded via
from_pretrained() function and the decoder is loaded via from_pretrained()
function. Cross-attention layers are automatically added to the decoder and should be fine-tuned on a downstream
generative task, like summarization. The effectiveness of initializing sequence-to-sequence models with pretrained checkpoints for sequence generation
tasks was shown in Leveraging Pre-trained Checkpoints for Sequence


TFEncoderDecoderModel

 Generation
Tasks by Sascha Rothe, Shashi Narayan, Aliaksei Severyn. Michael Matena, Yanqi
Zhou, Wei Li, Peter J. Liu. After such an Encoder Decoder model has been trained/fine-tuned, it can be saved/loaded just like any other models
(see the examples for more information). This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior. TFEncoderDecoderModel is a generic model class that will be instantiated as a transformer architecture with one
of the base model classes of the library as encoder and another one as decoder when created with the
from_pretrained() class method for the encoder and from_pretrained() class
me


TFEncoderDecoderModel

thod for the decoder.  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None decoder_input_ids: np.ndarray | tf.Tensor | None = None decoder_attention_mask: np.ndarray | tf.Tensor | None = None encoder_outputs: np.ndarray | tf.Tensor | None = None past_key_values: Tuple[Tuple[tf.Tensor]] | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None decoder_inputs_embeds: np.ndarray | tf.Tensor | None = None labels: np.ndarray | tf.Tensor | None = None use_cache: Optional[bool] = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None training: bool = False **kwargs  ) → transformers.modeling_tf_outputs.TFSeq2SeqLMOutput or tuple(tf.Tensor)  Parameters   input_ids (np.ndarray, tf.Tensor, List[tf.Tensor] `Dict[str, tf.Tensor] or Dict[str, np.ndarray] and each example must have the shape (batch_size, sequence_length)) —
Indices


TFEncoderDecoderModel

 of input sequence tokens in the vocabulary.
Indices can be obtained using PreTrainedTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   decoder_input_ids (np.ndarray or tf.Tensor of shape (batch_size, target_sequence_length), optional) —
Indices of decoder input sequence tokens in the vocabulary.
Indices can be obtained using PreTrainedTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?
If past_key_values is used, optionally only the last decoder_input_ids have to be input (see
past_key_values).
Provide for sequence to sequence training to the decoder. Indices can be obt


TFEncoderDecoderModel

ained using
PreTrainedTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for
details.   decoder_attention_mask (np.ndarray or tf.Tensor of shape (batch_size, target_sequence_length), optional) —
Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also
be used by default.   encoder_outputs (tuple(tuple(tf.Tensor), optional) —
This tuple must consist of (last_hidden_state, optional: hidden_states, optional: attentions)
last_hidden_state (tf.Tensor of shape (batch_size, sequence_length, hidden_size)) is a tensor of hidden-states at the output
of the last layer of the encoder. Used in the cross-attention of the decoder.   past_key_values (tuple(tuple(tf.Tensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)) —
Contains precomputed key and value hidden states of the attention blocks. Can be used to speed


TFEncoderDecoderModel

 up decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).   inputs_embeds (np.ndarray or tf.Tensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   decoder_inputs_embeds (np.ndarray or tf.Tensor of shape (batch_size, target_sequence_length, hidden_size), optional) —
Optionally, instead of passing decoder_input_ids you can choose to directly pass an embedded
representation. This is useful if you want more control over how to convert decoder_input_ids indices
into associated ve


TFEncoderDecoderModel

ctors than the model’s internal embedding lookup matrix.   labels (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Labels for computing the masked language modeling loss for the decoder. Indices should be in [-100, 0, ..., config.vocab_size] (see input_ids docstring) Tokens with indices set to -100 are ignored
(masked), the loss is only computed for the tokens with labels in [0, ..., config.vocab_size]   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
If set to True, the mode


TFEncoderDecoderModel

l will return a ~utils.Seq2SeqLMOutput instead of a plain tuple.   training (bool, optional, defaults to False) —
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).   kwargs (optional) — Remaining dictionary of keyword arguments. Keyword arguments come in two flavors:
Without a prefix which will be input as **encoder_kwargs for the encoder forward function.
With a decoder_ prefix which will be input as `**decoder_kwargs“ for the decoder forward function.
   Returns
transformers.modeling_tf_outputs.TFSeq2SeqLMOutput or tuple(tf.Tensor)

A transformers.modeling_tf_outputs.TFSeq2SeqLMOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (EncoderDecoderConfig) and inputs.
loss (tf.Tensor of shape (n,), optional, where n is the number of non-masked labels, returned w


TFEncoderDecoderModel

hen labels is provided) — Language modeling loss.
logits (tf.Tensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
past_key_values (List[tf.Tensor], optional, returned when use_cache=True is passed or when config.use_cache=True) — List of tf.Tensor of length config.n_layers, with each tensor of shape (2, batch_size, num_heads, sequence_length, embed_size_per_head)).
Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be
used (see past_key_values input) to speed up sequential decoding.
decoder_hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the decod


TFEncoderDecoderModel

er at the output of each layer plus the initial embedding outputs.
decoder_attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
cross_attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
encoder_last_hidden_state (tf.Tensor of shape (batch_size, sequence_length, hidden_size), optional) — Sequence of hidden-states at


TFEncoderDecoderModel

the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the initial embedding outputs.
encoder_attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
  The TFEncoderDecoderModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this f


TFEncoderDecoderModel

unction, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import TFEncoderDecoderModel, BertTokenizer
>>> # initialize a bert2gpt2 from a pretrained BERT and GPT2 models. Note that the cross-attention layers will be randomly initialized
>>> model = TFEncoderDecoderModel.from_encoder_decoder_pretrained("bert-base-cased", "gpt2")
>>> tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
>>> # forward
>>> input_ids = tokenizer.encode(
...     "Hello, my dog is cute", add_special_tokens=True, return_tensors="tf"
... )  # Batch size 1
>>> outputs = model(input_ids=input_ids, decoder_input_ids=input_ids)
>>> # training
>>> outputs = model(input_ids=input_ids, decoder_input_ids=input_ids, labels=input_ids)
>>> loss, logits = outputs.loss, outputs.logits
>>> # save and load from pretrained


TFEncoderDecoderModel

>>> model.save_pretrained("bert2gpt2")
>>> model = TFEncoderDecoderModel.from_pretrained("bert2gpt2")
>>> # generation
>>> generated = model.generate(input_ids, decoder_start_token_id=model.config.decoder.bos_token_id)  from_encoder_decoder_pretrained  < source > ( encoder_pretrained_model_name_or_path: str = None decoder_pretrained_model_name_or_path: str = None *model_args **kwargs  )   Parameters   encoder_pretrained_model_name_or_path (str, optional) —
Information necessary to initiate the encoder. Can be either:
A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like bert-base-uncased, or namespaced under a
user or organization name, like dbmdz/bert-base-german-cased.
A path to a directory containing model weights saved using
save_pretrained(), e.g., ./my_model_directory/.
A path or url to a pytorch index checkpoint file (e.g, ./pt_model/). In this case,
encoder


TFEncoderDecoderModel

_from_pt should be set to True.
   decoder_pretrained_model_name_or_path (str, optional, defaults to None) —
Information necessary to initiate the decoder. Can be either:
A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like bert-base-uncased, or namespaced under a
user or organization name, like dbmdz/bert-base-german-cased.
A path to a directory containing model weights saved using
save_pretrained(), e.g., ./my_model_directory/.
A path or url to a pytorch checkpoint file (e.g, ./pt_model/). In this case,
decoder_from_pt should be set to True.
   model_args (remaining positional arguments, optional) —
All remaning positional arguments will be passed to the underlying model’s __init__ method.   kwargs (remaining dictionary of keyword arguments, optional) —
Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
output_at


TFEncoderDecoderModel

tentions=True).
To update the encoder configuration, use the prefix encoder_ for each configuration parameter.
To update the decoder configuration, use the prefix decoder_ for each configuration parameter.
To update the parent model configuration, do not use a prefix for each configuration parameter.
Behaves differently depending on whether a config is provided or automatically loaded.    Instantiate an encoder and a decoder from one or two base classes of the library from pretrained model
checkpoints.  Example:   Copied >>> from transformers import TFEncoderDecoderModel
>>> # initialize a bert2gpt2 from two pretrained BERT models. Note that the cross-attention layers will be randomly initialized
>>> model = TFEncoderDecoderModel.from_encoder_decoder_pretrained("bert-base-uncased", "gpt2")
>>> # saving model after fine-tuning
>>> model.save_pretrained("./bert2gpt2")
>>> # load fine-tuned model
>>> model = TFEncoderDecoderModel.from_pretrained("./bert2gpt2


FlaxEncoderDecoderModel

class transformers.FlaxEncoderDecoderModel  < source > ( config: EncoderDecoderConfig input_shape: typing.Optional[typing.Tuple] = None seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True **kwargs  )   Parameters   config (EncoderDecoderConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.   dtype (jax.numpy.dtype, optional, defaults to jax.numpy.float32) —
The data type of the computation. Can be one of jax.numpy.float32, jax.numpy.float16 (on GPUs) and
jax.numpy.bfloat16 (on TPUs).
This can be used to enable mixed-precision training or half-precision inference on GPUs or TPUs. If
specified all the computation will be performed with the given dtype.
Note that this only specifies the dtype of the computation


FlaxEncoderDecoderModel

 and does not influence the dtype of model
parameters.
If you wish to change the dtype of the model parameters, see to_fp16() and
to_bf16().    This class can be used to initialize a sequence-to-sequence model with any pretrained autoencoding model as the
encoder and any pretrained autoregressive model as the decoder. The encoder is loaded via
from_pretrained() function and the decoder is loaded via from_pretrained()
function. Cross-attention layers are automatically added to the decoder and should be fine-tuned on a downstream
generative task, like summarization. The effectiveness of initializing sequence-to-sequence models with pretrained checkpoints for sequence generation
tasks was shown in Leveraging Pre-trained Checkpoints for Sequence Generation
Tasks by Sascha Rothe, Shashi Narayan, Aliaksei Severyn. Michael Matena, Yanqi
Zhou, Wei Li, Peter J. Liu. After such an Encoder Decoder model has been trained/fine-tuned, it c


FlaxEncoderDecoderModel

an be saved/loaded just like any other models
(see the examples for more information). This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a Flax Linen
flax.nn.Module subclass. Use it as a
regular Flax Module and refer to the Flax documentation for all matter related to general usage and behavior. FlaxEncoderDecoderModel is a generic model class that will be instantiated as a transformer architecture with
the module (flax.nn.Module) of one of the base model classes of the library as encoder module and another one as
decoder module when created with the :meth~transformers.FlaxAutoModel.from_pretrained class method for the
encoder and :meth~transformers.FlaxAutoModelForCausalLM.from_pretrained class method for the decoder.  __call__  <


FlaxEncoderDecoderModel

 source > ( input_ids: Array attention_mask: typing.Optional[jax.Array] = None decoder_input_ids: typing.Optional[jax.Array] = None decoder_attention_mask: typing.Optional[jax.Array] = None position_ids: typing.Optional[jax.Array] = None decoder_position_ids: typing.Optional[jax.Array] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None train: bool = False params: dict = None dropout_rng: PRNGKey = None  ) → transformers.modeling_flax_outputs.FlaxSeq2SeqLMOutput or tuple(torch.FloatTensor)  Parameters   input_ids (jnp.ndarray of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using PreTrainedTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention


FlaxEncoderDecoderModel

_mask (jnp.ndarray of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   decoder_input_ids (jnp.ndarray of shape (batch_size, target_sequence_length), optional) —
Indices of decoder input sequence tokens in the vocabulary.
Indices can be obtained using PreTrainedTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are decoder input IDs?
For sequence to sequence training, decoder_input_ids should be provided. decoder_input_ids should be
created outside of the model by shifting the labels to the right, replacing -100 by the pad_token_id
and prepending them with the decoder_start_token_id.   decoder_attention_mask (jnp.ndarray of shape (batch_size, target_sequence_length), optional) —
Default behavior: generat


FlaxEncoderDecoderModel

e a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also
be used by default.   position_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.encoder.max_position_embeddings - 1].   decoder_position_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Indices of positions of each decoder input sequence tokens in the position embeddings. Selected in the
range [0, config.decoder.max_position_embeddings - 1].   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
If


FlaxEncoderDecoderModel

 set to True, the model will return a ~utils.FlaxSeq2SeqLMOutput instead of a plain tuple.   Returns
transformers.modeling_flax_outputs.FlaxSeq2SeqLMOutput or tuple(torch.FloatTensor)

A transformers.modeling_flax_outputs.FlaxSeq2SeqLMOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (EncoderDecoderConfig) and inputs.
logits (jnp.ndarray of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
past_key_values (tuple(tuple(jnp.ndarray)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(jnp.ndarray) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(b


FlaxEncoderDecoderModel

atch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
decoder_hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the decoder at the output of each layer plus the initial embedding outputs.
decoder_attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder, after


FlaxEncoderDecoderModel

 the attention softmax, used to compute the weighted average in the
self-attention heads.
cross_attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
encoder_last_hidden_state (jnp.ndarray of shape (batch_size, sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch


FlaxEncoderDecoderModel

_size, sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the initial embedding outputs.
encoder_attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
  The FlaxEncoderDecoderModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import FlaxEncoderDecoderModel, BertTokenizer, G


FlaxEncoderDecoderModel

PT2Tokenizer
>>> # load a fine-tuned bert2gpt2 model
>>> model = FlaxEncoderDecoderModel.from_pretrained("patrickvonplaten/bert2gpt2-cnn_dailymail-fp16")
>>> # load input & output tokenizer
>>> tokenizer_input = BertTokenizer.from_pretrained("bert-base-cased")
>>> tokenizer_output = GPT2Tokenizer.from_pretrained("gpt2")
>>> article = '''Sigma Alpha Epsilon is under fire for a video showing party-bound fraternity members
>>> singing a racist chant. SAE's national chapter suspended the students,
>>> but University of Oklahoma President David Boren took it a step further,
>>> saying the university's affiliation with the fraternity is permanently done.'''
>>> input_ids = tokenizer_input(article, add_special_tokens=True, return_tensors="np").input_ids
>>> # use GPT2's eos_token as the pad as well as eos token
>>> model.config.eos_token_id = model.config.decoder.eos_token_id
>>> model.config.pad_token_id = model.config.eos_token_id


FlaxEncoderDecoderModel

>>> sequences = model.generate(input_ids, num_beams=4, max_length=12).sequences
>>> summary = tokenizer_output.batch_decode(sequences, skip_special_tokens=True)[0]
>>> assert summary == "SAS Alpha Epsilon suspended Sigma Alpha Epsilon members"  from_encoder_decoder_pretrained  < source > ( encoder_pretrained_model_name_or_path: typing.Union[str, os.PathLike, NoneType] = None decoder_pretrained_model_name_or_path: typing.Union[str, os.PathLike, NoneType] = None *model_args **kwargs  )   Parameters   encoder_pretrained_model_name_or_path (Union[str, os.PathLike], optional) —
Information necessary to initiate the encoder. Can be either:
A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like bert-base-uncased, or namespaced under a
user or organization name, like dbmdz/bert-base-german-cased.
A path to a directory containing model weights


FlaxEncoderDecoderModel

saved using
save_pretrained(), e.g., ./my_model_directory/.
   decoder_pretrained_model_name_or_path (Union[str, os.PathLike], optional, defaults to None) —
Information necessary to initiate the decoder. Can be either:
A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like bert-base-uncased, or namespaced under a
user or organization name, like dbmdz/bert-base-german-cased.
A path to a directory containing model weights saved using
save_pretrained(), e.g., ./my_model_directory/.
   model_args (remaining positional arguments, optional) —
All remaning positional arguments will be passed to the underlying model’s __init__ method.   kwargs (remaining dictionary of keyword arguments, optional) —
Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
output_attentions=True).
To update the encoder


FlaxEncoderDecoderModel

configuration, use the prefix encoder_ for each configuration parameter.
To update the decoder configuration, use the prefix decoder_ for each configuration parameter.
To update the parent model configuration, do not use a prefix for each configuration parameter.
Behaves differently depending on whether a config is provided or automatically loaded.    Instantiate an encoder and a decoder from one or two base classes of the library from pretrained model
checkpoints.  Example:   Copied >>> from transformers import FlaxEncoderDecoderModel
>>> # initialize a bert2gpt2 from pretrained BERT and GPT2 models. Note that the cross-attention layers will be randomly initialized
>>> model = FlaxEncoderDecoderModel.from_encoder_decoder_pretrained("bert-base-cased", "gpt2")
>>> # saving model after fine-tuning
>>> model.save_pretrained("./bert2gpt2")
>>> # load fine-tuned model
>>> model = FlaxEncoderDecoderModel.from_pretrained("./bert2gpt


Overview

The VITS model was proposed in Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech by Jaehyeon Kim, Jungil Kong, Juhee Son.
VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech) is an end-to-end
speech synthesis model that predicts a speech waveform conditional on an input text sequence. It is a conditional variational
autoencoder (VAE) comprised of a posterior encoder, decoder, and conditional prior.
A set of spectrogram-based acoustic features are predicted by the flow-based module, which is formed of a Transformer-based
text encoder and multiple coupling layers. The spectrogram is decoded using a stack of transposed convolutional layers,
much in the same style as the HiFi-GAN vocoder. Motivated by the one-to-many nature of the TTS problem, where the same text
input can be spoken in multiple ways, the model also includes a stochastic duration predictor, which allows the model to
synthesise speech with different


Overview

 rhythms from the same input text.
The model is trained end-to-end with a combination of losses derived from variational lower bound and adversarial training.
To improve the expressiveness of the model, normalizing flows are applied to the conditional prior distribution. During
inference, the text encodings are up-sampled based on the duration prediction module, and then mapped into the
waveform using a cascade of the flow module and HiFi-GAN decoder. Due to the stochastic nature of the duration predictor,
the model is non-deterministic, and thus requires a fixed seed to generate the same speech waveform.
The abstract from the paper is the following:
Several recent end-to-end text-to-speech (TTS) models enabling single-stage training and parallel sampling have been proposed, but their sample quality does not match that of two-stage TTS systems. In this work, we present a parallel end-to-end TTS method that generates more natural sounding audio than current two-stage models. Our meth


Overview

od adopts variational inference augmented with normalizing flows and an adversarial training process, which improves the expressive power of generative modeling. We also propose a stochastic duration predictor to synthesize speech with diverse rhythms from input text. With the uncertainty modeling over latent variables and the stochastic duration predictor, our method expresses the natural one-to-many relationship in which a text input can be spoken in multiple ways with different pitches and rhythms. A subjective human evaluation (mean opinion score, or MOS) on the LJ Speech, a single speaker dataset, shows that our method outperforms the best publicly available TTS systems and achieves a MOS comparable to ground truth.
This model can also be used with TTS checkpoints from Massively Multilingual Speech (MMS)
as these checkpoints use the same architecture and a slightly modified tokenizer.
This model was contributed by Matthijs and sanchit-gandhi. The original code can be found here


Model Usage

Both the VITS and MMS-TTS checkpoints can be used with the same API. Since the flow-based model is non-deterministic, it
is good practice to set a seed to ensure reproducibility of the outputs. For languages with a Roman alphabet,
such as English or French, the tokenizer can be used directly to pre-process the text inputs. The following code example
runs a forward pass using the MMS-TTS English checkpoint:
  Copied import torch
from transformers import VitsTokenizer, VitsModel, set_seed
tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-eng")
model = VitsModel.from_pretrained("facebook/mms-tts-eng")
inputs = tokenizer(text="Hello - my dog is cute", return_tensors="pt")
set_seed(555)  # make deterministic
with torch.no_grad():
   outputs = model(**inputs)
waveform = outputs.waveform[0]
The resulting waveform can be saved


Model Usage

as a .wav file:
  Copied import scipy
scipy.io.wavfile.write("techno.wav", rate=model.config.sampling_rate, data=waveform)
Or displayed in a Jupyter Notebook / Google Colab:
  Copied from IPython.display import Audio
Audio(waveform, rate=model.config.sampling_rate)
For certain languages with a non-Roman alphabet, such as Arabic, Mandarin or Hindi, the uroman
perl package is required to pre-process the text inputs to the Roman alphabet.
You can check whether you require the uroman package for your language by inspecting the is_uroman attribute of
the pre-trained tokenizer:
  Copied from transformers import VitsTokenizer
tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-eng")
print(tokenizer.is_uroman)
If required, you should apply the uroman package to your text inputs prior to passing them to the VitsTokenizer,
since cu


Model Usage

rrently the tokenizer does not support performing the pre-processing itself.
To do this, first clone the uroman repository to your local machine and set the bash variable UROMAN to the local path:
  Copied git clone https://github.com/isi-nlp/uroman.git
cd uroman
export UROMAN=$(pwd)
You can then pre-process the text input using the following code snippet. You can either rely on using the bash variable
UROMAN to point to the uroman repository, or you can pass the uroman directory as an argument to the uromaize function:
  Copied import torch
from transformers import VitsTokenizer, VitsModel, set_seed
import os
import subprocess
tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-kor")
model = VitsModel.from_pretrained("facebook/mms-tts-kor")
def uromanize(input_string, uroman_path):
    """Convert non-Roman strings to Rom


Model Usage

an using the `uroman` perl package."""
    script_path = os.path.join(uroman_path, "bin", "uroman.pl")
    command = ["perl", script_path]
    process = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    # Execute the perl command
    stdout, stderr = process.communicate(input=input_string.encode())
    if process.returncode != 0:
        raise ValueError(f"Error {process.returncode}: {stderr.decode()}")
    # Return the output as a string and skip the new-line character at the end
    return stdout.decode()[:-1]
text = "이봐 무슨 일이야"
uromaized_text = uromanize(text, uroman_path=os.environ["UROMAN"])
inputs = tokenizer(text=uromaized_text, return_tensors="pt")
set_seed(555)  # make deterministic
with torch.no_grad():
   outputs = model(inputs["input_ids"])
waveform = outputs.waveform


VitsConfig

class transformers.VitsConfig  < source > ( vocab_size = 38 hidden_size = 192 num_hidden_layers = 6 num_attention_heads = 2 window_size = 4 use_bias = True ffn_dim = 768 layerdrop = 0.1 ffn_kernel_size = 3 flow_size = 192 spectrogram_bins = 513 hidden_act = 'relu' hidden_dropout = 0.1 attention_dropout = 0.1 activation_dropout = 0.1 initializer_range = 0.02 layer_norm_eps = 1e-05 use_stochastic_duration_prediction = True num_speakers = 1 speaker_embedding_size = 0 upsample_initial_channel = 512 upsample_rates = [8, 8, 2, 2] upsample_kernel_sizes = [16, 16, 4, 4] resblock_kernel_sizes = [3, 7, 11] resblock_dilation_sizes = [[1, 3, 5], [1, 3, 5], [1, 3, 5]] leaky_relu_slope = 0.1 depth_separable_channels = 2 depth_separable_num_layers = 3 duration_predictor_flow_bins = 10 duration_predictor_tail_bound = 5.0 duration_predictor_kernel_size = 3 duration_predictor_dropout = 0.5 duration_predictor_num_flows = 4 duration_predictor_filter_channels = 256 prior_e


VitsConfig

ncoder_num_flows = 4 prior_encoder_num_wavenet_layers = 4 posterior_encoder_num_wavenet_layers = 16 wavenet_kernel_size = 5 wavenet_dilation_rate = 1 wavenet_dropout = 0.0 speaking_rate = 1.0 noise_scale = 0.667 noise_scale_duration = 0.8 sampling_rate = 16000 **kwargs  )   Parameters   vocab_size (int, optional, defaults to 38) —
Vocabulary size of the VITS model. Defines the number of different tokens that can be represented by the
inputs_ids passed to the forward method of VitsModel.   hidden_size (int, optional, defaults to 192) —
Dimensionality of the text encoder layers.   num_hidden_layers (int, optional, defaults to 6) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 2) —
Number of attention heads for each attention layer in the Transformer encoder.   window_size (int, optional, defaults to 4) —
Window size for the relative positional embeddings in the attention layers of the Transformer en


VitsConfig

coder.   use_bias (bool, optional, defaults to True) —
Whether to use bias in the key, query, value projection layers in the Transformer encoder.   ffn_dim (int, optional, defaults to 768) —
Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.   layerdrop (float, optional, defaults to 0.1) —
The LayerDrop probability for the encoder. See the [LayerDrop paper](see https://arxiv.org/abs/1909.11556)
for more details.   ffn_kernel_size (int, optional, defaults to 3) —
Kernel size of the 1D convolution layers used by the feed-forward network in the Transformer encoder.   flow_size (int, optional, defaults to 192) —
Dimensionality of the flow layers.   spectrogram_bins (int, optional, defaults to 513) —
Number of frequency bins in the target spectrogram.   hidden_act (str or function, optional, defaults to "relu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"re


VitsConfig

lu", "selu" and "gelu_new" are supported.   hidden_dropout (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the embeddings and encoder.   attention_dropout (float, optional, defaults to 0.1) —
The dropout ratio for the attention probabilities.   activation_dropout (float, optional, defaults to 0.1) —
The dropout ratio for activations inside the fully connected layer.   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   layer_norm_eps (float, optional, defaults to 1e-05) —
The epsilon used by the layer normalization layers.   use_stochastic_duration_prediction (bool, optional, defaults to True) —
Whether to use the stochastic duration prediction module or the regular duration predictor.   num_speakers (int, optional, defaults to 1) —
Number of speakers if this is a multi-speaker model.   speaker_embedding_


VitsConfig

size (int, optional, defaults to 0) —
Number of channels used by the speaker embeddings. Is zero for single-speaker models.   upsample_initial_channel (int, optional, defaults to 512) —
The number of input channels into the HiFi-GAN upsampling network.   upsample_rates (Tuple[int] or List[int], optional, defaults to [8, 8, 2, 2]) —
A tuple of integers defining the stride of each 1D convolutional layer in the HiFi-GAN upsampling network.
The length of upsample_rates defines the number of convolutional layers and has to match the length of
upsample_kernel_sizes.   upsample_kernel_sizes (Tuple[int] or List[int], optional, defaults to [16, 16, 4, 4]) —
A tuple of integers defining the kernel size of each 1D convolutional layer in the HiFi-GAN upsampling
network. The length of upsample_kernel_sizes defines the number of convolutional layers and has to match
the length of upsample_rates.   resblock_kernel_sizes (Tuple[int] or List[int], optional, defaults to


VitsConfig

 [3, 7, 11]) —
A tuple of integers defining the kernel sizes of the 1D convolutional layers in the HiFi-GAN
multi-receptive field fusion (MRF) module.   resblock_dilation_sizes (Tuple[Tuple[int]] or List[List[int]], optional, defaults to [[1, 3, 5], [1, 3, 5], [1, 3, 5]]) —
A nested tuple of integers defining the dilation rates of the dilated 1D convolutional layers in the
HiFi-GAN multi-receptive field fusion (MRF) module.   leaky_relu_slope (float, optional, defaults to 0.1) —
The angle of the negative slope used by the leaky ReLU activation.   depth_separable_channels (int, optional, defaults to 2) —
Number of channels to use in each depth-separable block.   depth_separable_num_layers (int, optional, defaults to 3) —
Number of convolutional layers to use in each depth-separable block.   duration_predictor_flow_bins (int, optional, defaults to 10) —
Number of channels to map using the unonstrained rational spline in the duration predictor model.   du


VitsConfig

ration_predictor_tail_bound (float, optional, defaults to 5.0) —
Value of the tail bin boundary when computing the unconstrained rational spline in the duration predictor
model.   duration_predictor_kernel_size (int, optional, defaults to 3) —
Kernel size of the 1D convolution layers used in the duration predictor model.   duration_predictor_dropout (float, optional, defaults to 0.5) —
The dropout ratio for the duration predictor model.   duration_predictor_num_flows (int, optional, defaults to 4) —
Number of flow stages used by the duration predictor model.   duration_predictor_filter_channels (int, optional, defaults to 256) —
Number of channels for the convolution layers used in the duration predictor model.   prior_encoder_num_flows (int, optional, defaults to 4) —
Number of flow stages used by the prior encoder flow model.   prior_encoder_num_wavenet_layers (int, optional, defaults to 4) —
Number of WaveNet layers used by the prior encoder flow mo


VitsConfig

del.   posterior_encoder_num_wavenet_layers (int, optional, defaults to 16) —
Number of WaveNet layers used by the posterior encoder model.   wavenet_kernel_size (int, optional, defaults to 5) —
Kernel size of the 1D convolution layers used in the WaveNet model.   wavenet_dilation_rate (int, optional, defaults to 1) —
Dilation rates of the dilated 1D convolutional layers used in the WaveNet model.   wavenet_dropout (float, optional, defaults to 0.0) —
The dropout ratio for the WaveNet layers.   speaking_rate (float, optional, defaults to 1.0) —
Speaking rate. Larger values give faster synthesised speech.   noise_scale (float, optional, defaults to 0.667) —
How random the speech prediction is. Larger values create more variation in the predicted speech.   noise_scale_duration (float, optional, defaults to 0.8) —
How random the duration prediction is. Larger values create more variation in the predicted durations.   sampling_rate (int, optional, defaults


VitsConfig

 to 16000) —
The sampling rate at which the output audio waveform is digitalized expressed in hertz (Hz).    This is the configuration class to store the configuration of a VitsModel. It is used to instantiate a VITS
model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
defaults will yield a similar configuration to that of the VITS
facebook/mms-tts-eng architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import VitsModel, VitsConfig
>>> # Initializing a "facebook/mms-tts-eng" style configuration
>>> configuration = VitsConfig()
>>> # Initializing a model (with random weights) from the "facebook/mms-tts-eng" style configuration
>>> model = VitsModel(configuration)
>>> # Accessing the model configuration
>>> configuration = mode


VitsTokenizer

class transformers.VitsTokenizer  < source > ( vocab_file pad_token = '<pad>' unk_token = '<unk>' language = None add_blank = True normalize = True phonemize = True is_uroman = False **kwargs  )   Parameters   vocab_file (str) —
Path to the vocabulary file.   language (str, optional) —
Language identifier.   add_blank (bool, optional, defaults to True) —
Whether to insert token id 0 in between the other tokens.   normalize (bool, optional, defaults to True) —
Whether to normalize the input text by removing all casing and punctuation.   phonemize (bool, optional, defaults to True) —
Whether to convert the input text into phonemes.   is_uroman (bool, optional, defaults to False) —
Whether the uroman Romanizer needs to be applied to the input text prior to tokenizing.    Construct a VITS tokenizer. Also supports MMS-TTS. This tokenizer inherits from PreTrainedTokenizer which contains most of the main methods. Users should refer to
this superclass for more information regarding th


VitsTokenizer

ose methods.  __call__  < source > ( text: typing.Union[str, typing.List[str], typing.List[typing.List[str]]] = None text_pair: typing.Union[str, typing.List[str], typing.List[typing.List[str]], NoneType] = None text_target: typing.Union[str, typing.List[str], typing.List[typing.List[str]]] = None text_pair_target: typing.Union[str, typing.List[str], typing.List[typing.List[str]], NoneType] = None add_special_tokens: bool = True padding: typing.Union[bool, str, transformers.utils.generic.PaddingStrategy] = False truncation: typing.Union[bool, str, transformers.tokenization_utils_base.TruncationStrategy] = None max_length: typing.Optional[int] = None stride: int = 0 is_split_into_words: bool = False pad_to_multiple_of: typing.Optional[int] = None return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None return_token_type_ids: typing.Optional[bool] = None return_attention_mask: typing.Optional[bool] = None return_overflowing_tokens: bool = False r


VitsTokenizer

eturn_special_tokens_mask: bool = False return_offsets_mapping: bool = False return_length: bool = False verbose: bool = True **kwargs  ) → BatchEncoding  Parameters   text (str, List[str], List[List[str]], optional) —
The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings
(pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set
is_split_into_words=True (to lift the ambiguity with a batch of sequences).   text_pair (str, List[str], List[List[str]], optional) —
The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings
(pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set
is_split_into_words=True (to lift the ambiguity with a batch of sequences).   text_target (str, List[str], List[List[str]], optional) —
The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or


VitsTokenizer

 a
list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized),
you must set is_split_into_words=True (to lift the ambiguity with a batch of sequences).   text_pair_target (str, List[str], List[List[str]], optional) —
The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or a
list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized),
you must set is_split_into_words=True (to lift the ambiguity with a batch of sequences).   add_special_tokens (bool, optional, defaults to True) —
Whether or not to add special tokens when encoding the sequences. This will use the underlying
PretrainedTokenizerBase.build_inputs_with_special_tokens function, which defines which tokens are
automatically added to the input ids. This is usefull if you want to add bos or eos tokens
automatically.   padding (bool, str or PaddingStrategy, optional, defaults to False) —
Activat


VitsTokenizer

es and controls padding. Accepts the following values:
True or 'longest': Pad to the longest sequence in the batch (or no padding if only a single
sequence if provided).
'max_length': Pad to a maximum length specified with the argument max_length or to the maximum
acceptable input length for the model if that argument is not provided.
False or 'do_not_pad' (default): No padding (i.e., can output a batch with sequences of different
lengths).
   truncation (bool, str or TruncationStrategy, optional, defaults to False) —
Activates and controls truncation. Accepts the following values:
True or 'longest_first': Truncate to a maximum length specified with the argument max_length or
to the maximum acceptable input length for the model if that argument is not provided. This will
truncate token by token, removing a token from the longest sequence in the pair if a pair of
sequences (or a batch of pairs) is provided.
'only_first': Truncate to a maximum length specified with the argument


VitsTokenizer

max_length or to the
maximum acceptable input length for the model if that argument is not provided. This will only
truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided.
'only_second': Truncate to a maximum length specified with the argument max_length or to the
maximum acceptable input length for the model if that argument is not provided. This will only
truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided.
False or 'do_not_truncate' (default): No truncation (i.e., can output batch with sequence lengths
greater than the model maximum admissible input size).
   max_length (int, optional) —
Controls the maximum length to use by one of the truncation/padding parameters.
If left unset or set to None, this will use the predefined model maximum length if a maximum length
is required by one of the truncation/padding parameters. If the model has no specific maximum input
length (like XLNet) truncatio


VitsTokenizer

n/padding to a maximum length will be deactivated.   stride (int, optional, defaults to 0) —
If set to a number along with max_length, the overflowing tokens returned when
return_overflowing_tokens=True will contain some tokens from the end of the truncated sequence
returned to provide some overlap between truncated and overflowing sequences. The value of this
argument defines the number of overlapping tokens.   is_split_into_words (bool, optional, defaults to False) —
Whether or not the input is already pre-tokenized (e.g., split into words). If set to True, the
tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace)
which it will tokenize. This is useful for NER or token classification.   pad_to_multiple_of (int, optional) —
If set will pad the sequence to a multiple of the provided value. Requires padding to be activated.
This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability
>= 7.


VitsTokenizer

5 (Volta).   return_tensors (str or TensorType, optional) —
If set, will return tensors instead of list of python integers. Acceptable values are:
'tf': Return TensorFlow tf.constant objects.
'pt': Return PyTorch torch.Tensor objects.
'np': Return Numpy np.ndarray objects.
   return_token_type_ids (bool, optional) —
Whether to return token type IDs. If left to the default, will return the token type IDs according to
the specific tokenizer’s default, defined by the return_outputs attribute.
What are token type IDs?   return_attention_mask (bool, optional) —
Whether to return the attention mask. If left to the default, will return the attention mask according
to the specific tokenizer’s default, defined by the return_outputs attribute.
What are attention masks?   return_overflowing_tokens (bool, optional, defaults to False) —
Whether or not to return overflowing token sequences. If a pair of sequences of input ids (or a batch
of pairs) is provided with truncation_strategy = long


VitsTokenizer

est_first or True, an error is raised instead
of returning overflowing tokens.   return_special_tokens_mask (bool, optional, defaults to False) —
Whether or not to return special tokens mask information.   return_offsets_mapping (bool, optional, defaults to False) —
Whether or not to return (char_start, char_end) for each token.
This is only available on fast tokenizers inheriting from PreTrainedTokenizerFast, if using
Python’s tokenizer, this method will raise NotImplementedError.   return_length  (bool, optional, defaults to False) —
Whether or not to return the lengths of the encoded inputs.   verbose (bool, optional, defaults to True) —
Whether or not to print more information and warnings.
**kwargs — passed to the self.tokenize() method   Returns
BatchEncoding

A BatchEncoding with the following fields:
input_ids — List of token ids to be fed to a model.
What are input IDs?
token_type_ids — List of token type ids to be fed to a model (when return_token_type_ids=True or


VitsTokenizer

if “token_type_ids” is in self.model_input_names).
What are token type IDs?
attention_mask — List of indices specifying which tokens should be attended to by the model (when
return_attention_mask=True or if “attention_mask” is in self.model_input_names).
What are attention masks?
overflowing_tokens — List of overflowing tokens sequences (when a max_length is specified and
return_overflowing_tokens=True).
num_truncated_tokens — Number of tokens truncated (when a max_length is specified and
return_overflowing_tokens=True).
special_tokens_mask — List of 0s and 1s, with 1 specifying added special tokens and 0 specifying
regular sequence tokens (when add_special_tokens=True and return_special_tokens_mask=True).
length — The length of the inputs (when return_length=True)
  Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of
sequences.  save_vocabulary  < source > ( save_directory: str filename_prefix: typing.Optional[str] = None


VitsModel

class transformers.VitsModel  < source > ( config: VitsConfig  )   Parameters   config (VitsConfig) —
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
from_pretrained() method to load the model weights.    The complete VITS model, for text-to-speech synthesis.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_


VitsModel

ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None speaker_id: typing.Optional[int] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None labels: typing.Optional[torch.FloatTensor] = None  ) → transformers.models.vits.modeling_vits.VitsModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mas


VitsModel

k to avoid performing convolution and attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   speaker_id (int, optional) —
Which speaker embedding to use. Only used for multispeaker models.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.FloatTensor of shape (batch_size, config.spectrogram_bins, sequence_length), optional) —
Float values o


VitsModel

f target spectrogram. Timesteps set to -100.0 are ignored (masked) for the loss
computation.   Returns
transformers.models.vits.modeling_vits.VitsModelOutput or tuple(torch.FloatTensor)

A transformers.models.vits.modeling_vits.VitsModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (VitsConfig) and inputs.
waveform (torch.FloatTensor of shape (batch_size, sequence_length)) — The final audio waveform predicted by the model.
sequence_lengths  (torch.FloatTensor of shape (batch_size,)) — The length in samples of each element in the waveform batch.
spectrogram (torch.FloatTensor of shape (batch_size, sequence_length, num_bins)) — The log-mel spectrogram predicted at the output of the flow model. This spectrogr


VitsModel

am is passed to the Hi-Fi
GAN decoder model to obtain the final audio waveform.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attention weights after the attention softmax, used to compute the


VitsModel

weighted average in the self-attention
heads.
  The VitsModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import VitsTokenizer, VitsModel, set_seed
>>> import torch
>>> tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-eng")
>>> model = VitsModel.from_pretrained("facebook/mms-tts-eng")
>>> inputs = tokenizer(text="Hello - my dog is cute", return_tensors="pt")
>>> set_seed(555)  # make deterministic
>>> with torch.no_grad():
...     outputs = model(inputs["input_ids"])
>>> outputs.waveform.shape
torch.Size([1, 4582


Overview

The Vision Transformer (ViT) model was proposed in An Image is Worth 16x16 Words: Transformers for Image Recognition
at Scale by Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk
Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob
Uszkoreit, Neil Houlsby. It’s the first paper that successfully trains a Transformer encoder on ImageNet, attaining
very good results compared to familiar convolutional architectures.
The abstract from the paper is the following:
While the Transformer architecture has become the de-facto standard for natural language processing tasks, its
applications to computer vision remain limited. In vision, attention is either applied in conjunction with
convolutional networks, or used to replace certain components of convolutional networks while keeping their


Overview

 overall
structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to
sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of
data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.),
Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring
substantially fewer computational resources to train.
Tips:
Demo notebooks regarding inference as well as fine-tuning ViT on custom data can be found here. To feed images to the Transformer encoder, each image is split into a sequence of fixed-size non-overlapping patches,
which are then linearly embedded. A [CLS] token is added to serve as representation of an entire image, which can


Overview

be
used for classification. The authors also add absolute position embeddings, and feed the resulting sequence of
vectors to a standard Transformer encoder. As the Vision Transformer expects each image to be of the same size (resolution), one can use
ViTImageProcessor to resize (or rescale) and normalize images for the model. Both the patch resolution and image resolution used during pre-training or fine-tuning are reflected in the name of
each checkpoint. For example, google/vit-base-patch16-224 refers to a base-sized architecture with patch
resolution of 16x16 and fine-tuning resolution of 224x224. All checkpoints can be found on the hub. The available checkpoints are either (1) pre-trained on ImageNet-21k (a collection of
14 million images and 21k classes) only, or (2) also fine-tuned on ImageNet (also referred to as ILSVRC 2012, a collection of 1.3


Overview

 million
images and 1,000 classes). The Vision Transformer was pre-trained using a resolution of 224x224. During fine-tuning, it is often beneficial to
use a higher resolution than pre-training (Touvron et al., 2019), (Kolesnikov
et al., 2020). In order to fine-tune at higher resolution, the authors perform
2D interpolation of the pre-trained position embeddings, according to their location in the original image. The best results are obtained with supervised pre-training, which is not the case in NLP. The authors also performed
an experiment with a self-supervised pre-training objective, namely masked patched prediction (inspired by masked
language modeling). With this approach, the smaller ViT-B/16 model achieves 79.9% accuracy on ImageNet, a significant
improvement of 2% to training from scratch, but still 4% behind supervised pre-training.
ViT archi


Overview

tecture. Taken from the original paper.
Following the original Vision Transformer, some follow-up works have been made:
DeiT (Data-efficient Image Transformers) by Facebook AI. DeiT models are distilled vision transformers.
The authors of DeiT also released more efficiently trained ViT models, which you can directly plug into ViTModel or
ViTForImageClassification. There are 4 variants available (in 3 different sizes): facebook/deit-tiny-patch16-224,
facebook/deit-small-patch16-224, facebook/deit-base-patch16-224 and facebook/deit-base-patch16-384. Note that one should
use DeiTImageProcessor in order to prepare images for the model. BEiT (BERT pre-training of Image Transformers) by Microsoft Research. BEiT models outperform supervised pre-trained
vision transformers using a self-supervised method inspired by BERT (masked image modeling) and based on a V


Overview

Q-VAE. DINO (a method for self-supervised training of Vision Transformers) by Facebook AI. Vision Transformers trained using
the DINO method show very interesting properties not seen with convolutional models. They are capable of segmenting
objects, without having ever been trained to do so. DINO checkpoints can be found on the hub. MAE (Masked Autoencoders) by Facebook AI. By pre-training Vision Transformers to reconstruct pixel values for a high portion
(75%) of masked patches (using an asymmetric encoder-decoder architecture), the authors show that this simple method outperforms
supervised pre-training after fine-tuning.
This model was contributed by nielsr. The original code (written in JAX) can be
found here.
Note that we converted the weights from Ross Wightman’s timm library, who already converted the weights from JAX to PyTorch. Credits
go to h


Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ViT.
 Image Classification
ViTForImageClassification is supported by this example script and notebook. A blog on fine-tuning ViTForImageClassification on a custom dataset can be found here. More demo notebooks to fine-tune ViTForImageClassification can be found here. Image classification task guide
Besides that:
ViTForMaskedImageModeling is supported by this example script.
If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we’ll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ViT. If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we’ll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.
ViTForImageClassification is supported by:
 Image Classification
A blog post on how to Fine-Tune ViT for Image Classification with Hugging Face Transformers A blog post on Image Classification with Hugging Face Transformers and Keras A notebook on Fine-tuning for Image Classification


Resources

with Hugging Face Transformers A notebook on how to Fine-tune the Vision Transformer on CIFAR-10 with the Hugging Face Trainer A notebook on how to Fine-tune the Vision Transformer on CIFAR-10 with PyTorch Lightning
⚗ Optimization
A blog post on how to Accelerate Vision Transformer (ViT) with Quantization using Optimum
⚡ Inference
A notebook on Quick demo: Vision Transformer (ViT) by Google Brain
🚀 Deploy
A blog post on Deploying Tensorflow Vision Models in Hugging Face with TF Serving A blog post on Deploying Hugging Face ViT on Vertex AI A blog post on Deploying Hugging Face ViT on Kubernetes with TF Serving


ViTConfig

class transformers.ViTConfig  < source > ( hidden_size = 768 num_hidden_layers = 12 num_attention_heads = 12 intermediate_size = 3072 hidden_act = 'gelu' hidden_dropout_prob = 0.0 attention_probs_dropout_prob = 0.0 initializer_range = 0.02 layer_norm_eps = 1e-12 image_size = 224 patch_size = 16 num_channels = 3 qkv_bias = True encoder_stride = 16 **kwargs  )   Parameters   hidden_size (int, optional, defaults to 768) —
Dimensionality of the encoder layers and the pooler layer.   num_hidden_layers (int, optional, defaults to 12) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 12) —
Number of attention heads for each attention layer in the Transformer encoder.   intermediate_size (int, optional, defaults to 3072) —
Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.   hidden_act (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in


ViTConfig

the encoder and pooler. If string, "gelu",
"relu", "selu" and "gelu_new" are supported.   hidden_dropout_prob (float, optional, defaults to 0.0) —
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.   attention_probs_dropout_prob (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   layer_norm_eps (float, optional, defaults to 1e-12) —
The epsilon used by the layer normalization layers.   image_size (int, optional, defaults to 224) —
The size (resolution) of each image.   patch_size (int, optional, defaults to 16) —
The size (resolution) of each patch.   num_channels (int, optional, defaults to 3) —
The number of input channels.   qkv_bias (bool, optional, defaults to True) —
Whether to add a bias to the queries, keys and values.   encoder_stride (i


ViTConfig

nt, optional, defaults to 16) —
Factor to increase the spatial resolution by in the decoder head for masked image modeling.    This is the configuration class to store the configuration of a ViTModel. It is used to instantiate an ViT
model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
defaults will yield a similar configuration to that of the ViT
google/vit-base-patch16-224 architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import ViTConfig, ViTModel
>>> # Initializing a ViT vit-base-patch16-224 style configuration
>>> configuration = ViTConfig()
>>> # Initializing a model (with random weights) from the vit-base-patch16-224 style configuration
>>> model = ViTModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config


ViTFeatureExtractor

class transformers.ViTFeatureExtractor  < source > ( *args **kwargs  )     __call__  < source > ( images **kwargs  )    Preprocess an image or a batch of images.


ViTImageProcessor

class transformers.ViTImageProcessor  < source > ( do_resize: bool = True size: typing.Union[typing.Dict[str, int], NoneType] = None resample: Resampling = <Resampling.BILINEAR: 2> do_rescale: bool = True rescale_factor: typing.Union[int, float] = 0.00392156862745098 do_normalize: bool = True image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float], NoneType] = None **kwargs  )   Parameters   do_resize (bool, optional, defaults to True) —
Whether to resize the image’s (height, width) dimensions to the specified (size["height"], size["width"]). Can be overridden by the do_resize parameter in the preprocess method.   size (dict, optional, defaults to {"height" -- 224, "width": 224}):
Size of the output image after resizing. Can be overridden by the size parameter in the preprocess
method.   resample (PILImageResampling, optional, defaults to Resampling.BILINEAR) —
Resampling


ViTImageProcessor

 filter to use if resizing the image. Can be overridden by the resample parameter in the
preprocess method.   do_rescale (bool, optional, defaults to True) —
Whether to rescale the image by the specified scale rescale_factor. Can be overridden by the do_rescale
parameter in the preprocess method.   rescale_factor (int or float, optional, defaults to 1/255) —
Scale factor to use if rescaling the image. Can be overridden by the rescale_factor parameter in the
preprocess method.   do_normalize (bool, optional, defaults to True) —
Whether to normalize the image. Can be overridden by the do_normalize parameter in the preprocess
method.   image_mean (float or List[float], optional, defaults to IMAGENET_STANDARD_MEAN) —
Mean to use if normalizing the image. This is a float or list of floats the length of the number of
channels in the image. Can be overridden by the image_mean parameter in the preprocess method.   image_std (float or List[fl


ViTImageProcessor

oat], optional, defaults to IMAGENET_STANDARD_STD) —
Standard deviation to use if normalizing the image. This is a float or list of floats the length of the
number of channels in the image. Can be overridden by the image_std parameter in the preprocess method.    Constructs a ViT image processor.  preprocess  < source > ( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]] do_resize: typing.Optional[bool] = None size: typing.Dict[str, int] = None resample: Resampling = None do_rescale: typing.Optional[bool] = None rescale_factor: typing.Optional[float] = None do_normalize: typing.Optional[bool] = None image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float], NoneType] = None return_tensors: typing.Union[str, transformer


ViTImageProcessor

s.utils.generic.TensorType, NoneType] = None data_format: typing.Union[str, transformers.image_utils.ChannelDimension] = <ChannelDimension.FIRST: 'channels_first'> input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None **kwargs  )   Parameters   images (ImageInput) —
Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If
passing in images with pixel values between 0 and 1, set do_rescale=False.   do_resize (bool, optional, defaults to self.do_resize) —
Whether to resize the image.   size (Dict[str, int], optional, defaults to self.size) —
Dictionary in the format {"height": h, "width": w} specifying the size of the output image after
resizing.   resample (PILImageResampling filter, optional, defaults to self.resample) —
PILImageResampling filter to use if resizing the image e.g. PILImageResampling.BILINEAR. Only has
an effect if do_resize is set t


ViTImageProcessor

o True.   do_rescale (bool, optional, defaults to self.do_rescale) —
Whether to rescale the image values between [0 - 1].   rescale_factor (float, optional, defaults to self.rescale_factor) —
Rescale factor to rescale the image by if do_rescale is set to True.   do_normalize (bool, optional, defaults to self.do_normalize) —
Whether to normalize the image.   image_mean (float or List[float], optional, defaults to self.image_mean) —
Image mean to use if do_normalize is set to True.   image_std (float or List[float], optional, defaults to self.image_std) —
Image standard deviation to use if do_normalize is set to True.   return_tensors (str or TensorType, optional) —
The type of tensors to return. Can be one of:
Unset: Return a list of np.ndarray.
TensorType.TENSORFLOW or 'tf': Return a batch of type tf.Tensor.
TensorType.PYTORCH or 'pt': Return a batch of type torch.Tensor.
TensorType.NUMPY or 'np': Return a batch of type np.ndarray.
T