SpeechT5Model

class transformers.SpeechT5Model  < source > ( config: SpeechT5Config encoder: typing.Optional[torch.nn.modules.module.Module] = None decoder: typing.Optional[torch.nn.modules.module.Module] = None  )   Parameters   config (SpeechT5Config) —
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
from_pretrained() method to load the model weights.   encoder (SpeechT5EncoderWithSpeechPrenet or SpeechT5EncoderWithTextPrenet or None) —
The Transformer encoder module that applies the appropiate speech or text encoder prenet. If None,
SpeechT5EncoderWithoutPrenet will be used and the input_values are assumed to be hidden states.   decoder (SpeechT5DecoderWithSpeechPrenet or SpeechT5DecoderWithTextPrenet or None) —
The Transformer decoder module that applies the appropiate speech or text decoder prenet. If None,
Sp


SpeechT5Model

eechT5DecoderWithoutPrenet will be used and the decoder_input_values are assumed to be hidden
states.    The bare SpeechT5 Encoder-Decoder Model outputting raw hidden-states without any specific pre- or post-nets.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_values: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.LongTensor] = None decoder_input_values: typing.Optional[torch.Tensor] = None decoder_attention_mask: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None decoder_head_mask: ty


SpeechT5Model

ping.Optional[torch.FloatTensor] = None cross_attn_head_mask: typing.Optional[torch.Tensor] = None encoder_outputs: typing.Optional[typing.Tuple[typing.Tuple[torch.FloatTensor]]] = None past_key_values: typing.Optional[typing.Tuple[typing.Tuple[torch.FloatTensor]]] = None use_cache: typing.Optional[bool] = None speaker_embeddings: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.Seq2SeqModelOutput or tuple(torch.FloatTensor)  Parameters   attention_mask (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing convolution and attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?
attention_mask should only be passed if the correspondi


SpeechT5Model

ng processor has config.return_attention_mask == True. For all models whose processor has config.return_attention_mask == False, attention_mask should
not be passed to avoid degraded performance when doing batched inference. For such models
input_values should simply be padded with 0 and passed without attention_mask. Be aware that these
models also yield slightly different results depending on whether input_values is padded or not.
   decoder_attention_mask (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Default behavior: generate a tensor that ignores pad tokens in decoder_input_values. Causal mask will
also be used by default.
If you want to change padding behavior, you should read SpeechT5Decoder._prepare_decoder_attention_mask
and modify to your needs. See diagram 1 in the paper for more
information on the default strategy.   head_mask (torch.FloatTensor of shape (encoder_layers, encoder_attention_heads)


SpeechT5Model

, optional) —
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   decoder_head_mask (torch.FloatTensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   cross_attn_head_mask (torch.Tensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the cross-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   encoder_outputs (tuple(tuple(torch.FloatTensor), optional) —
Tuple consists of (last_hidden_state, optional: hidden_states, optional: attentions)
last_hidden_state of shape (batch_size, sequence_length, hidden_size), opti


SpeechT5Model

onal) is a sequence of
hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.   past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) —
Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_values (those
that don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_val


SpeechT5Model

ues of shape (batch_size, sequence_length). decoder_inputs_embeds (torch.FloatTensor
of shape (batch_size, target_sequence_length, hidden_size), optional): Optionally, instead of passing
decoder_input_values you can choose to directly pass an embedded representation. If past_key_values is
used, optionally only the last decoder_inputs_embeds have to be input (see past_key_values). This is
useful if you want more control over how to convert decoder_input_values indices into associated vectors
than the model’s internal embedding lookup matrix.   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states


SpeechT5Model

 of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   input_values (torch.Tensor of shape (batch_size, sequence_length)) —
Depending on which encoder is being used, the input_values are either: float values of the input raw
speech waveform, or indices of input sequence tokens in the vocabulary, or hidden states.   decoder_input_values (torch.Tensor of shape (batch_size, target_sequence_length), optional) —
Depending on which decoder is being used, the decoder_input_values are either: float values of log-mel
filterbank features extracted from the raw speech waveform, or indices of decoder input sequence tokens in
the vocabulary, or hidden states.   speaker_embeddings (torch.FloatTensor of shape (batch_size, config.speaker_embedding_dim), optional) —
Tensor containing the speaker embeddings.   Returns
transformers.modeli


SpeechT5Model

ng_outputs.Seq2SeqModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.Seq2SeqModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (SpeechT5Config) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the decoder of the model.
If past_key_values is used only the last hidden-state of the sequences of shape (batch_size, 1, hidden_size) is output.
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads


SpeechT5Model

, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the decoder at the output of each layer plus the optional initial embedding outputs.
decoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
A


SpeechT5Model

ttentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embed


SpeechT5Model

dings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the optional initial embedding outputs.
encoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
  The SpeechT5Model forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores th


class transformers.SpeechT5ForSpeechToText  < source > ( config: SpeechT5Config  )   Parameters   config (SpeechT5Config) —
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
from_pretrained() method to load the model weights.    SpeechT5 Model with a speech encoder and a text decoder.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_values: typing.Optional[torch.FloatTensor] = None attention_mask: typing.Optional[torch.LongTensor] =


None decoder_input_ids: typing.Optional[torch.LongTensor] = None decoder_attention_mask: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None decoder_head_mask: typing.Optional[torch.FloatTensor] = None cross_attn_head_mask: typing.Optional[torch.Tensor] = None encoder_outputs: typing.Optional[typing.Tuple[typing.Tuple[torch.FloatTensor]]] = None past_key_values: typing.Optional[typing.Tuple[typing.Tuple[torch.FloatTensor]]] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None labels: typing.Optional[torch.LongTensor] = None  ) → transformers.modeling_outputs.Seq2SeqLMOutput or tuple(torch.FloatTensor)  Parameters   attention_mask (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing convolution and attention on padding token indices.


Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?
attention_mask should only be passed if the corresponding processor has config.return_attention_mask == True. For all models whose processor has config.return_attention_mask == False, attention_mask should
not be passed to avoid degraded performance when doing batched inference. For such models
input_values should simply be padded with 0 and passed without attention_mask. Be aware that these
models also yield slightly different results depending on whether input_values is padded or not.
   decoder_attention_mask (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Default behavior: generate a tensor that ignores pad tokens in decoder_input_values. Causal mask will
also be used by default.
If you want to change padding behavior, you should read SpeechT5Decoder._prepare_decoder_attention_mask
and modify to


your needs. See diagram 1 in the paper for more
information on the default strategy.   head_mask (torch.FloatTensor of shape (encoder_layers, encoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   decoder_head_mask (torch.FloatTensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   cross_attn_head_mask (torch.Tensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the cross-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   encoder_outputs (tuple(tuple(torch.FloatTensor), optional) —
Tuple


 consists of (last_hidden_state, optional: hidden_states, optional: attentions)
last_hidden_state of shape (batch_size, sequence_length, hidden_size), optional) is a sequence of
hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.   past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) —
Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
If past_key_values are used, the user can optionally input only the last decode


r_input_values (those
that don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_values of shape (batch_size, sequence_length). decoder_inputs_embeds (torch.FloatTensor
of shape (batch_size, target_sequence_length, hidden_size), optional): Optionally, instead of passing
decoder_input_values you can choose to directly pass an embedded representation. If past_key_values is
used, optionally only the last decoder_inputs_embeds have to be input (see past_key_values). This is
useful if you want more control over how to convert decoder_input_values indices into associated vectors
than the model’s internal embedding lookup matrix.   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See at


tentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   input_values (torch.FloatTensor of shape (batch_size, sequence_length)) —
Float values of input raw speech waveform. Values can be obtained by loading a .flac or .wav audio file
into an array of type List[float] or a numpy.ndarray, e.g. via the soundfile library (pip install
soundfile). To prepare the array into input_values, the SpeechT5Processor should be used for padding
and conversion into a tensor of type torch.FloatTensor. See SpeechT5Processor.call() for details.   decoder_input_ids (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Indices of decoder input sequence tokens in the vocabulary.
Indices can be obtained


 using SpeechT5Tokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are decoder input IDs?
SpeechT5 uses the eos_token_id as the starting token for decoder_input_ids generation. If
past_key_values is used, optionally only the last decoder_input_ids have to be input (see
past_key_values).   labels (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Labels for computing the language modeling loss. Indices should either be in [0, ..., config.vocab_size]
or -100 (see input_ids docstring). Tokens with indices set to -100 are ignored (masked), the loss is
only computed for the tokens with labels in [0, ..., config.vocab_size].
Label indices can be obtained using SpeechT5Tokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.   Returns
transformers.modeling_outputs.Seq2SeqLMOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.Seq2SeqLMOutput or a tu


ple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (SpeechT5Config) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Language modeling loss.
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and


in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the decoder at the output of each layer plus the initial embedding outputs.
decoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
cross_attentions (tuple(t


orch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the encoder at


the output of each layer plus the initial embedding outputs.
encoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
  The SpeechT5ForSpeechToText forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import SpeechT5Processor, SpeechT5ForSpeechToText
>>> from datasets import load_dataset
>>> dataset = load_dataset(
...     "hf-in


ternal-testing/librispeech_asr_demo", "clean", split="validation"
... )  # doctest: +IGNORE_RESULT
>>> dataset = dataset.sort("id")
>>> sampling_rate = dataset.features["audio"].sampling_rate
>>> processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_asr")
>>> model = SpeechT5ForSpeechToText.from_pretrained("microsoft/speecht5_asr")
>>> # audio file is decoded on the fly
>>> inputs = processor(audio=dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
>>> predicted_ids = model.generate(**inputs, max_length=100)
>>> # transcribe speech
>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
>>> transcription[0]
'mister quilter is the apostle of the middle classes and we are glad to welcome his gospel'    Copied >>> inputs["labels"] = processor(text_target=dataset[0]["text"], return_tensors="pt").input_ids
>>> # compute loss
>>> loss = model(**inputs).loss
>>> round(loss.item(), 2)
19.


class transformers.SpeechT5ForTextToSpeech  < source > ( config: SpeechT5Config  )   Parameters   config (SpeechT5Config) —
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
from_pretrained() method to load the model weights.    SpeechT5 Model with a text encoder and a speech decoder.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Option


al[torch.LongTensor] = None decoder_input_values: typing.Optional[torch.FloatTensor] = None decoder_attention_mask: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None decoder_head_mask: typing.Optional[torch.FloatTensor] = None cross_attn_head_mask: typing.Optional[torch.Tensor] = None encoder_outputs: typing.Optional[typing.Tuple[typing.Tuple[torch.FloatTensor]]] = None past_key_values: typing.Optional[typing.Tuple[typing.Tuple[torch.FloatTensor]]] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None speaker_embeddings: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.FloatTensor] = None stop_labels: typing.Optional[torch.Tensor] = None  ) → transformers.modeling_outputs.Seq2SeqSpectrogramOutput or tuple(torch.FloatTensor)


 Parameters   attention_mask (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing convolution and attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?
attention_mask should only be passed if the corresponding processor has config.return_attention_mask == True. For all models whose processor has config.return_attention_mask == False, attention_mask should
not be passed to avoid degraded performance when doing batched inference. For such models
input_values should simply be padded with 0 and passed without attention_mask. Be aware that these
models also yield slightly different results depending on whether input_values is padded or not.
   decoder_attention_mask (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Default behavior: generate a tensor that i


gnores pad tokens in decoder_input_values. Causal mask will
also be used by default.
If you want to change padding behavior, you should read SpeechT5Decoder._prepare_decoder_attention_mask
and modify to your needs. See diagram 1 in the paper for more
information on the default strategy.   head_mask (torch.FloatTensor of shape (encoder_layers, encoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   decoder_head_mask (torch.FloatTensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   cross_attn_head_mask (torch.Tensor of shape (decoder_layers, decoder_attention_heads), optional) —
Ma


sk to nullify selected heads of the cross-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   encoder_outputs (tuple(tuple(torch.FloatTensor), optional) —
Tuple consists of (last_hidden_state, optional: hidden_states, optional: attentions)
last_hidden_state of shape (batch_size, sequence_length, hidden_size), optional) is a sequence of
hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.   past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) —
Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-com


puted hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_values (those
that don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_values of shape (batch_size, sequence_length). decoder_inputs_embeds (torch.FloatTensor
of shape (batch_size, target_sequence_length, hidden_size), optional): Optionally, instead of passing
decoder_input_values you can choose to directly pass an embedded representation. If past_key_values is
used, optionally only the last decoder_inputs_embeds have to be input (see past_key_values). This is
useful if you want more control over how to convert decoder_input_values indices into associated vectors
than the model’s internal embedding lo


okup matrix.   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. The batch_size should be 1 currently.
Indices can be obtained using SpeechT5Tokenizer. See encode() and
call() for details.
What are input IDs?   decoder_input_values (torch.FloatTensor of shape (batch_size, sequence_lengt


h, config.num_mel_bins)) —
Float values of input mel spectrogram.
SpeechT5 uses an all-zero spectrum as the starting token for decoder_input_values generation. If
past_key_values is used, optionally only the last decoder_input_values have to be input (see
past_key_values).   speaker_embeddings (torch.FloatTensor of shape (batch_size, config.speaker_embedding_dim), optional) —
Tensor containing the speaker embeddings.   labels (torch.FloatTensor of shape (batch_size, sequence_length, config.num_mel_bins), optional) —
Float values of target mel spectrogram. Timesteps set to -100.0 are ignored (masked) for the loss
computation. Spectrograms can be obtained using SpeechT5Processor. See SpeechT5Processor.call()
for details.   Returns
transformers.modeling_outputs.Seq2SeqSpectrogramOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.Seq2SeqSpectrogramOutput or a tuple of
torch.FloatTensor (if return_dict=Fals


e is passed or when config.return_dict=False) comprising various
elements depending on the configuration (SpeechT5Config) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Spectrogram generation loss.
spectrogram (torch.FloatTensor of shape (batch_size, sequence_length, num_bins)) — The predicted spectrogram.
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up s


equential decoding.
decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the decoder at the output of each layer plus the initial embedding outputs.
decoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=T


rue is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the i


nitial embedding outputs.
encoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
  The SpeechT5ForTextToSpeech forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import SpeechT5Processor, SpeechT5ForTextToSpeech, SpeechT5HifiGan, set_seed
>>> import torch
>>> processor = SpeechT5Processor.from_pretr


ained("microsoft/speecht5_tts")
>>> model = SpeechT5ForTextToSpeech.from_pretrained("microsoft/speecht5_tts")
>>> vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan")
>>> inputs = processor(text="Hello, my dog is cute", return_tensors="pt")
>>> speaker_embeddings = torch.zeros((1, 512))  # or load xvectors from a file
>>> set_seed(555)  # make deterministic
>>> # generate speech
>>> speech = model.generate(inputs["input_ids"], speaker_embeddings, vocoder=vocoder)
>>> speech.shape
torch.Size([15872])  generate  < source > ( input_ids: LongTensor speaker_embeddings: typing.Optional[torch.FloatTensor] = None threshold: float = 0.5 minlenratio: float = 0.0 maxlenratio: float = 20.0 vocoder: typing.Optional[torch.nn.modules.module.Module] = None output_cross_attentions: bool = False **kwargs  ) → tuple(torch.FloatTensor) comprising various elements depending on the inputs  Parameters   input_ids (torch.LongT


ensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. The batch_size should be 1 currently.
Indices can be obtained using SpeechT5Tokenizer. See encode() and
call() for details.
What are input IDs?   speaker_embeddings (torch.FloatTensor of shape (batch_size, config.speaker_embedding_dim), optional) —
Tensor containing the speaker embeddings.   threshold (float, optional, defaults to 0.5) —
The generated sequence ends when the predicted stop token probability exceeds this value.   minlenratio (float, optional, defaults to 0.0) —
Used to calculate the minimum required length for the output sequence.   maxlenratio (float, optional, defaults to 20.0) —
Used to calculate the maximum allowed length for the output sequence.   vocoder (nn.Module, optional) —
The vocoder that converts the mel spectrogram into a speech waveform. If None, the output is the mel
spectrogram.   output_cros


s_attentions (bool, optional, defaults to False) —
Whether or not to return the attentions tensors of the decoder’s cross-attention layers.   Returns
tuple(torch.FloatTensor) comprising various elements depending on the inputs

spectrogram (optional, returned when no vocoder is provided) torch.FloatTensor of shape
(output_sequence_length, config.num_mel_bins) — The predicted log-mel spectrogram.
waveform (optional, returned when a vocoder is provided) torch.FloatTensor of shape
(num_frames,) — The predicted speech waveform.
cross_attentions (optional, returned when output_cross_attentions is True) torch.FloatTensor
of shape (config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length) — The outputs of the decoder’s cross-attention layers.
  Converts a sequence of input tokens into a sequence of mel spectrograms, which are subsequently turned into a
speech waveform using a vocode


class transformers.SpeechT5ForSpeechToSpeech  < source > ( config: SpeechT5Config  )   Parameters   config (SpeechT5Config) —
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
from_pretrained() method to load the model weights.    SpeechT5 Model with a speech encoder and a speech decoder.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_values: typing.Optional[torch.FloatTensor] = None attention_mask: typing.Optional[torch.LongTensor] = None decoder_input_v


alues: typing.Optional[torch.FloatTensor] = None decoder_attention_mask: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None decoder_head_mask: typing.Optional[torch.FloatTensor] = None cross_attn_head_mask: typing.Optional[torch.Tensor] = None encoder_outputs: typing.Optional[typing.Tuple[typing.Tuple[torch.FloatTensor]]] = None past_key_values: typing.Optional[typing.Tuple[typing.Tuple[torch.FloatTensor]]] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None speaker_embeddings: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.FloatTensor] = None stop_labels: typing.Optional[torch.Tensor] = None  ) → transformers.modeling_outputs.Seq2SeqSpectrogramOutput or tuple(torch.FloatTensor)  Parameters   attention_mask (torch.LongTensor of shape (batch_size, sequence_length), optiona


l) —
Mask to avoid performing convolution and attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?
attention_mask should only be passed if the corresponding processor has config.return_attention_mask == True. For all models whose processor has config.return_attention_mask == False, attention_mask should
not be passed to avoid degraded performance when doing batched inference. For such models
input_values should simply be padded with 0 and passed without attention_mask. Be aware that these
models also yield slightly different results depending on whether input_values is padded or not.
   decoder_attention_mask (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Default behavior: generate a tensor that ignores pad tokens in decoder_input_values. Causal mask will
also be used by default.
If you want to change padding behavior, you should read Spee


chT5Decoder._prepare_decoder_attention_mask
and modify to your needs. See diagram 1 in the paper for more
information on the default strategy.   head_mask (torch.FloatTensor of shape (encoder_layers, encoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   decoder_head_mask (torch.FloatTensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   cross_attn_head_mask (torch.Tensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the cross-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   encoder_outputs (tuple(tuple(tor


ch.FloatTensor), optional) —
Tuple consists of (last_hidden_state, optional: hidden_states, optional: attentions)
last_hidden_state of shape (batch_size, sequence_length, hidden_size), optional) is a sequence of
hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.   past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) —
Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
If past_key_values are used, the user can optionally input only the l


ast decoder_input_values (those
that don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_values of shape (batch_size, sequence_length). decoder_inputs_embeds (torch.FloatTensor
of shape (batch_size, target_sequence_length, hidden_size), optional): Optionally, instead of passing
decoder_input_values you can choose to directly pass an embedded representation. If past_key_values is
used, optionally only the last decoder_inputs_embeds have to be input (see past_key_values). This is
useful if you want more control over how to convert decoder_input_values indices into associated vectors
than the model’s internal embedding lookup matrix.   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under


 returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   input_values (torch.FloatTensor of shape (batch_size, sequence_length)) —
Float values of input raw speech waveform. Values can be obtained by loading a .flac or .wav audio file
into an array of type List[float] or a numpy.ndarray, e.g. via the soundfile library (pip install
soundfile). To prepare the array into input_values, the SpeechT5Processor should be used for padding
and conversion into a tensor of type torch.FloatTensor. See SpeechT5Processor.call() for details.   decoder_input_values (torch.FloatTensor of shape (batch_size, sequence_length, config.num_mel_bins)) —
Float values of input mel spectrogram.
SpeechT5 uses an all-zero spectrum as the starting token for decoder_input


_values generation. If
past_key_values is used, optionally only the last decoder_input_values have to be input (see
past_key_values).   speaker_embeddings (torch.FloatTensor of shape (batch_size, config.speaker_embedding_dim), optional) —
Tensor containing the speaker embeddings.   labels (torch.FloatTensor of shape (batch_size, sequence_length, config.num_mel_bins), optional) —
Float values of target mel spectrogram. Spectrograms can be obtained using SpeechT5Processor. See
SpeechT5Processor.call() for details.   Returns
transformers.modeling_outputs.Seq2SeqSpectrogramOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.Seq2SeqSpectrogramOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (SpeechT5Config) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Spectrogram generation loss.
spectrogram (t


orch.FloatTensor of shape (batch_size, sequence_length, num_bins)) — The predicted spectrogram.
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) o


f shape (batch_size, sequence_length, hidden_size).
Hidden-states of the decoder at the output of each layer plus the initial embedding outputs.
decoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
encoder_last_hidden_state (torch.Float


Tensor of shape (batch_size, sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the initial embedding outputs.
encoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.


  The SpeechT5ForSpeechToSpeech forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import SpeechT5Processor, SpeechT5ForSpeechToSpeech, SpeechT5HifiGan, set_seed
>>> from datasets import load_dataset
>>> import torch
>>> dataset = load_dataset(
...     "hf-internal-testing/librispeech_asr_demo", "clean", split="validation"
... )  # doctest: +IGNORE_RESULT
>>> dataset = dataset.sort("id")
>>> sampling_rate = dataset.features["audio"].sampling_rate
>>> processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_vc")
>>> model = SpeechT5ForSpeechToSpeech.from_pretrained("microsoft/speecht5_vc")
>>> vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan")
>>>


# audio file is decoded on the fly
>>> inputs = processor(audio=dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
>>> speaker_embeddings = torch.zeros((1, 512))  # or load xvectors from a file
>>> set_seed(555)  # make deterministic
>>> # generate speech
>>> speech = model.generate_speech(inputs["input_values"], speaker_embeddings, vocoder=vocoder)
>>> speech.shape
torch.Size([77824])  generate_speech  < source > ( input_values: FloatTensor speaker_embeddings: typing.Optional[torch.FloatTensor] = None threshold: float = 0.5 minlenratio: float = 0.0 maxlenratio: float = 20.0 vocoder: typing.Optional[torch.nn.modules.module.Module] = None output_cross_attentions: bool = False  ) → tuple(torch.FloatTensor) comprising various elements depending on the inputs  Parameters   input_values (torch.FloatTensor of shape (batch_size, sequence_length)) —
Float values of input raw speech waveform. The batch_size should be 1 currently.
Values can be obtained


 by loading a .flac or .wav audio file into an array of type List[float] or
a numpy.ndarray, e.g. via the soundfile library (pip install soundfile). To prepare the array
into input_values, the SpeechT5Processor should be used for padding and conversion into a tensor
of type torch.FloatTensor. See SpeechT5Processor.call() for details.   speaker_embeddings (torch.FloatTensor of shape (batch_size, config.speaker_embedding_dim), optional) —
Tensor containing the speaker embeddings.   threshold (float, optional, defaults to 0.5) —
The generated sequence ends when the predicted stop token probability exceeds this value.   minlenratio (float, optional, defaults to 0.0) —
Used to calculate the minimum required length for the output sequence.   maxlenratio (float, optional, defaults to 20.0) —
Used to calculate the maximum allowed length for the output sequence.   vocoder (nn.Module, optional, defaults to None) —
The vocoder that converts the mel spectrogram into a speech waveform


. If None, the output is the mel
spectrogram.   output_cross_attentions (bool, optional, defaults to False) —
Whether or not to return the attentions tensors of the decoder’s cross-attention layers.   Returns
tuple(torch.FloatTensor) comprising various elements depending on the inputs

spectrogram (optional, returned when no vocoder is provided) torch.FloatTensor of shape
(output_sequence_length, config.num_mel_bins) — The predicted log-mel spectrogram.
waveform (optional, returned when a vocoder is provided) torch.FloatTensor of shape
(num_frames,) — The predicted speech waveform.
cross_attentions (optional, returned when output_cross_attentions is True) torch.FloatTensor
of shape (config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length) — The outputs of the decoder’s cross-attention layers.
  Converts a raw speech waveform into a sequence of mel spectrograms, which are subsequently turned back into a
speech waveform using


SpeechT5HifiGan

class transformers.SpeechT5HifiGan  < source > ( config: SpeechT5HifiGanConfig  )   Parameters   config (SpeechT5HifiGanConfig) —
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
from_pretrained() method to load the model weights.    HiFi-GAN vocoder.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related


SpeechT5HifiGan

to general usage
and behavior.  forward  < source > ( spectrogram: FloatTensor  ) → torch.FloatTensor  Parameters   spectrogram (torch.FloatTensor) —
Tensor containing the log-mel spectrograms. Can be batched and of shape (batch_size, sequence_length, config.model_in_dim), or un-batched and of shape (sequence_length, config.model_in_dim).   Returns
torch.FloatTensor

Tensor containing the speech waveform. If the input spectrogram is batched, will be of
shape (batch_size, num_frames,). If un-batched, will be of shape (num_frames,).
  Converts a log-mel spectrogram into a speech waveform. Passing a batch of log-mel spectrograms returns a batch
of speech waveforms. Passing a single, un-batched log-mel spectrogram returns a single, un-batched speech
waveform


Overview

The Autoformer model was proposed in Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting by Haixu Wu, Jiehui Xu, Jianmin Wang, Mingsheng Long.
This model augments the Transformer as a deep decomposition architecture, which can progressively decompose the trend and seasonal components during the forecasting process.
The abstract from the paper is the following:
Extending the forecasting time is a critical demand for real applications, such as extreme weather early warning and long-term energy consumption planning. This paper studies the long-term forecasting problem of time series. Prior Transformer-based models adopt various self-attention mechanisms to discover the long-range dependencies. However, intricate temporal patterns of the long-term future prohibit the model from finding reliable dependencies. Also, Transformers have to adopt the sparse versions of point-wise self-attentions for long series efficiency, resulting i


Overview

n the information utilization bottleneck. Going beyond Transformers, we design Autoformer as a novel decomposition architecture with an Auto-Correlation mechanism. We break with the pre-processing convention of series decomposition and renovate it as a basic inner block of deep models. This design empowers Autoformer with progressive decomposition capacities for complex time series. Further, inspired by the stochastic process theory, we design the Auto-Correlation mechanism based on the series periodicity, which conducts the dependencies discovery and representation aggregation at the sub-series level. Auto-Correlation outperforms self-attention in both efficiency and accuracy. In long-term forecasting, Autoformer yields state-of-the-art accuracy, with a 38% relative improvement on six benchmarks, covering five practical applications: energy, traffic, economics, weather and disease.
This model was contributed by elisim and kashif.
The original code can be found here


Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started. If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we’ll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.
Check out the Autoformer blog-post in HuggingFace blog: Yes, Transformers are Effective for Time Series Forecasting (+ Autoformer)


AutoformerConfig

class transformers.AutoformerConfig  < source > ( prediction_length: typing.Optional[int] = None context_length: typing.Optional[int] = None distribution_output: str = 'student_t' loss: str = 'nll' input_size: int = 1 lags_sequence: typing.List[int] = [1, 2, 3, 4, 5, 6, 7] scaling: bool = True num_time_features: int = 0 num_dynamic_real_features: int = 0 num_static_categorical_features: int = 0 num_static_real_features: int = 0 cardinality: typing.Optional[typing.List[int]] = None embedding_dimension: typing.Optional[typing.List[int]] = None d_model: int = 64 encoder_attention_heads: int = 2 decoder_attention_heads: int = 2 encoder_layers: int = 2 decoder_layers: int = 2 encoder_ffn_dim: int = 32 decoder_ffn_dim: int = 32 activation_function: str = 'gelu' dropout: float = 0.1 encoder_layerdrop: float = 0.1 decoder_layerdrop: float = 0.1 attention_dropout: float = 0.1 activation_dropout: float = 0.1 num_parallel_samples: int = 100 init_std: float = 0.0


AutoformerConfig

2 use_cache: bool = True is_encoder_decoder = True label_length: int = 10 moving_average: int = 25 autocorrelation_factor: int = 3 **kwargs  )   Parameters   prediction_length (int) —
The prediction length for the decoder. In other words, the prediction horizon of the model.   context_length (int, optional, defaults to prediction_length) —
The context length for the encoder. If unset, the context length will be the same as the
prediction_length.   distribution_output (string, optional, defaults to "student_t") —
The distribution emission head for the model. Could be either “student_t”, “normal” or “negative_binomial”.   loss (string, optional, defaults to "nll") —
The loss function for the model corresponding to the distribution_output head. For parametric
distributions it is the negative log likelihood (nll) - which currently is the only supported one.   input_size (int, optional, defaults to 1) —
The size of the target variable which by default is 1


AutoformerConfig

 for univariate targets. Would be > 1 in case of
multivariate targets.   lags_sequence (list[int], optional, defaults to [1, 2, 3, 4, 5, 6, 7]) —
The lags of the input time series as covariates often dictated by the frequency. Default is [1, 2, 3, 4, 5, 6, 7].   scaling (bool, optional defaults to True) —
Whether to scale the input targets.   num_time_features (int, optional, defaults to 0) —
The number of time features in the input time series.   num_dynamic_real_features (int, optional, defaults to 0) —
The number of dynamic real valued features.   num_static_categorical_features (int, optional, defaults to 0) —
The number of static categorical features.   num_static_real_features (int, optional, defaults to 0) —
The number of static real valued features.   cardinality (list[int], optional) —
The cardinality (number of different values) for each of the static categorical features. Should be a list
of integers, having the same length as num_static_ca


AutoformerConfig

tegorical_features. Cannot be None if
num_static_categorical_features is > 0.   embedding_dimension (list[int], optional) —
The dimension of the embedding for each of the static categorical features. Should be a list of integers,
having the same length as num_static_categorical_features. Cannot be None if
num_static_categorical_features is > 0.   d_model (int, optional, defaults to 64) —
Dimensionality of the transformer layers.   encoder_layers (int, optional, defaults to 2) —
Number of encoder layers.   decoder_layers (int, optional, defaults to 2) —
Number of decoder layers.   encoder_attention_heads (int, optional, defaults to 2) —
Number of attention heads for each attention layer in the Transformer encoder.   decoder_attention_heads (int, optional, defaults to 2) —
Number of attention heads for each attention layer in the Transformer decoder.   encoder_ffn_dim (int, optional, defaults to 32) —
Dimension of the “intermediate” (often named feed-fo


AutoformerConfig

rward) layer in encoder.   decoder_ffn_dim (int, optional, defaults to 32) —
Dimension of the “intermediate” (often named feed-forward) layer in decoder.   activation_function (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and decoder. If string, "gelu" and
"relu" are supported.   dropout (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the encoder, and decoder.   encoder_layerdrop (float, optional, defaults to 0.1) —
The dropout probability for the attention and fully connected layers for each encoder layer.   decoder_layerdrop (float, optional, defaults to 0.1) —
The dropout probability for the attention and fully connected layers for each decoder layer.   attention_dropout (float, optional, defaults to 0.1) —
The dropout probability for the attention probabilities.   activation_dropout (float, optional, defaults to 0.1) —
The drop


AutoformerConfig

out probability used between the two layers of the feed-forward networks.   num_parallel_samples (int, optional, defaults to 100) —
The number of samples to generate in parallel for each time step of inference.   init_std (float, optional, defaults to 0.02) —
The standard deviation of the truncated normal weight initialization distribution.   use_cache (bool, optional, defaults to True) —
Whether to use the past key/values attentions (if applicable to the model) to speed up decoding.   label_length (int, optional, defaults to 10) —
Start token length of the Autoformer decoder, which is used for direct multi-step prediction (i.e.
non-autoregressive generation).   moving_average (int, defaults to 25) —
The window size of the moving average. In practice, it’s the kernel size in AvgPool1d of the Decomposition
Layer.   autocorrelation_factor (int, defaults to 3) —
“Attention” (i.e. AutoCorrelation mechanism) factor which is used to find top k autocorrelati


AutoformerConfig

ons delays.
It’s recommended in the paper to set it to a number between 1 and 5.    This is the configuration class to store the configuration of an AutoformerModel. It is used to instantiate an
Autoformer model according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the Autoformer
huggingface/autoformer-tourism-monthly
architecture. Configuration objects inherit from PretrainedConfig can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.    Copied >>> from transformers import AutoformerConfig, AutoformerModel
>>> # Initializing a default Autoformer configuration
>>> configuration = AutoformerConfig()
>>> # Randomly initializing a model (with random weights) from the configuration
>>> model = AutoformerModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.con


AutoformerModel

class transformers.AutoformerModel  < source > ( config: AutoformerConfig  )   Parameters   config (AutoformerConfig) —
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
from_pretrained() method to load the model weights.    The bare Autoformer Model outputting raw hidden-states without any specific head on top.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( past_values: Tensor past_time_features: Tensor past_observed_mask: Tensor static_categorical_features: typ


AutoformerModel

ing.Optional[torch.Tensor] = None static_real_features: typing.Optional[torch.Tensor] = None future_values: typing.Optional[torch.Tensor] = None future_time_features: typing.Optional[torch.Tensor] = None decoder_attention_mask: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.Tensor] = None decoder_head_mask: typing.Optional[torch.Tensor] = None cross_attn_head_mask: typing.Optional[torch.Tensor] = None encoder_outputs: typing.Optional[typing.List[torch.FloatTensor]] = None past_key_values: typing.Optional[typing.List[torch.FloatTensor]] = None output_hidden_states: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None use_cache: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.autoformer.modeling_autoformer.AutoformerModelOutput or tuple(torch.FloatTensor)  Parameters   past_values (torch.FloatTensor of shape (batch_size, sequence_length)) —
Past values of the time series, that s


AutoformerModel

erve as context in order to predict the future. These values may
contain lags, i.e. additional values from the past which are added in order to serve as “extra context”.
The past_values is what the Transformer encoder gets as input (with optional additional features, such as
static_categorical_features, static_real_features, past_time_features).
The sequence length here is equal to context_length + max(config.lags_sequence).
Missing values need to be replaced with zeros.   past_time_features (torch.FloatTensor of shape (batch_size, sequence_length, num_features), optional) —
Optional time features, which the model internally will add to past_values. These could be things like
“month of year”, “day of the month”, etc. encoded as vectors (for instance as Fourier features). These
could also be so-called “age” features, which basically help the model know “at which point in life” a
time-series is. Age features have small values for distant past time steps and increase monotonic


AutoformerModel

ally the
more we approach the current time step.
These features serve as the “positional encodings” of the inputs. So contrary to a model like BERT, where
the position encodings are learned from scratch internally as parameters of the model, the Time Series
Transformer requires to provide additional time features.
The Autoformer only learns additional embeddings for static_categorical_features.   past_observed_mask (torch.BoolTensor of shape (batch_size, sequence_length), optional) —
Boolean mask to indicate which past_values were observed and which were missing. Mask values selected in
[0, 1]:
1 for values that are observed,
0 for values that are missing (i.e. NaNs that were replaced by zeros).
   static_categorical_features (torch.LongTensor of shape (batch_size, number of static categorical features), optional) —
Optional static categorical features for which the model will learn an embedding, which it will add to the
values of the time series.
Static categorical feature


AutoformerModel

s are features which have the same value for all time steps (static over time).
A typical example of a static categorical feature is a time series ID.   static_real_features (torch.FloatTensor of shape (batch_size, number of static real features), optional) —
Optional static real features which the model will add to the values of the time series.
Static real features are features which have the same value for all time steps (static over time).
A typical example of a static real feature is promotion information.   future_values (torch.FloatTensor of shape (batch_size, prediction_length)) —
Future values of the time series, that serve as labels for the model. The future_values is what the
Transformer needs to learn to output, given the past_values.
See the demo notebook and code snippets for details.
Missing values need to be replaced with zeros.   future_time_features (torch.FloatTensor of shape (batch_size, prediction_length, num_features), optional) —
Optional time feature


AutoformerModel

s, which the model internally will add to future_values. These could be things like
“month of year”, “day of the month”, etc. encoded as vectors (for instance as Fourier features). These
could also be so-called “age” features, which basically help the model know “at which point in life” a
time-series is. Age features have small values for distant past time steps and increase monotonically the
more we approach the current time step.
These features serve as the “positional encodings” of the inputs. So contrary to a model like BERT, where
the position encodings are learned from scratch internally as parameters of the model, the Time Series
Transformer requires to provide additional features.
The Autoformer only learns additional embeddings for static_categorical_features.   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on certain token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked


AutoformerModel

,
0 for tokens that are masked.
What are attention masks?   decoder_attention_mask (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Mask to avoid performing attention on certain token indices. By default, a causal mask will be used, to
make sure the model can only look at previous inputs in order to predict the future.   head_mask (torch.Tensor of shape (encoder_layers, encoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   decoder_head_mask (torch.Tensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   cross_attn_head_mask (torch.Tensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask


AutoformerModel

 to nullify selected heads of the cross-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   encoder_outputs (tuple(tuple(torch.FloatTensor), optional) —
Tuple consists of last_hidden_state, hidden_states (optional) and attentions (optional)
last_hidden_state of shape (batch_size, sequence_length, hidden_size) (optional) is a sequence of
hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.   past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) —
Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-atte


AutoformerModel

ntion blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all


AutoformerModel

attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.models.autoformer.modeling_autoformer.AutoformerModelOutput or tuple(torch.FloatTensor)

A transformers.models.autoformer.modeling_autoformer.AutoformerModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (AutoformerConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the decoder of the model.
If past_key_values is used only the last hidden-state of the sequences of shape (batch_size, 1, hidden


AutoformerModel

_size) is output.
trend (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Trend tensor for each time series.
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +


AutoformerModel

one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the decoder at the output of each layer plus the optional initial embedding outputs.
decoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention hea


AutoformerModel

ds.
encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the optional initial embedding outputs.
encoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the encoder, after the attention softmax, used to compute


AutoformerModel

the weighted average in the
self-attention heads.
loc (torch.FloatTensor of shape (batch_size,) or (batch_size, input_size), optional) — Shift values of each time series’ context window which is used to give the model inputs of the same
magnitude and then used to shift back to the original magnitude.
scale (torch.FloatTensor of shape (batch_size,) or (batch_size, input_size), optional) — Scaling values of each time series’ context window which is used to give the model inputs of the same
magnitude and then used to rescale back to the original magnitude.
static_features: (torch.FloatTensor of shape (batch_size, feature size), optional) — Static features of each time series’ in a batch which are copied to the covariates at inference time.
  The AutoformerModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes


AutoformerModel

 care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from huggingface_hub import hf_hub_download
>>> import torch
>>> from transformers import AutoformerModel
>>> file = hf_hub_download(
...     repo_id="hf-internal-testing/tourism-monthly-batch", filename="train-batch.pt", repo_type="dataset"
... )
>>> batch = torch.load(file)
>>> model = AutoformerModel.from_pretrained("huggingface/autoformer-tourism-monthly")
>>> # during training, one provides both past and future values
>>> # as well as possible additional features
>>> outputs = model(
...     past_values=batch["past_values"],
...     past_time_features=batch["past_time_features"],
...     past_observed_mask=batch["past_observed_mask"],
...     static_categorical_features=batch["static_categorical_features"],
...     future_values=batch["future_values"],
...     future_time_features=batch["future_time_features"],
... )
>>> last_hidden_state = outputs.last_hi


AutoformerForPrediction

class transformers.AutoformerForPrediction  < source > ( config: AutoformerConfig  )   Parameters   config (AutoformerConfig) —
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
from_pretrained() method to load the model weights.    The Autoformer Model with a distribution head on top for time-series forecasting.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( past_values: Tensor past_time_features: Tensor past_observed_mask: Tensor


AutoformerForPrediction

static_categorical_features: typing.Optional[torch.Tensor] = None static_real_features: typing.Optional[torch.Tensor] = None future_values: typing.Optional[torch.Tensor] = None future_time_features: typing.Optional[torch.Tensor] = None future_observed_mask: typing.Optional[torch.Tensor] = None decoder_attention_mask: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.Tensor] = None decoder_head_mask: typing.Optional[torch.Tensor] = None cross_attn_head_mask: typing.Optional[torch.Tensor] = None encoder_outputs: typing.Optional[typing.List[torch.FloatTensor]] = None past_key_values: typing.Optional[typing.List[torch.FloatTensor]] = None output_hidden_states: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None use_cache: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.Seq2SeqTSPredictionOutput or tuple(torch.FloatTensor)  Parameters   past


AutoformerForPrediction

_values (torch.FloatTensor of shape (batch_size, sequence_length)) —
Past values of the time series, that serve as context in order to predict the future. These values may
contain lags, i.e. additional values from the past which are added in order to serve as “extra context”.
The past_values is what the Transformer encoder gets as input (with optional additional features, such as
static_categorical_features, static_real_features, past_time_features).
The sequence length here is equal to context_length + max(config.lags_sequence).
Missing values need to be replaced with zeros.   past_time_features (torch.FloatTensor of shape (batch_size, sequence_length, num_features), optional) —
Optional time features, which the model internally will add to past_values. These could be things like
“month of year”, “day of the month”, etc. encoded as vectors (for instance as Fourier features). These
could also be so-called “age” features, which basically help


AutoformerForPrediction

the model know “at which point in life” a
time-series is. Age features have small values for distant past time steps and increase monotonically the
more we approach the current time step.
These features serve as the “positional encodings” of the inputs. So contrary to a model like BERT, where
the position encodings are learned from scratch internally as parameters of the model, the Time Series
Transformer requires to provide additional time features.
The Autoformer only learns additional embeddings for static_categorical_features.   past_observed_mask (torch.BoolTensor of shape (batch_size, sequence_length), optional) —
Boolean mask to indicate which past_values were observed and which were missing. Mask values selected in
[0, 1]:
1 for values that are observed,
0 for values that are missing (i.e. NaNs that were replaced by zeros).
   static_categorical_features (torch.LongTensor of shape (batch_size, number of static categorical features), o


AutoformerForPrediction

ptional) —
Optional static categorical features for which the model will learn an embedding, which it will add to the
values of the time series.
Static categorical features are features which have the same value for all time steps (static over time).
A typical example of a static categorical feature is a time series ID.   static_real_features (torch.FloatTensor of shape (batch_size, number of static real features), optional) —
Optional static real features which the model will add to the values of the time series.
Static real features are features which have the same value for all time steps (static over time).
A typical example of a static real feature is promotion information.   future_values (torch.FloatTensor of shape (batch_size, prediction_length)) —
Future values of the time series, that serve as labels for the model. The future_values is what the
Transformer needs to learn to output, given the past_values.
See the demo notebook and co


AutoformerForPrediction

de snippets for details.
Missing values need to be replaced with zeros.   future_time_features (torch.FloatTensor of shape (batch_size, prediction_length, num_features), optional) —
Optional time features, which the model internally will add to future_values. These could be things like
“month of year”, “day of the month”, etc. encoded as vectors (for instance as Fourier features). These
could also be so-called “age” features, which basically help the model know “at which point in life” a
time-series is. Age features have small values for distant past time steps and increase monotonically the
more we approach the current time step.
These features serve as the “positional encodings” of the inputs. So contrary to a model like BERT, where
the position encodings are learned from scratch internally as parameters of the model, the Time Series
Transformer requires to provide additional features.
The Autoformer only learns additional embeddings for st


AutoformerForPrediction

atic_categorical_features.   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on certain token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   decoder_attention_mask (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Mask to avoid performing attention on certain token indices. By default, a causal mask will be used, to
make sure the model can only look at previous inputs in order to predict the future.   head_mask (torch.Tensor of shape (encoder_layers, encoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   decoder_head_mask (torch.Tensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nul


AutoformerForPrediction

lify selected heads of the attention modules in the decoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   cross_attn_head_mask (torch.Tensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the cross-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   encoder_outputs (tuple(tuple(torch.FloatTensor), optional) —
Tuple consists of last_hidden_state, hidden_states (optional) and attentions (optional)
last_hidden_state of shape (batch_size, sequence_length, hidden_size) (optional) is a sequence of
hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.   past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) —
Tuple of tuple(torch.FloatTensor) of len


AutoformerForPrediction

gth config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids


AutoformerForPrediction

 indices into associated vectors than the
model’s internal embedding lookup matrix.   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.Seq2SeqTSPredictionOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.Seq2SeqTSPredictionOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on t


AutoformerForPrediction

he configuration (AutoformerConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when a future_values is provided) — Distributional loss.
params (torch.FloatTensor of shape (batch_size, num_samples, num_params)) — Parameters of the chosen distribution.
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_h


AutoformerForPrediction

idden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the decoder at the output of each layer plus the initial embedding outputs.
decoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_siz


AutoformerForPrediction

e, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the initial embedding outputs.
encoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config


AutoformerForPrediction

.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
loc (torch.FloatTensor of shape (batch_size,) or (batch_size, input_size), optional) — Shift values of each time series’ context window which is used to give the model inputs of the same
magnitude and then used to shift back to the original magnitude.
scale (torch.FloatTensor of shape (batch_size,) or (batch_size, input_size), optional) — Scaling values of each time series’ context window which is used to give the model inputs of the same
magnitude and then used to rescale back to the original magnitude.
static_features (torch.FloatTensor of shape (batch_size, feature size), optional) — Static features of each time series’ in a batch which are copied to the covariates at inf


AutoformerForPrediction

erence time.
  The AutoformerForPrediction forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from huggingface_hub import hf_hub_download
>>> import torch
>>> from transformers import AutoformerForPrediction
>>> file = hf_hub_download(
...     repo_id="hf-internal-testing/tourism-monthly-batch", filename="train-batch.pt", repo_type="dataset"
... )
>>> batch = torch.load(file)
>>> model = AutoformerForPrediction.from_pretrained("huggingface/autoformer-tourism-monthly")
>>> # during training, one provides both past and future values
>>> # as well as possible additional features
>>> outputs = model(
...     past_values=batch["past_values"],
...     past_time_features=


AutoformerForPrediction

batch["past_time_features"],
...     past_observed_mask=batch["past_observed_mask"],
...     static_categorical_features=batch["static_categorical_features"],
...     static_real_features=batch["static_real_features"],
...     future_values=batch["future_values"],
...     future_time_features=batch["future_time_features"],
... )
>>> loss = outputs.loss
>>> loss.backward()
>>> # during inference, one only provides past values
>>> # as well as possible additional features
>>> # the model autoregressively generates future values
>>> outputs = model.generate(
...     past_values=batch["past_values"],
...     past_time_features=batch["past_time_features"],
...     past_observed_mask=batch["past_observed_mask"],
...     static_categorical_features=batch["static_categorical_features"],
...     static_real_features=batch["static_real_features"],
...     future_time_features=batch["future_time_features"],
... )
>>> mean_prediction = outputs.sequences.


Overview

The ELECTRA model was proposed in the paper ELECTRA: Pre-training Text Encoders as Discriminators Rather Than
Generators. ELECTRA is a new pretraining approach which trains two
transformer models: the generator and the discriminator. The generator’s role is to replace tokens in a sequence, and
is therefore trained as a masked language model. The discriminator, which is the model we’re interested in, tries to
identify which tokens were replaced by the generator in the sequence.
The abstract from the paper is the following:
Masked language modeling (MLM) pretraining methods such as BERT corrupt the input by replacing some tokens with [MASK]
and then train a model to reconstruct the original tokens. While they produce good results when transferred to
downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a
more sample-efficient pretraining task called replaced token de


Overview

tection. Instead of masking the input, our approach
corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead
of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that
predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments
demonstrate this new pretraining task is more efficient than MLM because the task is defined over all input tokens
rather than just the small subset that was masked out. As a result, the contextual representations learned by our
approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are
particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained
using 30x more compute) on the GLUE natural language understanding


Overview

 benchmark. Our approach also works well at scale,
where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when
using the same amount of compute.
Tips:
ELECTRA is the pretraining approach, therefore there is nearly no changes done to the underlying model: BERT. The
only change is the separation of the embedding size and the hidden size: the embedding size is generally smaller,
while the hidden size is larger. An additional projection layer (linear) is used to project the embeddings from their
embedding size to the hidden size. In the case where the embedding size is the same as the hidden size, no projection
layer is used. ELECTRA is a transformer model pretrained with the use of another (small) masked language model. The inputs are corrupted by that language model, which takes an input text that is randomly masked and outputs a text in which ELECTRA has to predict whic


Overview

h token is an original and which one has been replaced. Like for GAN training, the small language model is trained for a few steps (but with the original texts as objective, not to fool the ELECTRA model like in a traditional GAN setting) then the ELECTRA model is trained for a few steps. The ELECTRA checkpoints saved using Google Research’s implementation
contain both the generator and discriminator. The conversion script requires the user to name which model to export
into the correct architecture. Once converted to the HuggingFace format, these checkpoints may be loaded into all
available ELECTRA models, however. This means that the discriminator may be loaded in the
ElectraForMaskedLM model, and the generator may be loaded in the
ElectraForPreTraining model (the classification head will be randomly initialized as it
doesn’t exist in the generator).
This model was contributed by lysandre. The original code can be found here


Documentation resources

Text classification task guide Token classification task guide Question answering task guide Causal language modeling task guide Masked language modeling task guide Multiple choice task guide


ElectraConfig

class transformers.ElectraConfig  < source > ( vocab_size = 30522 embedding_size = 128 hidden_size = 256 num_hidden_layers = 12 num_attention_heads = 4 intermediate_size = 1024 hidden_act = 'gelu' hidden_dropout_prob = 0.1 attention_probs_dropout_prob = 0.1 max_position_embeddings = 512 type_vocab_size = 2 initializer_range = 0.02 layer_norm_eps = 1e-12 summary_type = 'first' summary_use_proj = True summary_activation = 'gelu' summary_last_dropout = 0.1 pad_token_id = 0 position_embedding_type = 'absolute' use_cache = True classifier_dropout = None **kwargs  )   Parameters   vocab_size (int, optional, defaults to 30522) —
Vocabulary size of the ELECTRA model. Defines the number of different tokens that can be represented by the
inputs_ids passed when calling ElectraModel or TFElectraModel.   embedding_size (int, optional, defaults to 128) —
Dimensionality of the encoder layers and the p