MegatronBertForTokenClassification

atTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention


MegatronBertForTokenClassification

modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_siz


MegatronBertForTokenClassification

e, sequence_length), optional) —
Labels for computing the token classification loss. Indices should be in [0, ..., config.num_labels - 1].   Returns
transformers.modeling_outputs.TokenClassifierOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.TokenClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (MegatronBertConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided)  — Classification loss.
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.num_labels)) — Classification scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the mode


MegatronBertForTokenClassification

l has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The MegatronBertForTokenClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.


MegatronBertForTokenClassification

Example:   Copied >>> from transformers import AutoTokenizer, MegatronBertForTokenClassification
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("nvidia/megatron-bert-cased-345m")
>>> model = MegatronBertForTokenClassification.from_pretrained("nvidia/megatron-bert-cased-345m")
>>> inputs = tokenizer(
...     "HuggingFace is a company based in Paris and New York", add_special_tokens=False, return_tensors="pt"
... )
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_token_class_ids = logits.argmax(-1)
>>> # Note that tokens are classified rather then input words which means that
>>> # there might be more predicted token classes than words.
>>> # Multiple token classes might account for the same word
>>> predicted_tokens_classes = [model.config.id2label[t.item()] for t in predicted_token_class_ids[0]]
>>> labels = predicted_token_class_ids
>>> loss = model(**inputs, labels=labels


MegatronBertForQuestionAnswering

class transformers.MegatronBertForQuestionAnswering  < source > ( config  )   Parameters   config (MegatronBertConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    MegatronBert Model with a span classification head on top for extractive question-answering tasks like SQuAD (a
linear layers on top of the hidden-states output to compute span start logits and span end logits). This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTo


MegatronBertForQuestionAnswering

rch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None start_positions: typing.Optional[torch.LongTensor] = None end_positions: typing.Optional[torch.LongTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.QuestionAnsweringModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
I


MegatronBertForQuestionAnswering

ndices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_posit


MegatronBertForQuestionAnswering

ion_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of al


MegatronBertForQuestionAnswering

l layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   start_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the start of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
are not taken into account for computing the loss.   end_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the end of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
are not taken into account for computing the loss.   Returns
transformers.modeling_outputs.QuestionAnsweringModelOutput or tuple(to


MegatronBertForQuestionAnswering

rch.FloatTensor)

A transformers.modeling_outputs.QuestionAnsweringModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (MegatronBertConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Total span extraction loss is the sum of a Cross-Entropy for the start and end positions.
start_logits (torch.FloatTensor of shape (batch_size, sequence_length)) — Span-start scores (before SoftMax).
end_logits (torch.FloatTensor of shape (batch_size, sequence_length)) — Span-end scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +


MegatronBertForQuestionAnswering

one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The MegatronBertForQuestionAnswering forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example


MegatronBertForQuestionAnswering

:   Copied >>> from transformers import AutoTokenizer, MegatronBertForQuestionAnswering
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("nvidia/megatron-bert-cased-345m")
>>> model = MegatronBertForQuestionAnswering.from_pretrained("nvidia/megatron-bert-cased-345m")
>>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
>>> inputs = tokenizer(question, text, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> answer_start_index = outputs.start_logits.argmax()
>>> answer_end_index = outputs.end_logits.argmax()
>>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
>>> # target is "nice puppet"
>>> target_start_index = torch.tensor([14])
>>> target_end_index = torch.tensor([15])
>>> outputs = model(**inputs, start_positions=target_start_index, end_positions=target_end_index)
>>> loss = outputs.loss


Overview

The Wav2Vec2-Conformer was added to an updated version of fairseq S2T: Fast Speech-to-Text Modeling with fairseq by Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Sravya Popuri, Dmytro Okhonko, Juan Pino.
The official results of the model can be found in Table 3 and Table 4 of the paper.
The Wav2Vec2-Conformer weights were released by the Meta AI team within the Fairseq library.
Tips:
Wav2Vec2-Conformer follows the same architecture as Wav2Vec2, but replaces the Attention-block with a Conformer-block
as introduced in Conformer: Convoluti


Overview

on-augmented Transformer for Speech Recognition. For the same number of layers, Wav2Vec2-Conformer requires more parameters than Wav2Vec2, but also yields
an improved word error rate. Wav2Vec2-Conformer uses the same tokenizer and feature extractor as Wav2Vec2. Wav2Vec2-Conformer can use either no relative position embeddings, Transformer-XL-like position embeddings, or
rotary position embeddings by setting the correct config.position_embeddings_type.
This model was contributed by patrickvonplaten.
The original code can be found here.


class transformers.Wav2Vec2ConformerConfig  < source > ( vocab_size = None hidden_size = 768 num_hidden_layers = 12 num_attention_heads = 12 intermediate_size = 3072 hidden_act = 'gelu' hidden_dropout = 0.1 activation_dropout = 0.1 attention_dropout = 0.1 feat_proj_dropout = 0.0 feat_quantizer_dropout = 0.0 final_dropout = 0.1 layerdrop = 0.1 initializer_range = 0.02 layer_norm_eps = 1e-05 feat_extract_norm = 'group' feat_extract_activation = 'gelu' conv_dim = (512, 512, 512, 512, 512, 512, 512) conv_stride = (5, 2, 2, 2, 2, 2, 2) conv_kernel = (10, 3, 3, 3, 3, 2, 2) conv_bias = False num_conv_pos_embeddings = 128 num_conv_pos_embedding_groups = 16 apply_spec_augment = True mask_time_prob = 0.05 mask_time_length = 10 mask_time_min_masks = 2 mask_feature_prob = 0.0 mask_feature_length = 10 mask_feature_min_masks = 0 num_codevectors_per_group = 320 num_codevector_groups = 2 contrastive_logits_temperature = 0.1 num_negatives = 100 cod


evector_dim = 256 proj_codevector_dim = 256 diversity_loss_weight = 0.1 ctc_loss_reduction = 'sum' ctc_zero_infinity = False use_weighted_layer_sum = False classifier_proj_size = 256 tdnn_dim = (512, 512, 512, 512, 1500) tdnn_kernel = (5, 3, 3, 1, 1) tdnn_dilation = (1, 2, 3, 1, 1) xvector_output_dim = 512 pad_token_id = 0 bos_token_id = 1 eos_token_id = 2 add_adapter = False adapter_kernel_size = 3 adapter_stride = 2 num_adapter_layers = 3 output_hidden_size = None position_embeddings_type = 'relative' rotary_embedding_base = 10000 max_source_positions = 5000 conv_depthwise_kernel_size = 31 conformer_conv_dropout = 0.1 **kwargs  )   Parameters   vocab_size (int, optional) —
Vocabulary size of the Wav2Vec2Conformer model. Defines the number of different tokens that can be
represented by the inputs_ids passed when calling Wav2Vec2ConformerModel. Vocabulary size of the
model. Defines the different tokens that can be represented by th


e inputs_ids passed to the forward
method of Wav2Vec2ConformerModel.   hidden_size (int, optional, defaults to 768) —
Dimensionality of the encoder layers and the pooler layer.   num_hidden_layers (int, optional, defaults to 12) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 12) —
Number of attention heads for each attention layer in the Transformer encoder.   intermediate_size (int, optional, defaults to 3072) —
Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.   hidden_act (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "selu" and "gelu_new" are supported.   hidden_dropout (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.   activation_dropout


 (float, optional, defaults to 0.1) —
The dropout ratio for activations inside the fully connected layer.   attention_dropout (float, optional, defaults to 0.1) —
The dropout ratio for the attention probabilities.   final_dropout (float, optional, defaults to 0.1) —
The dropout probability for the final projection layer of Wav2Vec2ConformerForCTC.   layerdrop (float, optional, defaults to 0.1) —
The LayerDrop probability. See the [LayerDrop paper](see https://arxiv.org/abs/1909.11556) for more
details.   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   layer_norm_eps (float, optional, defaults to 1e-12) —
The epsilon used by the layer normalization layers.   feat_extract_norm (str, optional, defaults to "group") —
The norm to be applied to 1D convolutional layers in feature encoder. One of "group" for group
normalization of onl


y the first 1D convolutional layer or "layer" for layer normalization of all 1D
convolutional layers.   feat_proj_dropout (float, optional, defaults to 0.0) —
The dropout probability for output of the feature encoder.   feat_extract_activation (str, optional, defaults to “gelu”) -- The non-linear activation function (function or string) in the 1D convolutional layers of the feature extractor. If string, “gelu”, “relu”, “selu”and“gelu_new”` are supported.   feat_quantizer_dropout (float, optional, defaults to 0.0) —
The dropout probabilitiy for quantized feature encoder states.   conv_dim (Tuple[int] or List[int], optional, defaults to (512, 512, 512, 512, 512, 512, 512)) —
A tuple of integers defining the number of input and output channels of each 1D convolutional layer in the
feature encoder. The length of conv_dim defines the number of 1D convolutional layers.   conv_stride (Tuple[int] or List[int], optional, defaults to (5, 2,


2, 2, 2, 2, 2)) —
A tuple of integers defining the stride of each 1D convolutional layer in the feature encoder. The length
of conv_stride defines the number of convolutional layers and has to match the length of conv_dim.   conv_kernel (Tuple[int] or List[int], optional, defaults to (10, 3, 3, 3, 3, 3, 3)) —
A tuple of integers defining the kernel size of each 1D convolutional layer in the feature encoder. The
length of conv_kernel defines the number of convolutional layers and has to match the length of
conv_dim.   conv_bias (bool, optional, defaults to False) —
Whether the 1D convolutional layers have a bias.   num_conv_pos_embeddings (int, optional, defaults to 128) —
Number of convolutional positional embeddings. Defines the kernel size of 1D convolutional positional
embeddings layer.   num_conv_pos_embedding_groups (int, optional, defaults to 16) —
Number of groups of 1D convolutional positional embeddings layer.   apply_spec


_augment (bool, optional, defaults to True) —
Whether to apply SpecAugment data augmentation to the outputs of the feature encoder. For reference see
SpecAugment: A Simple Data Augmentation Method for Automatic Speech
Recognition.   mask_time_prob (float, optional, defaults to 0.05) —
Percentage (between 0 and 1) of all feature vectors along the time axis which will be masked. The masking
procecure generates ”mask_time_problen(time_axis)/mask_time_length” independent masks over the axis. If
reasoning from the propability of each feature vector to be chosen as the start of the vector span to be
masked, mask_time_prob should be `prob_vector_startmask_time_length. Note that overlap may decrease the actual percentage of masked vectors. This is only relevant if apply_spec_augment is True`.   mask_time_length (int, optional, defaults to 10) —
Length of vector span along the time axis.   mask_time_min_masks (int, optional, defaults to 2),


 —
The minimum number of masks of length mask_feature_length generated along the time axis, each time step,
irrespectively of mask_feature_prob. Only relevant if ”mask_time_prob*len(time_axis)/mask_time_length <
mask_time_min_masks”   mask_feature_prob (float, optional, defaults to 0.0) —
Percentage (between 0 and 1) of all feature vectors along the feature axis which will be masked. The
masking procecure generates ”mask_feature_problen(feature_axis)/mask_time_length” independent masks over
the axis. If reasoning from the propability of each feature vector to be chosen as the start of the vector
span to be masked, mask_feature_prob should be `prob_vector_startmask_feature_length. Note that overlap may decrease the actual percentage of masked vectors. This is only relevant if apply_spec_augment is
True`.   mask_feature_length (int, optional, defaults to 10) —
Length of vector span along the feature axis.   mask_feature_min_masks (in


t, optional, defaults to 0), —
The minimum number of masks of length mask_feature_length generated along the feature axis, each time
step, irrespectively of mask_feature_prob. Only relevant if
”mask_feature_prob*len(feature_axis)/mask_feature_length < mask_feature_min_masks”   num_codevectors_per_group (int, optional, defaults to 320) —
Number of entries in each quantization codebook (group).   num_codevector_groups (int, optional, defaults to 2) —
Number of codevector groups for product codevector quantization.   contrastive_logits_temperature (float, optional, defaults to 0.1) —
The temperature kappa in the contrastive loss.   feat_quantizer_dropout (float, optional, defaults to 0.0) —
The dropout probabilitiy for the output of the feature encoder that’s used by the quantizer.   num_negatives (int, optional, defaults to 100) —
Number of negative samples for the contrastive loss.   codevector_dim (int, optional, defaults to 256) —


Dimensionality of the quantized feature vectors.   proj_codevector_dim (int, optional, defaults to 256) —
Dimensionality of the final projection of both the quantized and the transformer features.   diversity_loss_weight (int, optional, defaults to 0.1) —
The weight of the codebook diversity loss component.   ctc_loss_reduction (str, optional, defaults to "sum") —
Specifies the reduction to apply to the output of torch.nn.CTCLoss. Only relevant when training an
instance of Wav2Vec2ConformerForCTC.   ctc_zero_infinity (bool, optional, defaults to False) —
Whether to zero infinite losses and the associated gradients of torch.nn.CTCLoss. Infinite losses mainly
occur when the inputs are too short to be aligned to the targets. Only relevant when training an instance
of Wav2Vec2ConformerForCTC.   use_weighted_layer_sum (bool, optional, defaults to False) —
Whether to use a weighted average of layer outputs with learned weights. Only rel


evant when using an
instance of Wav2Vec2ConformerForSequenceClassification.   classifier_proj_size (int, optional, defaults to 256) —
Dimensionality of the projection before token mean-pooling for classification.   tdnn_dim (Tuple[int] or List[int], optional, defaults to (512, 512, 512, 512, 1500)) —
A tuple of integers defining the number of output channels of each 1D convolutional layer in the TDNN
module of the XVector model. The length of tdnn_dim defines the number of TDNN layers.   tdnn_kernel (Tuple[int] or List[int], optional, defaults to (5, 3, 3, 1, 1)) —
A tuple of integers defining the kernel size of each 1D convolutional layer in the TDNN module of the
XVector model. The length of tdnn_kernel has to match the length of tdnn_dim.   tdnn_dilation (Tuple[int] or List[int], optional, defaults to (1, 2, 3, 1, 1)) —
A tuple of integers defining the dilation factor of each 1D convolutional layer in TDNN module of the
XVector


model. The length of tdnn_dilation has to match the length of tdnn_dim.   xvector_output_dim (int, optional, defaults to 512) —
Dimensionality of the XVector embedding vectors.   add_adapter (bool, optional, defaults to False) —
Whether a convolutional network should be stacked on top of the Wav2Vec2Conformer Encoder. Can be very
useful for warm-starting Wav2Vec2Conformer for SpeechEncoderDecoder models.   adapter_kernel_size (int, optional, defaults to 3) —
Kernel size of the convolutional layers in the adapter network. Only relevant if add_adapter is True.   adapter_stride (int, optional, defaults to 2) —
Stride of the convolutional layers in the adapter network. Only relevant if add_adapter is True.   num_adapter_layers (int, optional, defaults to 3) —
Number of convolutional layers that should be used in the adapter network. Only relevant if add_adapter is True.   output_hidden_size (int, optional) —
Dimensionality of the encod


er output layer. If not defined, this defaults to hidden-size. Only relevant
if add_adapter is True.   position_embeddings_type (str, optional, defaults to "relative") —
Can be specified to relative or rotary for relative or rotary position embeddings respectively. If left
None no relative position embedding is applied.   rotary_embedding_base (int, optional, defaults to 10000) —
If "rotary" position embeddings are used, defines the size of the embedding base.   max_source_positions (int, optional, defaults to 5000) —
if "relative" position embeddings are used, defines the maximum source input positions.   conv_depthwise_kernel_size (int, defaults to 31) —
Kernel size of convolutional depthwise 1D layer in Conformer blocks.   conformer_conv_dropout (float, defaults to 0.1) —
The dropout probability for all convolutional layers in Conformer blocks.    This is the configuration class to store the configuration of a Wav2Vec2ConformerM


odel. It is used to
instantiate an Wav2Vec2Conformer model according to the specified arguments, defining the model architecture.
Instantiating a configuration with the defaults will yield a similar configuration to that of the Wav2Vec2Conformer
facebook/wav2vec2-conformer-rel-pos-large
architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import Wav2Vec2ConformerConfig, Wav2Vec2ConformerModel
>>> # Initializing a Wav2Vec2Conformer facebook/wav2vec2-conformer-rel-pos-large style configuration
>>> configuration = Wav2Vec2ConformerConfig()
>>> # Initializing a model (with random weights) from the facebook/wav2vec2-conformer-rel-pos-large style configuration
>>> model = Wav2Vec2ConformerModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model


Wav2Vec2Conformer specific outputs

class transformers.models.wav2vec2_conformer.modeling_wav2vec2_conformer.Wav2Vec2ConformerForPreTrainingOutput  < source > ( loss: typing.Optional[torch.FloatTensor] = None projected_states: FloatTensor = None projected_quantized_states: FloatTensor = None codevector_perplexity: FloatTensor = None hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None attentions: typing.Optional[typing.Tuple[torch.FloatTensor]] = None contrastive_loss: typing.Optional[torch.FloatTensor] = None diversity_loss: typing.Optional[torch.FloatTensor] = None  )   Parameters   loss (optional, returned when sample_negative_indices are passed, torch.FloatTensor of shape (1,)) —
Total loss as the sum of the contrastive loss (L_m) and the diversity loss (L_d) as stated in the official
paper . (classification) loss.   projected_states (torch


Wav2Vec2Conformer specific outputs

.FloatTensor of shape (batch_size, sequence_length, config.proj_codevector_dim)) —
Hidden-states of the model projected to config.proj_codevector_dim that can be used to predict the masked
projected quantized states.   projected_quantized_states (torch.FloatTensor of shape (batch_size, sequence_length, config.proj_codevector_dim)) —
Quantized extracted feature vectors projected to config.proj_codevector_dim representing the positive
target vectors for contrastive loss.   hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the


Wav2Vec2Conformer specific outputs

 initial embedding outputs.   attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) —
Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.   contrastive_loss (optional, returned when sample_negative_indices are passed, torch.FloatTensor of shape (1,)) —
The contrastive loss (L_m) as stated in the official paper .   diversity_loss (optional, returned when sample_negative_indices are passed, torch.FloatTensor of shape (1,)) —
The diversity loss (L_d) as stated in the official paper .    Output type of Wav2Vec2ConformerForPreTraining, with potential hidden states and attentions


class transformers.Wav2Vec2ConformerModel  < source > ( config: Wav2Vec2ConformerConfig  )   Parameters   config (Wav2Vec2ConformerConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare Wav2Vec2Conformer Model transformer outputting raw hidden-states without any specific head on top.
Wav2Vec2Conformer was proposed in wav2vec 2.0: A Framework for Self-Supervised Learning of Speech
Representations by Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael
Auli. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving etc.). This model is a PyTorch nn.Module sub-class. Use it as a
regular PyTorch Module and refer to the PyTorch docume


ntation for all matter related to general usage and behavior.  forward  < source > ( input_values: typing.Optional[torch.Tensor] attention_mask: typing.Optional[torch.Tensor] = None mask_time_indices: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.Wav2Vec2BaseModelOutput or tuple(torch.FloatTensor)  Parameters   input_values (torch.FloatTensor of shape (batch_size, sequence_length)) —
Float values of input raw speech waveform. Values can be obtained by loading a .flac or .wav audio file
into an array of type List[float] or a numpy.ndarray, e.g. via the soundfile library (pip install soundfile). To prepare the array into input_values, the AutoProcessor should be used for padding and
conversion into a tensor of type torch.FloatTensor. See Wav2Vec2Processor.call() for details.   atte


ntion_mask (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing convolution and attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?
attention_mask should only be passed if the corresponding processor has config.return_attention_mask == True. For all models whose processor has config.return_attention_mask == False, such as
wav2vec2-conformer-rel-pos-large,
attention_mask should not be passed to avoid degraded performance when doing batched inference. For
such models input_values should simply be padded with 0 and passed without attention_mask. Be aware
that these models also yield slightly different results depending on whether input_values is padded or
not.
   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tenso


rs for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.Wav2Vec2BaseModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.Wav2Vec2BaseModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (Wav2Vec2ConformerConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
extract_features (torch.FloatTensor of shape (batch_size, sequence_length, conv_dim[-1])) — Sequence of extracted feature vectors of the last convolutional layer of the model.


hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The Wav2Vec2ConformerModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterw


ards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoProcessor, Wav2Vec2ConformerModel
>>> import torch
>>> from datasets import load_dataset
>>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
>>> dataset = dataset.sort("id")
>>> sampling_rate = dataset.features["audio"].sampling_rate
>>> processor = AutoProcessor.from_pretrained("facebook/wav2vec2-conformer-rope-large-960h-ft")
>>> model = Wav2Vec2ConformerModel.from_pretrained("facebook/wav2vec2-conformer-rope-large-960h-ft")
>>> # audio file is decoded on the fly
>>> inputs = processor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1, 292,


class transformers.Wav2Vec2ConformerForCTC  < source > ( config target_lang: typing.Optional[str] = None  )   Parameters   config (Wav2Vec2ConformerConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Wav2Vec2Conformer Model with a language modeling head on top for Connectionist Temporal Classification (CTC).
Wav2Vec2Conformer was proposed in wav2vec 2.0: A Framework for Self-Supervised Learning of Speech
Representations by Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael
Auli. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving etc.). This model is a PyTorch nn.Module sub-class. Use it as a
reg


ular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.  forward  < source > ( input_values: typing.Optional[torch.Tensor] attention_mask: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None labels: typing.Optional[torch.Tensor] = None  ) → transformers.modeling_outputs.CausalLMOutput or tuple(torch.FloatTensor)  Parameters   input_values (torch.FloatTensor of shape (batch_size, sequence_length)) —
Float values of input raw speech waveform. Values can be obtained by loading a .flac or .wav audio file
into an array of type List[float] or a numpy.ndarray, e.g. via the soundfile library (pip install soundfile). To prepare the array into input_values, the AutoProcessor should be used for padding and
conversion into a tensor of type torch.Flo


atTensor. See Wav2Vec2Processor.call() for details.   attention_mask (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing convolution and attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?
attention_mask should only be passed if the corresponding processor has config.return_attention_mask == True. For all models whose processor has config.return_attention_mask == False, such as
wav2vec2-conformer-rel-pos-large,
attention_mask should not be passed to avoid degraded performance when doing batched inference. For
such models input_values should simply be padded with 0 and passed without attention_mask. Be aware
that these models also yield slightly different results depending on whether input_values is padded or
not.
   output_attentions (bool, optional) —
Whether or not to


return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size, target_length), optional) —
Labels for connectionist temporal classification. Note that target_length has to be smaller or equal to
the sequence length of the output logits. Indices are selected in [-100, 0, ..., config.vocab_size - 1].
All labels set to -100 are ignored (masked), the loss is only computed for labels in [0, ..., config.vocab_size - 1].   Returns
transformers.modeling_outputs.CausalLMOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.CausalLMOutput or a tuple of
torch.FloatT


ensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (Wav2Vec2ConformerConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Language modeling loss (for next-token prediction).
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
a


ttentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The Wav2Vec2ConformerForCTC forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoProcessor, Wav2Vec2ConformerForCTC
>>> from datasets import load_dataset
>>> import torch
>>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", spl


it="validation")
>>> dataset = dataset.sort("id")
>>> sampling_rate = dataset.features["audio"].sampling_rate
>>> processor = AutoProcessor.from_pretrained("facebook/wav2vec2-conformer-rope-large-960h-ft")
>>> model = Wav2Vec2ConformerForCTC.from_pretrained("facebook/wav2vec2-conformer-rope-large-960h-ft")
>>> # audio file is decoded on the fly
>>> inputs = processor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_ids = torch.argmax(logits, dim=-1)
>>> # transcribe speech
>>> transcription = processor.batch_decode(predicted_ids)
>>> transcription[0]
'MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL'
>>> inputs["labels"] = processor(text=dataset[0]["text"], return_tensors="pt").input_ids
>>> # compute loss
>>> loss = model(**inputs).loss
>>> round(loss.item(), 2)
64.2


class transformers.Wav2Vec2ConformerForSequenceClassification  < source > ( config  )   Parameters   config (Wav2Vec2ConformerConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Wav2Vec2Conformer Model with a sequence classification head on top (a linear layer over the pooled output) for
tasks like SUPERB Keyword Spotting. Wav2Vec2Conformer was proposed in wav2vec 2.0: A Framework for Self-Supervised Learning of Speech
Representations by Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael
Auli. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving etc.). This model is a PyTorch nn.Module sub


-class. Use it as a
regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.  forward  < source > ( input_values: typing.Optional[torch.Tensor] attention_mask: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None labels: typing.Optional[torch.Tensor] = None  ) → transformers.modeling_outputs.SequenceClassifierOutput or tuple(torch.FloatTensor)  Parameters   input_values (torch.FloatTensor of shape (batch_size, sequence_length)) —
Float values of input raw speech waveform. Values can be obtained by loading a .flac or .wav audio file
into an array of type List[float] or a numpy.ndarray, e.g. via the soundfile library (pip install soundfile). To prepare the array into input_values, the AutoProcessor should be used for padding and
c


onversion into a tensor of type torch.FloatTensor. See Wav2Vec2Processor.call() for details.   attention_mask (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing convolution and attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?
attention_mask should only be passed if the corresponding processor has config.return_attention_mask == True. For all models whose processor has config.return_attention_mask == False, such as
wav2vec2-conformer-rel-pos-large,
attention_mask should not be passed to avoid degraded performance when doing batched inference. For
such models input_values should simply be padded with 0 and passed without attention_mask. Be aware
that these models also yield slightly different results depending on whether input_values is padded or
not.
   output


_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-Entropy).   Returns
transformers.modeling_outputs.SequenceClassifierOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.SequenceClassifierOutput or a tuple of
torc


h.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (Wav2Vec2ConformerConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (


tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The Wav2Vec2ConformerForSequenceClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoFeatureExtractor, Wav2Vec2ConformerForSequenceClassification
>>> from datasets import load_dataset
>>> import torch
>>> dataset = load_dataset("hf-internal-t


esting/librispeech_asr_demo", "clean", split="validation")
>>> dataset = dataset.sort("id")
>>> sampling_rate = dataset.features["audio"].sampling_rate
>>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-conformer-rope-large-960h-ft")
>>> model = Wav2Vec2ConformerForSequenceClassification.from_pretrained("facebook/wav2vec2-conformer-rope-large-960h-ft")
>>> # audio file is decoded on the fly
>>> inputs = feature_extractor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_class_ids = torch.argmax(logits, dim=-1).item()
>>> predicted_label = model.config.id2label[predicted_class_ids]
>>> # compute loss - target_label is e.g. "down"
>>> target_label = model.config.id2label[0]
>>> inputs["labels"] = torch.tensor([model.config.label2id[target_label]])
>>> loss = model(**inputs).


class transformers.Wav2Vec2ConformerForAudioFrameClassification  < source > ( config  )   Parameters   config (Wav2Vec2ConformerConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Wav2Vec2Conformer Model with a frame classification head on top for tasks like Speaker Diarization. Wav2Vec2Conformer was proposed in wav2vec 2.0: A Framework for Self-Supervised Learning of Speech
Representations by Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael
Auli. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving etc.). This model is a PyTorch nn.Module sub-clas


s. Use it as a
regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.  forward  < source > ( input_values: typing.Optional[torch.Tensor] attention_mask: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.TokenClassifierOutput or tuple(torch.FloatTensor)  Parameters   input_values (torch.FloatTensor of shape (batch_size, sequence_length)) —
Float values of input raw speech waveform. Values can be obtained by loading a .flac or .wav audio file
into an array of type List[float] or a numpy.ndarray, e.g. via the soundfile library (pip install soundfile). To prepare the array into input_values, the AutoProcessor


should be used for padding and
conversion into a tensor of type torch.FloatTensor. See Wav2Vec2Processor.call() for details.   attention_mask (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing convolution and attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?
attention_mask should only be passed if the corresponding processor has config.return_attention_mask == True. For all models whose processor has config.return_attention_mask == False, such as
wav2vec2-conformer-rel-pos-large,
attention_mask should not be passed to avoid degraded performance when doing batched inference. For
such models input_values should simply be padded with 0 and passed without attention_mask. Be aware
that these models also yield slightly differe


nt results depending on whether input_values is padded or
not.
   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-Entropy).   Returns
transformers.modeling_outputs.TokenClassifierOutp


ut or tuple(torch.FloatTensor)

A transformers.modeling_outputs.TokenClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (Wav2Vec2ConformerConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided)  — Classification loss.
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.num_labels)) — Classification scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model a


t the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The Wav2Vec2ConformerForAudioFrameClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoFeatureExtractor, Wav2Vec2Confo


rmerForAudioFrameClassification
>>> from datasets import load_dataset
>>> import torch
>>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
>>> dataset = dataset.sort("id")
>>> sampling_rate = dataset.features["audio"].sampling_rate
>>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-conformer-rope-large-960h-ft")
>>> model = Wav2Vec2ConformerForAudioFrameClassification.from_pretrained("facebook/wav2vec2-conformer-rope-large-960h-ft")
>>> # audio file is decoded on the fly
>>> inputs = feature_extractor(dataset[0]["audio"]["array"], return_tensors="pt", sampling_rate=sampling_rate)
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> probabilities = torch.sigmoid(logits[0])
>>> # labels is a one-hot array of shape (num_frames, num_speakers)
>>> labels = (probabilities > 0.5).long


class transformers.Wav2Vec2ConformerForXVector  < source > ( config  )   Parameters   config (Wav2Vec2ConformerConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Wav2Vec2Conformer Model with an XVector feature extraction head on top for tasks like Speaker Verification. Wav2Vec2Conformer was proposed in wav2vec 2.0: A Framework for Self-Supervised Learning of Speech
Representations by Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael
Auli. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving etc.). This model is a PyTorch nn.Module sub-class. Use it as a
regular PyTorch Module and refer to th


e PyTorch documentation for all matter related to general usage and behavior.  forward  < source > ( input_values: typing.Optional[torch.Tensor] attention_mask: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None labels: typing.Optional[torch.Tensor] = None  ) → transformers.modeling_outputs.XVectorOutput or tuple(torch.FloatTensor)  Parameters   input_values (torch.FloatTensor of shape (batch_size, sequence_length)) —
Float values of input raw speech waveform. Values can be obtained by loading a .flac or .wav audio file
into an array of type List[float] or a numpy.ndarray, e.g. via the soundfile library (pip install soundfile). To prepare the array into input_values, the AutoProcessor should be used for padding and
conversion into a tensor of type torch.FloatTensor. See Wav2Vec2Processor.


call() for details.   attention_mask (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing convolution and attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?
attention_mask should only be passed if the corresponding processor has config.return_attention_mask == True. For all models whose processor has config.return_attention_mask == False, such as
wav2vec2-conformer-rel-pos-large,
attention_mask should not be passed to avoid degraded performance when doing batched inference. For
such models input_values should simply be padded with 0 and passed without attention_mask. Be aware
that these models also yield slightly different results depending on whether input_values is padded or
not.
   output_attentions (bool, optional) —
Whether or not to return the attentions tensor


s of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-Entropy).   Returns
transformers.modeling_outputs.XVectorOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.XVectorOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elem


ents depending on the configuration (Wav2Vec2ConformerConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification loss.
logits (torch.FloatTensor of shape (batch_size, config.xvector_output_dim)) — Classification hidden states before AMSoftmax.
embeddings (torch.FloatTensor of shape (batch_size, config.xvector_output_dim)) — Utterance embeddings used for vector similarity-based retrieval.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True


 is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The Wav2Vec2ConformerForXVector forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoFeatureExtractor, Wav2Vec2ConformerForXVector
>>> from datasets import load_dataset
>>> import torch
>>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
>>> dataset = dataset.sort("id")
>>> sampling_r


ate = dataset.features["audio"].sampling_rate
>>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-conformer-rope-large-960h-ft")
>>> model = Wav2Vec2ConformerForXVector.from_pretrained("facebook/wav2vec2-conformer-rope-large-960h-ft")
>>> # audio file is decoded on the fly
>>> inputs = feature_extractor(
...     [d["array"] for d in dataset[:2]["audio"]], sampling_rate=sampling_rate, return_tensors="pt", padding=True
... )
>>> with torch.no_grad():
...     embeddings = model(**inputs).embeddings
>>> embeddings = torch.nn.functional.normalize(embeddings, dim=-1).cpu()
>>> # the resulting embeddings can be used for cosine similarity-based retrieval
>>> cosine_sim = torch.nn.CosineSimilarity(dim=-1)
>>> similarity = cosine_sim(embeddings[0], embeddings[1])
>>> threshold = 0.7  # the optimal threshold is dataset-dependent
>>> if similarity < threshold:
...     print("Speakers are not the s


class transformers.Wav2Vec2ConformerForPreTraining  < source > ( config: Wav2Vec2ConformerConfig  )   Parameters   config (Wav2Vec2ConformerConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Wav2Vec2Conformer Model with a quantizer and VQ head on top.
Wav2Vec2Conformer was proposed in wav2vec 2.0: A Framework for Self-Supervised Learning of Speech
Representations by Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael
Auli. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving etc.). This model is a PyTorch nn.Module sub-class. Use it as a
regular PyTorch Module and refer to the PyTorch documentation for all matter rela


ted to general usage and behavior.  forward  < source > ( input_values: typing.Optional[torch.Tensor] attention_mask: typing.Optional[torch.Tensor] = None mask_time_indices: typing.Optional[torch.BoolTensor] = None sampled_negative_indices: typing.Optional[torch.BoolTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.wav2vec2_conformer.modeling_wav2vec2_conformer.Wav2Vec2ConformerForPreTrainingOutput or tuple(torch.FloatTensor)  Parameters   input_values (torch.FloatTensor of shape (batch_size, sequence_length)) —
Float values of input raw speech waveform. Values can be obtained by loading a .flac or .wav audio file
into an array of type List[float] or a numpy.ndarray, e.g. via the soundfile library (pip install soundfile). To prepare the array into input_values, the AutoProcessor should be used for padding and


conversion into a tensor of type torch.FloatTensor. See Wav2Vec2Processor.call() for details.   attention_mask (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing convolution and attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?
attention_mask should only be passed if the corresponding processor has config.return_attention_mask == True. For all models whose processor has config.return_attention_mask == False, such as
wav2vec2-conformer-rel-pos-large,
attention_mask should not be passed to avoid degraded performance when doing batched inference. For
such models input_values should simply be padded with 0 and passed without attention_mask. Be aware
that these models also yield slightly different results depending on whether input_values is padded or
not.
   output_attentions (bool, optional


) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   mask_time_indices (torch.BoolTensor of shape (batch_size, sequence_length), optional) —
Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict
masked extracted features in config.proj_codevector_dim space.   sampled_negative_indices (torch.BoolTensor of shape (batch_size, sequence_length, num_negatives), optional) —
Indices indicating which quantized target vectors are used as negative sampled vectors in contrastive loss.
Required input for pre-training.   Returns
transformers.models.wav2vec2_conformer.mod


eling_wav2vec2_conformer.Wav2Vec2ConformerForPreTrainingOutput or tuple(torch.FloatTensor)

A transformers.models.wav2vec2_conformer.modeling_wav2vec2_conformer.Wav2Vec2ConformerForPreTrainingOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (Wav2Vec2ConformerConfig) and inputs.
loss (optional, returned when sample_negative_indices are passed, torch.FloatTensor of shape (1,)) — Total loss as the sum of the contrastive loss (L_m) and the diversity loss (L_d) as stated in the official
paper . (classification) loss.
projected_states (torch.FloatTensor of shape (batch_size, sequence_length, config.proj_codevector_dim)) — Hidden-states of the model projected to config.proj_codevector_dim that can be used to predict the masked
projected quantized states.
projected_quantized_states (torch.FloatTensor of shape (batch_size, sequence


_length, config.proj_codevector_dim)) — Quantized extracted feature vectors projected to config.proj_codevector_dim representing the positive
target vectors for contrastive loss.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
contrastive_loss


 (optional, returned when sample_negative_indices are passed, torch.FloatTensor of shape (1,)) — The contrastive loss (L_m) as stated in the official paper .
diversity_loss (optional, returned when sample_negative_indices are passed, torch.FloatTensor of shape (1,)) — The diversity loss (L_d) as stated in the official paper .
  The Wav2Vec2ConformerForPreTraining forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> import torch
>>> from transformers import AutoFeatureExtractor, Wav2Vec2ConformerForPreTraining
>>> from transformers.models.wav2vec2_conformer.modeling_wav2vec2_conformer import (
...     _compute_mask_indices,
...     _sample_negative_indices,
... )


>>> from datasets import load_dataset
>>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-conformer-rel-pos-large")
>>> model = Wav2Vec2ConformerForPreTraining.from_pretrained("facebook/wav2vec2-conformer-rel-pos-large")
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> input_values = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt").input_values  # Batch size 1
>>> # compute masked indices
>>> batch_size, raw_sequence_length = input_values.shape
>>> sequence_length = model._get_feat_extract_output_lengths(raw_sequence_length).item()
>>> mask_time_indices = _compute_mask_indices(
...     shape=(batch_size, sequence_length), mask_prob=0.2, mask_length=2
... )
>>> sampled_negative_indices = _sample_negative_indices(
...     features_shape=(batch_size, sequence_length),
...     num_negatives=model.config.num_negatives,
...     mask_time_indices=mask_


time_indices,
... )
>>> mask_time_indices = torch.tensor(data=mask_time_indices, device=input_values.device, dtype=torch.long)
>>> sampled_negative_indices = torch.tensor(
...     data=sampled_negative_indices, device=input_values.device, dtype=torch.long
... )
>>> with torch.no_grad():
...     outputs = model(input_values, mask_time_indices=mask_time_indices)
>>> # compute cosine similarity between predicted (=projected_states) and target (=projected_quantized_states)
>>> cosine_sim = torch.cosine_similarity(outputs.projected_states, outputs.projected_quantized_states, dim=-1)
>>> # show that cosine similarity is much higher than random
>>> cosine_sim[mask_time_indices.to(torch.bool)].mean() > 0.5
tensor(True)
>>> # for contrastive loss training model should be put into train mode
>>> model = model.train()
>>> loss = model(
...     input_values, mask_time_indices=mask_time_indices, sampled_negative_indices=sampled_negative_indices
... )


Overview

The Informer model was proposed in Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting by Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang.
This method introduces a Probabilistic Attention mechanism to select the “active” queries rather than the “lazy” queries and provides a sparse Transformer thus mitigating the quadratic compute and memory requirements of vanilla attention.
The abstract from the paper is the following:
Many real-world applications require the prediction of long sequence time-series, such as electricity consumption planning. Long sequence time-series forecasting (LSTF) demands a high prediction capacit


Overview

y of the model, which is the ability to capture precise long-range dependency coupling between output and input efficiently. Recent studies have shown the potential of Transformer to increase the prediction capacity. However, there are several severe issues with Transformer that prevent it from being directly applicable to LSTF, including quadratic time complexity, high memory usage, and inherent limitation of the encoder-decoder architecture. To address these issues, we design an efficient transformer-based model for LSTF, named Informer, with three distinctive characteristics: (i) a ProbSparse self-attention mechanism, which achieves O(L logL) in time complexity and memory usage, and has co


Overview

mparable performance on sequences’ dependency alignment. (ii) the self-attention distilling highlights dominating attention by halving cascading layer input, and efficiently handles extreme long input sequences. (iii) the generative style decoder, while conceptually simple, predicts the long time-series sequences at one forward operation rather than a step-by-step way, which drastically improves the inference speed of long-sequence predictions. Extensive experiments on four large-scale datasets demonstrate that Informer significantly outperforms existing methods and provides a new solution to the LSTF problem.
This model was contributed by elisim and kashif.
The original code can be found her


Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started. If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we’ll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.
Check out the Informer blog-post in HuggingFace blog: Multivariate Probabilistic Time Series Forecasting with Informer


InformerConfig

class transformers.InformerConfig  < source > ( prediction_length: typing.Optional[int] = None context_length: typing.Optional[int] = None distribution_output: str = 'student_t' loss: str = 'nll' input_size: int = 1 lags_sequence: typing.List[int] = None scaling: typing.Union[str, bool, NoneType] = 'mean' num_dynamic_real_features: int = 0 num_static_real_features: int = 0 num_static_categorical_features: int = 0 num_time_features: int = 0 cardinality: typing.Optional[typing.List[int]] = None embedding_dimension: typing.Optional[typing.List[int]] = None d_model: int = 64 encoder_ffn_dim: int = 32 decoder_ffn_dim: int = 32 encoder_attention_heads: int = 2 decoder_attention_heads: int = 2 encoder_layers: int = 2 decoder_layers: int = 2 is_encoder_decoder: bool = True activation_function: str = 'gelu' dropout: float = 0.05 encoder_layerdrop: float = 0.1 decoder_layerdrop: float = 0.1 attention_dropout: float = 0.1 activation_dropout: float = 0.1 num_parallel_samples: int = 100 init_std:


InformerConfig

 float = 0.02 use_cache = True attention_type: str = 'prob' sampling_factor: int = 5 distil: bool = True **kwargs  )   Parameters   prediction_length (int) —
The prediction length for the decoder. In other words, the prediction horizon of the model. This value is
typically dictated by the dataset and we recommend to set it appropriately.   context_length (int, optional, defaults to prediction_length) —
The context length for the encoder. If None, the context length will be the same as the
prediction_length.   distribution_output (string, optional, defaults to "student_t") —
The distribution emission head for the model. Could be either “student_t”, “normal” or “negative_binomial”.   loss (string, optional, defaults to "nll") —
The loss function for the model corresponding to the distribution_output head. For parametric
distributions it is the negative log likelihood (nll) - which currently is the only supported one.   input_size (int, optional, defaults to 1) —
The size of the target


InformerConfig

variable which by default is 1 for univariate targets. Would be > 1 in case of
multivariate targets.   scaling (string or bool, optional defaults to "mean") —
Whether to scale the input targets via “mean” scaler, “std” scaler or no scaler if None. If True, the
scaler is set to “mean”.   lags_sequence (list[int], optional, defaults to [1, 2, 3, 4, 5, 6, 7]) —
The lags of the input time series as covariates often dictated by the frequency of the data. Default is
[1, 2, 3, 4, 5, 6, 7] but we recommend to change it based on the dataset appropriately.   num_time_features (int, optional, defaults to 0) —
The number of time features in the input time series.   num_dynamic_real_features (int, optional, defaults to 0) —
The number of dynamic real valued features.   num_static_categorical_features (int, optional, defaults to 0) —
The number of static categorical features.   num_static_real_features (int, optional, defaults to 0) —
The number of static real valued features.   cardinality (list[


InformerConfig

int], optional) —
The cardinality (number of different values) for each of the static categorical features. Should be a list
of integers, having the same length as num_static_categorical_features. Cannot be None if
num_static_categorical_features is > 0.   embedding_dimension (list[int], optional) —
The dimension of the embedding for each of the static categorical features. Should be a list of integers,
having the same length as num_static_categorical_features. Cannot be None if
num_static_categorical_features is > 0.   d_model (int, optional, defaults to 64) —
Dimensionality of the transformer layers.   encoder_layers (int, optional, defaults to 2) —
Number of encoder layers.   decoder_layers (int, optional, defaults to 2) —
Number of decoder layers.   encoder_attention_heads (int, optional, defaults to 2) —
Number of attention heads for each attention layer in the Transformer encoder.   decoder_attention_heads (int, optional, defaults to 2) —
Number of attention heads for each atte


InformerConfig

ntion layer in the Transformer decoder.   encoder_ffn_dim (int, optional, defaults to 32) —
Dimension of the “intermediate” (often named feed-forward) layer in encoder.   decoder_ffn_dim (int, optional, defaults to 32) —
Dimension of the “intermediate” (often named feed-forward) layer in decoder.   activation_function (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and decoder. If string, "gelu" and
"relu" are supported.   dropout (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the encoder, and decoder.   encoder_layerdrop (float, optional, defaults to 0.1) —
The dropout probability for the attention and fully connected layers for each encoder layer.   decoder_layerdrop (float, optional, defaults to 0.1) —
The dropout probability for the attention and fully connected layers for each decoder layer.   attention_dropout (float, optional, defaults to 0.1) —
The dropout


InformerConfig

probability for the attention probabilities.   activation_dropout (float, optional, defaults to 0.1) —
The dropout probability used between the two layers of the feed-forward networks.   num_parallel_samples (int, optional, defaults to 100) —
The number of samples to generate in parallel for each time step of inference.   init_std (float, optional, defaults to 0.02) —
The standard deviation of the truncated normal weight initialization distribution.   use_cache (bool, optional, defaults to True) —
Whether to use the past key/values attentions (if applicable to the model) to speed up decoding.   attention_type (str, optional, defaults to “prob”) —
Attention used in encoder. This can be set to “prob” (Informer’s ProbAttention) or “full” (vanilla
transformer’s canonical self-attention).   sampling_factor (int, optional, defaults to 5) —
ProbSparse sampling factor (only makes affect when attention_type=“prob”). It is used to control the
reduced query matrix (Q_reduce) input length.   dis


InformerConfig

til (bool, optional, defaults to True) —
Whether to use distilling in encoder.    This is the configuration class to store the configuration of an InformerModel. It is used to instantiate an
Informer model according to the specified arguments, defining the model architecture. Instantiating a configuration
with the defaults will yield a similar configuration to that of the Informer
huggingface/informer-tourism-monthly architecture. Configuration objects inherit from PretrainedConfig can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import InformerConfig, InformerModel
>>> # Initializing an Informer configuration with 12 time steps for prediction
>>> configuration = InformerConfig(prediction_length=12)
>>> # Randomly initializing a model (with random weights) from the configuration
>>> model = InformerModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.


InformerModel

class transformers.InformerModel  < source > ( config: InformerConfig  )   Parameters   config (TimeSeriesTransformerConfig) —
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
from_pretrained() method to load the model weights.    The bare Informer Model outputting raw hidden-states without any specific head on top.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( past_values: Tensor past_time_features: Tensor past_observed_mask: Tensor static_categorical_featur


InformerModel

es: typing.Optional[torch.Tensor] = None static_real_features: typing.Optional[torch.Tensor] = None future_values: typing.Optional[torch.Tensor] = None future_time_features: typing.Optional[torch.Tensor] = None decoder_attention_mask: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.Tensor] = None decoder_head_mask: typing.Optional[torch.Tensor] = None cross_attn_head_mask: typing.Optional[torch.Tensor] = None encoder_outputs: typing.Optional[typing.List[torch.FloatTensor]] = None past_key_values: typing.Optional[typing.List[torch.FloatTensor]] = None output_hidden_states: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None use_cache: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.Seq2SeqTSModelOutput or tuple(torch.FloatTensor)  Parameters   past_values (torch.FloatTensor of shape (batch_size, sequence_length) or (batch_size, sequence_length, input_size)) —
Past v


InformerModel

alues of the time series, that serve as context in order to predict the future. The sequence size of
this tensor must be larger than the context_length of the model, since the model will use the larger size
to construct lag features, i.e. additional values from the past which are added in order to serve as “extra
context”.
The sequence_length here is equal to config.context_length + max(config.lags_sequence), which if no
lags_sequence is configured, is equal to config.context_length + 7 (as by default, the largest
look-back index in config.lags_sequence is 7). The property _past_length returns the actual length of
the past.
The past_values is what the Transformer encoder gets as input (with optional additional features, such as
static_categorical_features, static_real_features, past_time_features and lags).
Optionally, missing values need to be replaced with zeros and indicated via the past_observed_mask.
For multivariate time series, the input_size > 1 dimension is requi


InformerModel

red and corresponds to the number of
variates in the time series per time step.   past_time_features (torch.FloatTensor of shape (batch_size, sequence_length, num_features)) —
Required time features, which the model internally will add to past_values. These could be things like
“month of year”, “day of the month”, etc. encoded as vectors (for instance as Fourier features). These
could also be so-called “age” features, which basically help the model know “at which point in life” a
time-series is. Age features have small values for distant past time steps and increase monotonically the
more we approach the current time step. Holiday features are also a good example of time features.
These features serve as the “positional encodings” of the inputs. So contrary to a model like BERT, where
the position encodings are learned from scratch internally as parameters of the model, the Time Series
Transformer requires to provide additional time features. The Time Series Transformer o


InformerModel

nly learns
additional embeddings for static_categorical_features.
Additional dynamic real covariates can be concatenated to this tensor, with the caveat that these features
must but known at prediction time.
The num_features here is equal to config.num_time_features+config.num_dynamic_real_features`.   past_observed_mask (torch.BoolTensor of shape (batch_size, sequence_length) or (batch_size, sequence_length, input_size), optional) —
Boolean mask to indicate which past_values were observed and which were missing. Mask values selected in
[0, 1]:
1 for values that are observed,
0 for values that are missing (i.e. NaNs that were replaced by zeros).
   static_categorical_features (torch.LongTensor of shape (batch_size, number of static categorical features), optional) —
Optional static categorical features for which the model will learn an embedding, which it will add to the
values of the time series.
Static categorical features are features which have the same value for all


InformerModel

time steps (static over time).
A typical example of a static categorical feature is a time series ID.   static_real_features (torch.FloatTensor of shape (batch_size, number of static real features), optional) —
Optional static real features which the model will add to the values of the time series.
Static real features are features which have the same value for all time steps (static over time).
A typical example of a static real feature is promotion information.   future_values (torch.FloatTensor of shape (batch_size, prediction_length) or (batch_size, prediction_length, input_size), optional) —
Future values of the time series, that serve as labels for the model. The future_values is what the
Transformer needs during training to learn to output, given the past_values.
The sequence length here is equal to prediction_length.
See the demo notebook and code snippets for details.
Optionally, during training any missing values need to be replaced with zeros and indicated via


InformerModel

the
future_observed_mask.
For multivariate time series, the input_size > 1 dimension is required and corresponds to the number of
variates in the time series per time step.   future_time_features (torch.FloatTensor of shape (batch_size, prediction_length, num_features)) —
Required time features for the prediction window, which the model internally will add to future_values.
These could be things like “month of year”, “day of the month”, etc. encoded as vectors (for instance as
Fourier features). These could also be so-called “age” features, which basically help the model know “at
which point in life” a time-series is. Age features have small values for distant past time steps and
increase monotonically the more we approach the current time step. Holiday features are also a good example
of time features.
These features serve as the “positional encodings” of the inputs. So contrary to a model like BERT, where
the position encodings are learned from scratch internally as par


InformerModel

ameters of the model, the Time Series
Transformer requires to provide additional time features. The Time Series Transformer only learns
additional embeddings for static_categorical_features.
Additional dynamic real covariates can be concatenated to this tensor, with the caveat that these features
must but known at prediction time.
The num_features here is equal to config.num_time_features+config.num_dynamic_real_features`.   future_observed_mask (torch.BoolTensor of shape (batch_size, sequence_length) or (batch_size, sequence_length, input_size), optional) —
Boolean mask to indicate which future_values were observed and which were missing. Mask values selected
in [0, 1]:
1 for values that are observed,
0 for values that are missing (i.e. NaNs that were replaced by zeros).
This mask is used to filter out missing values for the final loss calculation.   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on cer


InformerModel

tain token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   decoder_attention_mask (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Mask to avoid performing attention on certain token indices. By default, a causal mask will be used, to
make sure the model can only look at previous inputs in order to predict the future.   head_mask (torch.Tensor of shape (encoder_layers, encoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   decoder_head_mask (torch.Tensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   cross_attn_head_m


InformerModel

ask (torch.Tensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the cross-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   encoder_outputs (tuple(tuple(torch.FloatTensor), optional) —
Tuple consists of last_hidden_state, hidden_states (optional) and attentions (optional)
last_hidden_state of shape (batch_size, sequence_length, hidden_size) (optional) is a sequence of
hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.   past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) —
Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, em


InformerModel

bed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   ou


InformerModel

tput_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.Seq2SeqTSModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.Seq2SeqTSModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (InformerConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the decoder of the model.
If past_key_values is used only the last hidden-state of


InformerModel

 the sequences of shape (batch_size, 1, hidden_size) is output.
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_le


InformerModel

ngth, hidden_size).
Hidden-states of the decoder at the output of each layer plus the optional initial embedding outputs.
decoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
encoder_last_hidden_state (torch.FloatTensor of shape (batch_


InformerModel

size, sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the optional initial embedding outputs.
encoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
loc (torch.Fl


InformerModel

oatTensor of shape (batch_size,) or (batch_size, input_size), optional) — Shift values of each time series’ context window which is used to give the model inputs of the same
magnitude and then used to shift back to the original magnitude.
scale (torch.FloatTensor of shape (batch_size,) or (batch_size, input_size), optional) — Scaling values of each time series’ context window which is used to give the model inputs of the same
magnitude and then used to rescale back to the original magnitude.
static_features (torch.FloatTensor of shape (batch_size, feature size), optional) — Static features of each time series’ in a batch which are copied to the covariates at inference time.
  The InformerModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the lat


InformerModel

ter silently ignores them.  Examples:   Copied >>> from huggingface_hub import hf_hub_download
>>> import torch
>>> from transformers import InformerModel
>>> file = hf_hub_download(
...     repo_id="hf-internal-testing/tourism-monthly-batch", filename="train-batch.pt", repo_type="dataset"
... )
>>> batch = torch.load(file)
>>> model = InformerModel.from_pretrained("huggingface/informer-tourism-monthly")
>>> # during training, one provides both past and future values
>>> # as well as possible additional features
>>> outputs = model(
...     past_values=batch["past_values"],
...     past_time_features=batch["past_time_features"],
...     past_observed_mask=batch["past_observed_mask"],
...     static_categorical_features=batch["static_categorical_features"],
...     static_real_features=batch["static_real_features"],
...     future_values=batch["future_values"],
...     future_time_features=batch["future_time_features"],
... )
>>> last_hidden_state = outputs.last_hidden_sta


InformerForPrediction

class transformers.InformerForPrediction  < source > ( config: InformerConfig  )   Parameters   config (TimeSeriesTransformerConfig) —
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
from_pretrained() method to load the model weights.    The Informer Model with a distribution head on top for time-series forecasting.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( past_values: Tensor past_time_features: Tensor past_observed_mask: Tensor static


InformerForPrediction

_categorical_features: typing.Optional[torch.Tensor] = None static_real_features: typing.Optional[torch.Tensor] = None future_values: typing.Optional[torch.Tensor] = None future_time_features: typing.Optional[torch.Tensor] = None future_observed_mask: typing.Optional[torch.Tensor] = None decoder_attention_mask: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.Tensor] = None decoder_head_mask: typing.Optional[torch.Tensor] = None cross_attn_head_mask: typing.Optional[torch.Tensor] = None encoder_outputs: typing.Optional[typing.List[torch.FloatTensor]] = None past_key_values: typing.Optional[typing.List[torch.FloatTensor]] = None output_hidden_states: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None use_cache: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.Seq2SeqTSModelOutput or tuple(torch.FloatTensor)  Parameters   past_values (torch.FloatTens


InformerForPrediction

or of shape (batch_size, sequence_length) or (batch_size, sequence_length, input_size)) —
Past values of the time series, that serve as context in order to predict the future. The sequence size of
this tensor must be larger than the context_length of the model, since the model will use the larger size
to construct lag features, i.e. additional values from the past which are added in order to serve as “extra
context”.
The sequence_length here is equal to config.context_length + max(config.lags_sequence), which if no
lags_sequence is configured, is equal to config.context_length + 7 (as by default, the largest
look-back index in config.lags_sequence is 7). The property _past_length returns the actual length of
the past.
The past_values is what the Transformer encoder gets as input (with optional additional features, such as
static_categorical_features, static_real_features, past_time_features and lags).
Optionally, missing values need to be replaced with z