MarkupLMProcessor

class transformers.MarkupLMProcessor  < source > ( *args **kwargs  )   Parameters   feature_extractor (MarkupLMFeatureExtractor) —
An instance of MarkupLMFeatureExtractor. The feature extractor is a required input.   tokenizer (MarkupLMTokenizer or MarkupLMTokenizerFast) —
An instance of MarkupLMTokenizer or MarkupLMTokenizerFast. The tokenizer is a required input.   parse_html (bool, optional, defaults to True) —
Whether or not to use MarkupLMFeatureExtractor to parse HTML strings into nodes and corresponding xpaths.    Constructs a MarkupLM processor which combines a MarkupLM feature extractor and a MarkupLM tokenizer into a single
processor. MarkupLMProcessor offers all the functionalities you need to prepare data


MarkupLMProcessor

for the model. It first uses MarkupLMFeatureExtractor to extract nodes and corresponding xpaths from one or more HTML strings.
Next, these are provided to MarkupLMTokenizer or MarkupLMTokenizerFast, which turns them into token-level
input_ids, attention_mask, token_type_ids, xpath_tags_seq and xpath_subs_seq.  __call__  < source > ( html_strings = None nodes = None xpaths = None node_labels = None questions = None add_special_tokens: bool = True padding: typing.Union[bool, str, transformers.utils.generic.PaddingStrategy] = False truncation: typing.Union[bool, str, transformers.tokenization_utils_base.TruncationStrategy] = None max_length: typing.Optional[int] = None stride: int = 0 pad_to_multiple_of: typing.Optional[


MarkupLMProcessor

int] = None return_token_type_ids: typing.Optional[bool] = None return_attention_mask: typing.Optional[bool] = None return_overflowing_tokens: bool = False return_special_tokens_mask: bool = False return_offsets_mapping: bool = False return_length: bool = False verbose: bool = True return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None **kwargs  )    This method first forwards the html_strings argument to call(). Next, it
passes the nodes and xpaths along with the additional arguments to __call__() and
returns the output. Optionally, one can also provide a text argument which is passed along as first sequence. Please refer to the docstring of the above two methods for more informatio


MarkupLMModel

class transformers.MarkupLMModel  < source > ( config add_pooling_layer = True  )   Parameters   config (MarkupLMConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare MarkupLM Model transformer outputting raw hidden-states without any specific head on top.
This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None xpath_tags_seq: typing.Optional[torch.LongTensor] = None xpath_subs_seq: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None position_i


MarkupLMModel

ds: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   xpath_tags_seq (torch.LongTensor of shape (batch_size, sequence_length, config.max_depth), optional) —
Tag IDs for each token in the input sequence, padded up to config.max_depth.   xpath_subs_seq (torch.LongTensor of shape (batch_size, sequence_length, config.max_depth),


MarkupLMModel

optional) —
Subscript IDs for each token in the input sequence, padded up to config.max_depth.   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]: 1 for
tokens that are NOT MASKED, 0 for MASKED tokens.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]: 0 corresponds to a sentence A token, 1 corresponds to a sentence B token
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (n


MarkupLMModel

um_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]: 1
indicates the head is not masked, 0 indicates the head is masked.   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
If set to True, the attentions tensors of all attention layers are returned. See attentions under
returned tensors for more detail.   output_hidden_states (bool, optional) —
If set to True, the hidden states of all layers are returned. See hidden_states under returned tensors
for more detail.   return_dict (bool, optional) —
If set to True, the model will return a


MarkupLMModel

ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (MarkupLMConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing
through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns
the classification token after processing through a linear layer and a tanh activation funct


MarkupLMModel

ion. The linear
layer weights are trained from the next sentence prediction (classification) objective during pretraining.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
cross_attentions (tuple(torch.F


MarkupLMModel

loatTensor), optional, returned when output_attentions=True and config.add_cross_attention=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and optionally if
config.is_encoder_decoder=True 2 additional tensors of shape (batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and option


MarkupLMModel

ally if
config.is_encoder_decoder=True in the cross-attention blocks) that can be used (see past_key_values
input) to speed up sequential decoding.
  The MarkupLMModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoProcessor, MarkupLMModel
>>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base")
>>> model = MarkupLMModel.from_pretrained("microsoft/markuplm-base")
>>> html_string = "<html> <head> <title>Page Title</title> </head> </html>"
>>> encoding = processor(html_string, return_tensors="pt")
>>> outputs = model(**encoding)
>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1, 4,


MarkupLMForSequenceClassification

class transformers.MarkupLMForSequenceClassification  < source > ( config  )   Parameters   config (MarkupLMConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    MarkupLM Model transformer with a sequence classification/regression head on top (a linear layer on top of the
pooled output) e.g. for GLUE tasks. This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None xpath_tags_seq: typing.Optional[torch.Tensor] = None xpath_subs_seq: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None token_type_ids: typin


MarkupLMForSequenceClassification

g.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.SequenceClassifierOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   xpath_tags_seq (torch.LongTensor of shape (batch_size, sequence_length, config.max_depth), optional) —
Tag IDs for each token in the input sequence, padded up to config.max_depth.   xpath_subs_seq (to


MarkupLMForSequenceClassification

rch.LongTensor of shape (batch_size, sequence_length, config.max_depth), optional) —
Subscript IDs for each token in the input sequence, padded up to config.max_depth.   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]: 1 for
tokens that are NOT MASKED, 0 for MASKED tokens.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]: 0 corresponds to a sentence A token, 1 corresponds to a sentence B token
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embed


MarkupLMForSequenceClassification

dings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]: 1
indicates the head is not masked, 0 indicates the head is masked.   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
If set to True, the attentions tensors of all attention layers are returned. See attentions under
returned tensors for more detail.   output_hidden_states (bool, optional) —
If set to True, the hidden states of all layers are returned. See hidden_states


MarkupLMForSequenceClassification

under returned tensors
for more detail.   return_dict (bool, optional) —
If set to True, the model will return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-Entropy).   Returns
transformers.modeling_outputs.SequenceClassifierOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.SequenceClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (MarkupLMConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or regress


MarkupLMForSequenceClassification

ion if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute


MarkupLMForSequenceClassification

 the weighted average in the self-attention
heads.
  The MarkupLMForSequenceClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoProcessor, AutoModelForSequenceClassification
>>> import torch
>>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base")
>>> model = AutoModelForSequenceClassification.from_pretrained("microsoft/markuplm-base", num_labels=7)
>>> html_string = "<html> <head> <title>Page Title</title> </head> </html>"
>>> encoding = processor(html_string, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**encoding)
>>> loss = outputs.loss
>>> logits = outputs.l


MarkupLMForTokenClassification

class transformers.MarkupLMForTokenClassification  < source > ( config  )   Parameters   config (MarkupLMConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    MarkupLM Model with a token_classification head on top.
This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None xpath_tags_seq: typing.Optional[torch.Tensor] = None xpath_subs_seq: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor]


MarkupLMForTokenClassification

 = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.MaskedLMOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   xpath_tags_seq (torch.LongTensor of shape (batch_size, sequence_length, config.max_depth), optional) —
Tag IDs for each token in the input sequence, padded up to config.max_depth.   xpath_subs_seq (torch.LongTensor of shape (batch_size, sequence_length, config.max_d


MarkupLMForTokenClassification

epth), optional) —
Subscript IDs for each token in the input sequence, padded up to config.max_depth.   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]: 1 for
tokens that are NOT MASKED, 0 for MASKED tokens.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]: 0 corresponds to a sentence A token, 1 corresponds to a sentence B token
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask


MarkupLMForTokenClassification

(torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]: 1
indicates the head is not masked, 0 indicates the head is masked.   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
If set to True, the attentions tensors of all attention layers are returned. See attentions under
returned tensors for more detail.   output_hidden_states (bool, optional) —
If set to True, the hidden states of all layers are returned. See hidden_states under returned tensors
for m


MarkupLMForTokenClassification

ore detail.   return_dict (bool, optional) —
If set to True, the model will return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Labels for computing the token classification loss. Indices should be in [0, ..., config.num_labels - 1].   Returns
transformers.modeling_outputs.MaskedLMOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.MaskedLMOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (MarkupLMConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Masked language modeling (MLM) loss.
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token


MarkupLMForTokenClassification

 before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The MarkupLMForTokenClassification forward method, overrides the __call__ special method. Altho


MarkupLMForTokenClassification

ugh the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoProcessor, AutoModelForTokenClassification
>>> import torch
>>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base")
>>> processor.parse_html = False
>>> model = AutoModelForTokenClassification.from_pretrained("microsoft/markuplm-base", num_labels=7)
>>> nodes = ["hello", "world"]
>>> xpaths = ["/html/body/div/li[1]/div/span", "/html/body/div/li[1]/div/span"]
>>> node_labels = [1, 2]
>>> encoding = processor(nodes=nodes, xpaths=xpaths, node_labels=node_labels, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**encoding)
>>> loss = outputs.loss
>>> logits = outputs.logits


MarkupLMForQuestionAnswering

class transformers.MarkupLMForQuestionAnswering  < source > ( config  )   Parameters   config (MarkupLMConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    MarkupLM Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear
layers on top of the hidden-states output to compute span start logits and span end logits). This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None xpath_tags_seq: typing.Optional[torch.Tensor] = None xpath_subs_seq: typing.Optional[torch.Tensor] = None atte


MarkupLMForQuestionAnswering

ntion_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None start_positions: typing.Optional[torch.Tensor] = None end_positions: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.QuestionAnsweringModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   xpath_tags_seq (torch.LongTensor of shape (batch_si


MarkupLMForQuestionAnswering

ze, sequence_length, config.max_depth), optional) —
Tag IDs for each token in the input sequence, padded up to config.max_depth.   xpath_subs_seq (torch.LongTensor of shape (batch_size, sequence_length, config.max_depth), optional) —
Subscript IDs for each token in the input sequence, padded up to config.max_depth.   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]: 1 for
tokens that are NOT MASKED, 0 for MASKED tokens.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]: 0 corresponds to a sentence A token, 1 corresponds to a sentence B token
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_siz


MarkupLMForQuestionAnswering

e, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]: 1
indicates the head is not masked, 0 indicates the head is masked.   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
If set to True, the attentions tensors of all attention layers are returned.


MarkupLMForQuestionAnswering

See attentions under
returned tensors for more detail.   output_hidden_states (bool, optional) —
If set to True, the hidden states of all layers are returned. See hidden_states under returned tensors
for more detail.   return_dict (bool, optional) —
If set to True, the model will return a ModelOutput instead of a plain tuple.   start_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the start of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
are not taken into account for computing the loss.   end_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the end of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outsid


MarkupLMForQuestionAnswering

e of the sequence
are not taken into account for computing the loss.   Returns
transformers.modeling_outputs.QuestionAnsweringModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.QuestionAnsweringModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (MarkupLMConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Total span extraction loss is the sum of a Cross-Entropy for the start and end positions.
start_logits (torch.FloatTensor of shape (batch_size, sequence_length)) — Span-start scores (before SoftMax).
end_logits (torch.FloatTensor of shape (batch_size, sequence_length)) — Span-end scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config


MarkupLMForQuestionAnswering

.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The MarkupLMForQuestionAnswering forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this sin


MarkupLMForQuestionAnswering

ce the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoProcessor, MarkupLMForQuestionAnswering
>>> import torch
>>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base-finetuned-websrc")
>>> model = MarkupLMForQuestionAnswering.from_pretrained("microsoft/markuplm-base-finetuned-websrc")
>>> html_string = "<html> <head> <title>My name is Niels</title> </head> </html>"
>>> question = "What's his name?"
>>> encoding = processor(html_string, questions=question, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**encoding)
>>> answer_start_index = outputs.start_logits.argmax()
>>> answer_end_index = outputs.end_logits.argmax()
>>> predict_answer_tokens = encoding.input_ids[0, answer_start_index : answer_end_index + 1]
>>> processor.decode(predict_answer_tokens).strip()
'Nie


Time Series Transformer

This is a recently introduced model so the API hasn’t been tested extensively. There may be some bugs or slight
breaking changes to fix it in the future. If you see something strange, file a Github Issue.


Overview

The Time Series Transformer model is a vanilla encoder-decoder Transformer for time series forecasting.
Tips:
Similar to other models in the library, TimeSeriesTransformerModel is the raw Transformer without any head on top, and TimeSeriesTransformerForPrediction
adds a distribution head on top of the former, which can be used for time-series forecasting. Note that this is a so-called probabilistic forecasting model, not a
point forecasting model. This means that the model learns a distribution, from which one can sample. The model doesn’t directly output values. TimeSeriesTransformerForPrediction consists of 2 blocks: an encoder, which takes a context_length of time series values as input (called past_values),
and a decoder, which predicts a prediction_length of time series values into the future (called future_values). During training, one needs to provide
pairs of (past_


Overview

values and future_values) to the model. In addition to the raw (past_values and future_values), one typically provides additional features to the model. These can be the following:past_time_features: temporal features which the model will add to past_values. These serve as “positional encodings” for the Transformer encoder.
Examples are “day of the month”, “month of the year”, etc. as scalar values (and then stacked together as a vector).
e.g. if a given time-series value was obtained on the 11th of August, then one could have [11, 8] as time feature vector (11 being “day of the month”, 8 being “month of the year”). future_time_features: temporal features which the model will add to future_values. These serve as “positional encodings” for the Transformer decoder.
Examples are “day of the month”, “month of the year”, etc. as scalar values (and then stacked together as a vect


Overview

or).
e.g. if a given time-series value was obtained on the 11th of August, then one could have [11, 8] as time feature vector (11 being “day of the month”, 8 being “month of the year”). static_categorical_features: categorical features which are static over time (i.e., have the same value for all past_values and future_values).
An example here is the store ID or region ID that identifies a given time-series.
Note that these features need to be known for ALL data points (also those in the future). static_real_features: real-valued features which are static over time (i.e., have the same value for all past_values and future_values).
An example here is the image representation of the product for which you have the time-series values (like the ResNet embedding of a “shoe” picture,
if your time-series is about the sales of shoes).
Note that these features need to be known for AL


Overview

L data points (also those in the future). The model is trained using “teacher-forcing”, similar to how a Transformer is trained for machine translation. This means that, during training, one shifts the
future_values one position to the right as input to the decoder, prepended by the last value of past_values. At each time step, the model needs to predict the
next target. So the set-up of training is similar to a GPT model for language, except that there’s no notion of decoder_start_token_id (we just use the last value
of the context as initial input for the decoder). At inference time, we give the final value of the past_values as input to the decoder. Next, we can sample from the model to make a prediction at the next time step,
which is then fed to the decoder in order to make the next prediction (also called autoregressive generation).
This model was contributed by kashi


Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started. If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we’ll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.
Check out the Time Series Transformer blog-post in HuggingFace blog: Probabilistic Time Series Forecasting with 🤗 Transformers


TimeSeriesTransformerConfig

class transformers.TimeSeriesTransformerConfig  < source > ( prediction_length: typing.Optional[int] = None context_length: typing.Optional[int] = None distribution_output: str = 'student_t' loss: str = 'nll' input_size: int = 1 lags_sequence: typing.List[int] = [1, 2, 3, 4, 5, 6, 7] scaling: typing.Union[str, bool, NoneType] = 'mean' num_dynamic_real_features: int = 0 num_static_categorical_features: int = 0 num_static_real_features: int = 0 num_time_features: int = 0 cardinality: typing.Optional[typing.List[int]] = None embedding_dimension: typing.Optional[typing.List[int]] = None encoder_ffn_dim: int = 32 decoder_ffn_dim: int = 32 encoder_attention_heads: int = 2 decoder_attention_heads: int = 2 encoder_layers: int = 2 decoder_layers: int = 2 is_encoder_decoder: bool = True activation_function: str = 'gelu' d_model: int = 64 dropout: float = 0.1 encoder_layerdrop: float = 0.1 decoder_layerdrop: float = 0.1 attention_dropout:


TimeSeriesTransformerConfig

 float = 0.1 activation_dropout: float = 0.1 num_parallel_samples: int = 100 init_std: float = 0.02 use_cache = True **kwargs  )   Parameters   prediction_length (int) —
The prediction length for the decoder. In other words, the prediction horizon of the model. This value is
typically dictated by the dataset and we recommend to set it appropriately.   context_length (int, optional, defaults to prediction_length) —
The context length for the encoder. If None, the context length will be the same as the
prediction_length.   distribution_output (string, optional, defaults to "student_t") —
The distribution emission head for the model. Could be either “student_t”, “normal” or “negative_binomial”.   loss (string, optional, defaults to "nll") —
The loss function for the model corresponding to the distribution_output head. For parametric
distributions it is the negative log likelihood (nll) - which currently is the only supported one.


TimeSeriesTransformerConfig

  input_size (int, optional, defaults to 1) —
The size of the target variable which by default is 1 for univariate targets. Would be > 1 in case of
multivariate targets.   scaling (string or bool, optional defaults to "mean") —
Whether to scale the input targets via “mean” scaler, “std” scaler or no scaler if None. If True, the
scaler is set to “mean”.   lags_sequence (list[int], optional, defaults to [1, 2, 3, 4, 5, 6, 7]) —
The lags of the input time series as covariates often dictated by the frequency of the data. Default is
[1, 2, 3, 4, 5, 6, 7] but we recommend to change it based on the dataset appropriately.   num_time_features (int, optional, defaults to 0) —
The number of time features in the input time series.   num_dynamic_real_features (int, optional, defaults to 0) —
The number of dynamic real valued features.   num_static_categorical_features (int, optional, defaults to 0) —
The number of static categorical feature


TimeSeriesTransformerConfig

s.   num_static_real_features (int, optional, defaults to 0) —
The number of static real valued features.   cardinality (list[int], optional) —
The cardinality (number of different values) for each of the static categorical features. Should be a list
of integers, having the same length as num_static_categorical_features. Cannot be None if
num_static_categorical_features is > 0.   embedding_dimension (list[int], optional) —
The dimension of the embedding for each of the static categorical features. Should be a list of integers,
having the same length as num_static_categorical_features. Cannot be None if
num_static_categorical_features is > 0.   d_model (int, optional, defaults to 64) —
Dimensionality of the transformer layers.   encoder_layers (int, optional, defaults to 2) —
Number of encoder layers.   decoder_layers (int, optional, defaults to 2) —
Number of decoder layers.   encoder_attention_heads (int, optional, defaults to


TimeSeriesTransformerConfig

 2) —
Number of attention heads for each attention layer in the Transformer encoder.   decoder_attention_heads (int, optional, defaults to 2) —
Number of attention heads for each attention layer in the Transformer decoder.   encoder_ffn_dim (int, optional, defaults to 32) —
Dimension of the “intermediate” (often named feed-forward) layer in encoder.   decoder_ffn_dim (int, optional, defaults to 32) —
Dimension of the “intermediate” (often named feed-forward) layer in decoder.   activation_function (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and decoder. If string, "gelu" and
"relu" are supported.   dropout (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the encoder, and decoder.   encoder_layerdrop (float, optional, defaults to 0.1) —
The dropout probability for the attention and fully connected layers for


TimeSeriesTransformerConfig

 each encoder layer.   decoder_layerdrop (float, optional, defaults to 0.1) —
The dropout probability for the attention and fully connected layers for each decoder layer.   attention_dropout (float, optional, defaults to 0.1) —
The dropout probability for the attention probabilities.   activation_dropout (float, optional, defaults to 0.1) —
The dropout probability used between the two layers of the feed-forward networks.   num_parallel_samples (int, optional, defaults to 100) —
The number of samples to generate in parallel for each time step of inference.   init_std (float, optional, defaults to 0.02) —
The standard deviation of the truncated normal weight initialization distribution.   use_cache (bool, optional, defaults to True) —
Whether to use the past key/values attentions (if applicable to the model) to speed up decoding.
Example —    This is the configuration class to store the configuration of a TimeSeriesTransformerMod


TimeSeriesTransformerConfig

el. It is used to
instantiate a Time Series Transformer model according to the specified arguments, defining the model architecture.
Instantiating a configuration with the defaults will yield a similar configuration to that of the Time Series
Transformer
huggingface/time-series-transformer-tourism-monthly
architecture. Configuration objects inherit from PretrainedConfig can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.    Copied >>> from transformers import TimeSeriesTransformerConfig, TimeSeriesTransformerModel
>>> # Initializing a Time Series Transformer configuration with 12 time steps for prediction
>>> configuration = TimeSeriesTransformerConfig(prediction_length=12)
>>> # Randomly initializing a model (with random weights) from the configuration
>>> model = TimeSeriesTransformerModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.


TimeSeriesTransformerModel

class transformers.TimeSeriesTransformerModel  < source > ( config: TimeSeriesTransformerConfig  )   Parameters   config (TimeSeriesTransformerConfig) —
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
from_pretrained() method to load the model weights.    The bare Time Series Transformer Model outputting raw hidden-states without any specific head on top.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( past_values: Tensor past_time_features: Tensor past_observed_mas


TimeSeriesTransformerModel

k: Tensor static_categorical_features: typing.Optional[torch.Tensor] = None static_real_features: typing.Optional[torch.Tensor] = None future_values: typing.Optional[torch.Tensor] = None future_time_features: typing.Optional[torch.Tensor] = None decoder_attention_mask: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.Tensor] = None decoder_head_mask: typing.Optional[torch.Tensor] = None cross_attn_head_mask: typing.Optional[torch.Tensor] = None encoder_outputs: typing.Optional[typing.List[torch.FloatTensor]] = None past_key_values: typing.Optional[typing.List[torch.FloatTensor]] = None output_hidden_states: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None use_cache: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.Seq2SeqTSModelOutput or tuple(torch.FloatTensor)  Parameters   past_values (torch.FloatTensor of shape (batch_size, sequence_length) or (batch_size, sequence_


TimeSeriesTransformerModel

length, input_size)) —
Past values of the time series, that serve as context in order to predict the future. The sequence size of
this tensor must be larger than the context_length of the model, since the model will use the larger size
to construct lag features, i.e. additional values from the past which are added in order to serve as “extra
context”.
The sequence_length here is equal to config.context_length + max(config.lags_sequence), which if no
lags_sequence is configured, is equal to config.context_length + 7 (as by default, the largest
look-back index in config.lags_sequence is 7). The property _past_length returns the actual length of
the past.
The past_values is what the Transformer encoder gets as input (with optional additional features, such as
static_categorical_features, static_real_features, past_time_features and lags).
Optionally, missing values need to be replaced with zeros and indicated via the past_observed_mask.
For multivariate time series, the input_size


TimeSeriesTransformerModel

 > 1 dimension is required and corresponds to the number of
variates in the time series per time step.   past_time_features (torch.FloatTensor of shape (batch_size, sequence_length, num_features)) —
Required time features, which the model internally will add to past_values. These could be things like
“month of year”, “day of the month”, etc. encoded as vectors (for instance as Fourier features). These
could also be so-called “age” features, which basically help the model know “at which point in life” a
time-series is. Age features have small values for distant past time steps and increase monotonically the
more we approach the current time step. Holiday features are also a good example of time features.
These features serve as the “positional encodings” of the inputs. So contrary to a model like BERT, where
the position encodings are learned from scratch internally as parameters of the model, the Time Series
Transformer requires to provide additional time features. The Time Ser


TimeSeriesTransformerModel

ies Transformer only learns
additional embeddings for static_categorical_features.
Additional dynamic real covariates can be concatenated to this tensor, with the caveat that these features
must but known at prediction time.
The num_features here is equal to config.num_time_features+config.num_dynamic_real_features`.   past_observed_mask (torch.BoolTensor of shape (batch_size, sequence_length) or (batch_size, sequence_length, input_size), optional) —
Boolean mask to indicate which past_values were observed and which were missing. Mask values selected in
[0, 1]:
1 for values that are observed,
0 for values that are missing (i.e. NaNs that were replaced by zeros).
   static_categorical_features (torch.LongTensor of shape (batch_size, number of static categorical features), optional) —
Optional static categorical features for which the model will learn an embedding, which it will add to the
values of the time series.
Static categorical features are features which have the same val


TimeSeriesTransformerModel

ue for all time steps (static over time).
A typical example of a static categorical feature is a time series ID.   static_real_features (torch.FloatTensor of shape (batch_size, number of static real features), optional) —
Optional static real features which the model will add to the values of the time series.
Static real features are features which have the same value for all time steps (static over time).
A typical example of a static real feature is promotion information.   future_values (torch.FloatTensor of shape (batch_size, prediction_length) or (batch_size, prediction_length, input_size), optional) —
Future values of the time series, that serve as labels for the model. The future_values is what the
Transformer needs during training to learn to output, given the past_values.
The sequence length here is equal to prediction_length.
See the demo notebook and code snippets for details.
Optionally, during training any missing values need to be replaced with zeros and indicated


TimeSeriesTransformerModel

 via the
future_observed_mask.
For multivariate time series, the input_size > 1 dimension is required and corresponds to the number of
variates in the time series per time step.   future_time_features (torch.FloatTensor of shape (batch_size, prediction_length, num_features)) —
Required time features for the prediction window, which the model internally will add to future_values.
These could be things like “month of year”, “day of the month”, etc. encoded as vectors (for instance as
Fourier features). These could also be so-called “age” features, which basically help the model know “at
which point in life” a time-series is. Age features have small values for distant past time steps and
increase monotonically the more we approach the current time step. Holiday features are also a good example
of time features.
These features serve as the “positional encodings” of the inputs. So contrary to a model like BERT, where
the position encodings are learned from scratch internally as para


TimeSeriesTransformerModel

meters of the model, the Time Series
Transformer requires to provide additional time features. The Time Series Transformer only learns
additional embeddings for static_categorical_features.
Additional dynamic real covariates can be concatenated to this tensor, with the caveat that these features
must but known at prediction time.
The num_features here is equal to config.num_time_features+config.num_dynamic_real_features`.   future_observed_mask (torch.BoolTensor of shape (batch_size, sequence_length) or (batch_size, sequence_length, input_size), optional) —
Boolean mask to indicate which future_values were observed and which were missing. Mask values selected
in [0, 1]:
1 for values that are observed,
0 for values that are missing (i.e. NaNs that were replaced by zeros).
This mask is used to filter out missing values for the final loss calculation.   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on certain to


TimeSeriesTransformerModel

ken indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   decoder_attention_mask (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Mask to avoid performing attention on certain token indices. By default, a causal mask will be used, to
make sure the model can only look at previous inputs in order to predict the future.   head_mask (torch.Tensor of shape (encoder_layers, encoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   decoder_head_mask (torch.Tensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   cross_attn_head_mask (torch.Te


TimeSeriesTransformerModel

nsor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the cross-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   encoder_outputs (tuple(tuple(torch.FloatTensor), optional) —
Tuple consists of last_hidden_state, hidden_states (optional) and attentions (optional)
last_hidden_state of shape (batch_size, sequence_length, hidden_size) (optional) is a sequence of
hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.   past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) —
Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).


TimeSeriesTransformerModel

Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, op


TimeSeriesTransformerModel

tional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.Seq2SeqTSModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.Seq2SeqTSModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (TimeSeriesTransformerConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the decoder of the model.
If past_key_values is used only the last hidden-state of the sequences of


TimeSeriesTransformerModel

shape (batch_size, 1, hidden_size) is output.
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidd


TimeSeriesTransformerModel

en-states of the decoder at the output of each layer plus the optional initial embedding outputs.
decoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_


TimeSeriesTransformerModel

size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the optional initial embedding outputs.
encoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
loc (torch.FloatTensor of shape (batch_size,) or


TimeSeriesTransformerModel

(batch_size, input_size), optional) — Shift values of each time series’ context window which is used to give the model inputs of the same
magnitude and then used to shift back to the original magnitude.
scale (torch.FloatTensor of shape (batch_size,) or (batch_size, input_size), optional) — Scaling values of each time series’ context window which is used to give the model inputs of the same
magnitude and then used to rescale back to the original magnitude.
static_features (torch.FloatTensor of shape (batch_size, feature size), optional) — Static features of each time series’ in a batch which are copied to the covariates at inference time.
  The TimeSeriesTransformerModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  E


TimeSeriesTransformerModel

xamples:   Copied >>> from huggingface_hub import hf_hub_download
>>> import torch
>>> from transformers import TimeSeriesTransformerModel
>>> file = hf_hub_download(
...     repo_id="hf-internal-testing/tourism-monthly-batch", filename="train-batch.pt", repo_type="dataset"
... )
>>> batch = torch.load(file)
>>> model = TimeSeriesTransformerModel.from_pretrained("huggingface/time-series-transformer-tourism-monthly")
>>> # during training, one provides both past and future values
>>> # as well as possible additional features
>>> outputs = model(
...     past_values=batch["past_values"],
...     past_time_features=batch["past_time_features"],
...     past_observed_mask=batch["past_observed_mask"],
...     static_categorical_features=batch["static_categorical_features"],
...     static_real_features=batch["static_real_features"],
...     future_values=batch["future_values"],
...     future_time_features=batch["future_time_features"],
... )
>>> last_hidden_state = outputs.last_hidd


TimeSeriesTransformerForPrediction

class transformers.TimeSeriesTransformerForPrediction  < source > ( config: TimeSeriesTransformerConfig  )   Parameters   config (TimeSeriesTransformerConfig) —
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
from_pretrained() method to load the model weights.    The Time Series Transformer Model with a distribution head on top for time-series forecasting.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( past_values: Tensor past_time_features: Tensor


TimeSeriesTransformerForPrediction

past_observed_mask: Tensor static_categorical_features: typing.Optional[torch.Tensor] = None static_real_features: typing.Optional[torch.Tensor] = None future_values: typing.Optional[torch.Tensor] = None future_time_features: typing.Optional[torch.Tensor] = None future_observed_mask: typing.Optional[torch.Tensor] = None decoder_attention_mask: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.Tensor] = None decoder_head_mask: typing.Optional[torch.Tensor] = None cross_attn_head_mask: typing.Optional[torch.Tensor] = None encoder_outputs: typing.Optional[typing.List[torch.FloatTensor]] = None past_key_values: typing.Optional[typing.List[torch.FloatTensor]] = None output_hidden_states: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None use_cache: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.Seq2SeqTSModelOutput or tuple(torch.FloatTensor)  Parameters   p


TimeSeriesTransformerForPrediction

ast_values (torch.FloatTensor of shape (batch_size, sequence_length) or (batch_size, sequence_length, input_size)) —
Past values of the time series, that serve as context in order to predict the future. The sequence size of
this tensor must be larger than the context_length of the model, since the model will use the larger size
to construct lag features, i.e. additional values from the past which are added in order to serve as “extra
context”.
The sequence_length here is equal to config.context_length + max(config.lags_sequence), which if no
lags_sequence is configured, is equal to config.context_length + 7 (as by default, the largest
look-back index in config.lags_sequence is 7). The property _past_length returns the actual length of
the past.
The past_values is what the Transformer encoder gets as input (with optional additional features, such as
static_categorical_features, static_real_features, past_time_features and lags).
Optionally, missing values need t


TimeSeriesTransformerForPrediction

o be replaced with zeros and indicated via the past_observed_mask.
For multivariate time series, the input_size > 1 dimension is required and corresponds to the number of
variates in the time series per time step.   past_time_features (torch.FloatTensor of shape (batch_size, sequence_length, num_features)) —
Required time features, which the model internally will add to past_values. These could be things like
“month of year”, “day of the month”, etc. encoded as vectors (for instance as Fourier features). These
could also be so-called “age” features, which basically help the model know “at which point in life” a
time-series is. Age features have small values for distant past time steps and increase monotonically the
more we approach the current time step. Holiday features are also a good example of time features.
These features serve as the “positional encodings” of the inputs. So contrary to a model like BERT, where
the position encodings are learned from scrat


TimeSeriesTransformerForPrediction

ch internally as parameters of the model, the Time Series
Transformer requires to provide additional time features. The Time Series Transformer only learns
additional embeddings for static_categorical_features.
Additional dynamic real covariates can be concatenated to this tensor, with the caveat that these features
must but known at prediction time.
The num_features here is equal to config.num_time_features+config.num_dynamic_real_features`.   past_observed_mask (torch.BoolTensor of shape (batch_size, sequence_length) or (batch_size, sequence_length, input_size), optional) —
Boolean mask to indicate which past_values were observed and which were missing. Mask values selected in
[0, 1]:
1 for values that are observed,
0 for values that are missing (i.e. NaNs that were replaced by zeros).
   static_categorical_features (torch.LongTensor of shape (batch_size, number of static categorical features), optional) —
Optional static categorical features for which the mo


TimeSeriesTransformerForPrediction

del will learn an embedding, which it will add to the
values of the time series.
Static categorical features are features which have the same value for all time steps (static over time).
A typical example of a static categorical feature is a time series ID.   static_real_features (torch.FloatTensor of shape (batch_size, number of static real features), optional) —
Optional static real features which the model will add to the values of the time series.
Static real features are features which have the same value for all time steps (static over time).
A typical example of a static real feature is promotion information.   future_values (torch.FloatTensor of shape (batch_size, prediction_length) or (batch_size, prediction_length, input_size), optional) —
Future values of the time series, that serve as labels for the model. The future_values is what the
Transformer needs during training to learn to output, given the past_values.
The sequence length here is equal to p


TimeSeriesTransformerForPrediction

rediction_length.
See the demo notebook and code snippets for details.
Optionally, during training any missing values need to be replaced with zeros and indicated via the
future_observed_mask.
For multivariate time series, the input_size > 1 dimension is required and corresponds to the number of
variates in the time series per time step.   future_time_features (torch.FloatTensor of shape (batch_size, prediction_length, num_features)) —
Required time features for the prediction window, which the model internally will add to future_values.
These could be things like “month of year”, “day of the month”, etc. encoded as vectors (for instance as
Fourier features). These could also be so-called “age” features, which basically help the model know “at
which point in life” a time-series is. Age features have small values for distant past time steps and
increase monotonically the more we approach the current time step. Holiday features are also a good example
of time fea


TimeSeriesTransformerForPrediction

tures.
These features serve as the “positional encodings” of the inputs. So contrary to a model like BERT, where
the position encodings are learned from scratch internally as parameters of the model, the Time Series
Transformer requires to provide additional time features. The Time Series Transformer only learns
additional embeddings for static_categorical_features.
Additional dynamic real covariates can be concatenated to this tensor, with the caveat that these features
must but known at prediction time.
The num_features here is equal to config.num_time_features+config.num_dynamic_real_features`.   future_observed_mask (torch.BoolTensor of shape (batch_size, sequence_length) or (batch_size, sequence_length, input_size), optional) —
Boolean mask to indicate which future_values were observed and which were missing. Mask values selected
in [0, 1]:
1 for values that are observed,
0 for values that are missing (i.e. NaNs that were replaced by zeros).
This mask is u


TimeSeriesTransformerForPrediction

sed to filter out missing values for the final loss calculation.   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on certain token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   decoder_attention_mask (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Mask to avoid performing attention on certain token indices. By default, a causal mask will be used, to
make sure the model can only look at previous inputs in order to predict the future.   head_mask (torch.Tensor of shape (encoder_layers, encoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   decoder_head_mask (torch.Tensor of shape (decoder_layers, decoder_attention_heads), opti


TimeSeriesTransformerForPrediction

onal) —
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   cross_attn_head_mask (torch.Tensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the cross-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   encoder_outputs (tuple(tuple(torch.FloatTensor), optional) —
Tuple consists of last_hidden_state, hidden_states (optional) and attentions (optional)
last_hidden_state of shape (batch_size, sequence_length, hidden_size) (optional) is a sequence of
hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.   past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) —
Tuple of tuple(torch.FloatTensor) of len


TimeSeriesTransformerForPrediction

gth config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into assoc


TimeSeriesTransformerForPrediction

iated vectors than the
model’s internal embedding lookup matrix.   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.Seq2SeqTSModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.Seq2SeqTSModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (TimeSeriesTransformerConfig) a


TimeSeriesTransformerForPrediction

nd inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the decoder of the model.
If past_key_values is used only the last hidden-state of the sequences of shape (batch_size, 1, hidden_size) is output.
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
decoder_hidden_states (tuple(torch.FloatTensor), optional, returned


TimeSeriesTransformerForPrediction

when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the decoder at the output of each layer plus the optional initial embedding outputs.
decoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_


TimeSeriesTransformerForPrediction

size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the optional initial embedding outputs.
encoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output


TimeSeriesTransformerForPrediction

_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
loc (torch.FloatTensor of shape (batch_size,) or (batch_size, input_size), optional) — Shift values of each time series’ context window which is used to give the model inputs of the same
magnitude and then used to shift back to the original magnitude.
scale (torch.FloatTensor of shape (batch_size,) or (batch_size, input_size), optional) — Scaling values of each time series’ context window which is used to give the model inputs of the same
magnitude and then used to rescale back to the original magnitude.
static_features (torch.FloatTensor of shape (batch_size, feature size), optional) — Static features of each time series’ in a batch which are copied to the covariates at inference time.
  The TimeSer


TimeSeriesTransformerForPrediction

iesTransformerForPrediction forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from huggingface_hub import hf_hub_download
>>> import torch
>>> from transformers import TimeSeriesTransformerForPrediction
>>> file = hf_hub_download(
...     repo_id="hf-internal-testing/tourism-monthly-batch", filename="train-batch.pt", repo_type="dataset"
... )
>>> batch = torch.load(file)
>>> model = TimeSeriesTransformerForPrediction.from_pretrained(
...     "huggingface/time-series-transformer-tourism-monthly"
... )
>>> # during training, one provides both past and future values
>>> # as well as possible additional features
>>> outputs = model(
...     past_values=batch["past_values"],
...     past


TimeSeriesTransformerForPrediction

_time_features=batch["past_time_features"],
...     past_observed_mask=batch["past_observed_mask"],
...     static_categorical_features=batch["static_categorical_features"],
...     static_real_features=batch["static_real_features"],
...     future_values=batch["future_values"],
...     future_time_features=batch["future_time_features"],
... )
>>> loss = outputs.loss
>>> loss.backward()
>>> # during inference, one only provides past values
>>> # as well as possible additional features
>>> # the model autoregressively generates future values
>>> outputs = model.generate(
...     past_values=batch["past_values"],
...     past_time_features=batch["past_time_features"],
...     past_observed_mask=batch["past_observed_mask"],
...     static_categorical_features=batch["static_categorical_features"],
...     static_real_features=batch["static_real_features"],
...     future_time_features=batch["future_time_features"],
... )
>>> mean_prediction = outputs.sequences.mean


Overview

FSMT (FairSeq MachineTranslation) models were introduced in Facebook FAIR’s WMT19 News Translation Task Submission by Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, Sergey Edunov.
The abstract of the paper is the following:
This paper describes Facebook FAIR’s submission to the WMT19 shared news translation task. We participate in two
language pairs and four language directions, English <-> German and English <-> Russian. Following our submission from
last year, our baseline systems are large BPE-based transformer models trained with the Fairseq sequence modeling
toolkit which


Overview

rely on sampled back-translations. This year we experiment with different bitext data filtering schemes,
as well as with adding filtered back-translated data. We also ensemble and fine-tune our models on domain-specific
data, then decode using noisy channel model reranking. Our submissions are ranked first in all four directions of the
human evaluation campaign. On En->De, our system significantly outperforms other systems as well as human translations.
This system improves upon our WMT’18 submission by 4.5 BLEU points.
This model was contributed by stas. The original code can be found
here.


Implementation Notes

FSMT uses source and target vocabulary pairs that aren’t combined into one. It doesn’t share embeddings tokens
either. Its tokenizer is very similar to XLMTokenizer and the main model is derived from
BartModel.


FSMTConfig

class transformers.FSMTConfig  < source > ( langs = ['en', 'de'] src_vocab_size = 42024 tgt_vocab_size = 42024 activation_function = 'relu' d_model = 1024 max_length = 200 max_position_embeddings = 1024 encoder_ffn_dim = 4096 encoder_layers = 12 encoder_attention_heads = 16 encoder_layerdrop = 0.0 decoder_ffn_dim = 4096 decoder_layers = 12 decoder_attention_heads = 16 decoder_layerdrop = 0.0 attention_dropout = 0.0 dropout = 0.1 activation_dropout = 0.0 init_std = 0.02 decoder_start_token_id = 2 is_encoder_decoder = True scale_embedding = True tie_word_embeddings = False num_beams = 5 length_penalty = 1.0 early_stopping = False use_cache = True pad_token_id = 1 bos_token_id = 0 eos_token_id = 2 forced_eos_token_id = 2 **common_kwargs  )   Parameters   langs (List[str]) —
A list with source language and target_language (e.g., [‘en’, ‘ru’]).   src_vocab_size (int) —
Vocabulary size of the encoder. Defines the number of different tokens that can be


FSMTConfig

represented by the
inputs_ids passed to the forward method in the encoder.   tgt_vocab_size (int) —
Vocabulary size of the decoder. Defines the number of different tokens that can be represented by the
inputs_ids passed to the forward method in the decoder.   d_model (int, optional, defaults to 1024) —
Dimensionality of the layers and the pooler layer.   encoder_layers (int, optional, defaults to 12) —
Number of encoder layers.   decoder_layers (int, optional, defaults to 12) —
Number of decoder layers.   encoder_attention_heads (int, optional, defaults to 16) —
Number of attention heads for each attention layer in the Transformer encoder.   decoder_attention_heads (int, optional, defaults to 16) —
Number of attention heads for each attention layer in the Transformer decoder.   decoder_ffn_dim (int, optional, defaults to 4096) —
Dimensionality of the “intermediate” (often named feed-forward) layer in decoder.   encoder_ffn_dim (int, optional, def


FSMTConfig

aults to 4096) —
Dimensionality of the “intermediate” (often named feed-forward) layer in decoder.   activation_function (str or Callable, optional, defaults to "relu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "silu" and "gelu_new" are supported.   dropout (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.   attention_dropout (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.   activation_dropout (float, optional, defaults to 0.0) —
The dropout ratio for activations inside the fully connected layer.   max_position_embeddings (int, optional, defaults to 1024) —
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).   init_std (float, optional, defaults to 0.02) —
The st


FSMTConfig

andard deviation of the truncated_normal_initializer for initializing all weight matrices.   scale_embedding (bool, optional, defaults to True) —
Scale embeddings by diving by sqrt(d_model).   bos_token_id (int, optional, defaults to 0) —
Beginning of stream token id.   pad_token_id (int, optional, defaults to 1) —
Padding token id.   eos_token_id (int, optional, defaults to 2) —
End of stream token id.   decoder_start_token_id (int, optional) —
This model starts decoding with eos_token_id   encoder_layerdrop (float, optional, defaults to 0.0) —
Google “layerdrop arxiv”, as its not explainable in one line.   decoder_layerdrop (float, optional, defaults to 0.0) —
Google “layerdrop arxiv”, as its not explainable in one line.   is_encoder_decoder (bool, optional, defaults to True) —
Whether this is an encoder/decoder model.   tie_word_embeddings (bool, optional, defaults to False) —
Whether to tie input and output embeddings.   num_beams (int, optio


FSMTConfig

nal, defaults to 5) —
Number of beams for beam search that will be used by default in the generate method of the model. 1 means
no beam search.   length_penalty (float, optional, defaults to 1) —
Exponential penalty to the length that is used with beam-based generation. It is applied as an exponent to
the sequence length, which in turn is used to divide the score of the sequence. Since the score is the log
likelihood of the sequence (i.e. negative), length_penalty > 0.0 promotes longer sequences, while
length_penalty < 0.0 encourages shorter sequences.   early_stopping (bool, optional, defaults to False) —
Flag that will be used by default in the generate method of the model. Whether to stop the beam search
when at least num_beams sentences are finished per batch or not.   use_cache (bool, optional, defaults to True) —
Whether or not the model should return the last key/values attentions (not used by all models).   forced_eos_token_id (int, optio


FSMTConfig

nal, defaults to 2) —
The id of the token to force as the last generated token when max_length is reached. Usually set to
eos_token_id.    This is the configuration class to store the configuration of a FSMTModel. It is used to instantiate a FSMT
model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
defaults will yield a similar configuration to that of the FSMT
facebook/wmt19-en-ru architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Examples:   Copied >>> from transformers import FSMTConfig, FSMTModel
>>> # Initializing a FSMT facebook/wmt19-en-ru style configuration
>>> config = FSMTConfig()
>>> # Initializing a model (with random weights) from the configuration
>>> model = FSMTModel(config)
>>> # Accessing the model configuration
>>> configuration = model.c


FSMTTokenizer

class transformers.FSMTTokenizer  < source > ( langs = None src_vocab_file = None tgt_vocab_file = None merges_file = None do_lower_case = False unk_token = '<unk>' bos_token = '<s>' sep_token = '</s>' pad_token = '<pad>' **kwargs  )   Parameters   langs (List[str], optional) —
A list of two languages to translate from and to, for instance ["en", "ru"].   src_vocab_file (str, optional) —
File containing the vocabulary for the source language.   tgt_vocab_file (st, optional) —
File containing the vocabulary for the target language.   merges_file (str, optional) —
File containing the merges.   do_lower_case (bool, optional, defaults to False) —
Whether or not to lowercase the input when tokenizing.   unk_token (str, optional, defaults to "<unk>") —
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.   bos_token (str, optional, defaul


FSMTTokenizer

ts to "<s>") —
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the cls_token.
   sep_token (str, optional, defaults to "</s>") —
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.   pad_token (str, optional, defaults to "<pad>") —
The token used for padding, for example when batching sequences of different lengths.    Construct an FAIRSEQ Transformer tokenizer. Based on Byte-Pair Encoding. The tokenization process is the following: Moses preprocessing and tokenization. Normalizing all inputs text. The argument


FSMTTokenizer

s special_tokens and the function set_special_tokens, can be used to add additional symbols (like
”classify”) to a vocabulary. The argument langs defines a pair of languages. This tokenizer inherits from PreTrainedTokenizer which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.  build_inputs_with_special_tokens  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None  ) → List[int]  Parameters   token_ids_0 (List[int]) —
List of IDs to which the special tokens will be added.   token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.   Returns
List[int]

List of input IDs with the appropriate special tokens.
  Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A FAIRSEQ Transformer


FSMTTokenizer

 sequence has the following format: single sequence: <s> X </s> pair of sequences: <s> A </s> B </s>  get_special_tokens_mask  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None already_has_special_tokens: bool = False  ) → List[int]  Parameters   token_ids_0 (List[int]) —
List of IDs.   token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.   already_has_special_tokens (bool, optional, defaults to False) —
Whether or not the token list is already formatted with special tokens for the model.   Returns
List[int]

A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
  Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer prepare_for_model method.  create_token_type_ids_from_sequences  < source > ( token_id


FSMTTokenizer

s_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None  ) → List[int]  Parameters   token_ids_0 (List[int]) —
List of IDs.   token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.   Returns
List[int]

List of token type IDs according to the given sequence(s).
  Create a mask from the two sequences passed to be used in a sequence-pair classification task. A FAIRSEQ  Transformer sequence pair mask has the following format:   Copied 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
| first sequence    | second sequence | If token_ids_1 is None, this method only returns the first portion of the mask (0s). Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An
FAIRSEQ_TRANSFORMER sequence pair mask has the following format:  save_vocabulary  < source > ( save_directory: str filename_prefix: typing.Optional[str] = None  )


FSMTModel

class transformers.FSMTModel  < source > ( config: FSMTConfig  )   Parameters   config (FSMTConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare FSMT Model outputting raw hidden-states without any specific head on top. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: LongTensor attention_mask: typing.Optional[torch.Tensor] = None decoder_input_ids: t


FSMTModel

yping.Optional[torch.LongTensor] = None decoder_attention_mask: typing.Optional[torch.BoolTensor] = None head_mask: typing.Optional[torch.Tensor] = None decoder_head_mask: typing.Optional[torch.Tensor] = None cross_attn_head_mask: typing.Optional[torch.Tensor] = None encoder_outputs: typing.Optional[typing.Tuple[torch.FloatTensor]] = None past_key_values: typing.Optional[typing.Tuple[torch.FloatTensor]] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None decoder_inputs_embeds: typing.Optional[torch.FloatTensor] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.Seq2SeqModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obt


FSMTModel

ained using FSTMTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   decoder_input_ids (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Indices of decoder input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are decoder input IDs?
FSMT uses the eos_token_id as the starting token for decoder_input_ids generation. If past_key_values
is used, optionally only the last decoder_input_ids have to be input (see past_key_values).   decoder_attention_mask (torch.BoolTensor of shape (batch_siz


FSMTModel

e, target_sequence_length), optional) —
Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also
be used by default.   head_mask (torch.Tensor of shape (encoder_layers, encoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   decoder_head_mask (torch.Tensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   cross_attn_head_mask (torch.Tensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indica


FSMTModel

tes the head is masked.
   encoder_outputs (Tuple(torch.FloatTensor), optional) —
Tuple consists of (last_hidden_state, optional: hidden_states, optional: attentions)
last_hidden_state of shape (batch_size, sequence_length, hidden_size) is a sequence of hidden-states at
the output of the last layer of the encoder. Used in the cross-attention of the decoder.   past_key_values (Tuple(torch.FloatTensor) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)) —
Contains precomputed key and value hidden-states of the attention blocks. Can be used to speed up decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).   inputs_embeds (torch.FloatTensor of shape


FSMTModel

 (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   decoder_inputs_embeds (torch.FloatTensor of shape (batch_size, target_sequence_length, hidden_size), optional) —
Optionally, instead of passing decoder_input_ids you can choose to directly pass an embedded
representation. If past_key_values is used, optionally only the last decoder_inputs_embeds have to be
input (see past_key_values). This is useful if you want more control over how to convert
decoder_input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
If decoder_input_ids and decoder_inputs_embeds are both unset, decoder_inputs_embeds takes the value
of inputs_embeds.   use_cache (bool, opti


FSMTModel

onal, defaults to True) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.Seq2SeqModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.Seq2SeqModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (FSMTConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size,


FSMTModel

 sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the decoder of the model.
If past_key_values is used only the last hidden-state of the sequences of shape (batch_size, 1, hidden_size) is output.
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or whe


FSMTModel

n config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the decoder at the output of each layer plus the optional initial embedding outputs.
decoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence


FSMTModel

_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the optional initial embedding outputs.
encoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_at