XLMProphetNetDecoder

 the output of the last layer of the decoder of the model.
If past_key_values is used only the last hidden-state of the sequences of shape (batch_size, 1, hidden_size) is output.
last_hidden_state_ngram (torch.FloatTensor of shape (batch_size, ngram * decoder_sequence_length, config.vocab_size)) — Sequence of predict stream hidden-states at the output of the last layer of the decoder of the model.
past_key_values (List[torch.FloatTensor], optional, returned when use_cache=True is passed or when config.use_cache=True) — List of torch.FloatTensor of length config.n_layers, with each tensor of shape (2, batch_size, num_attn_heads, decoder_sequence_length, embed_size_per_head)).
Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be
used (see past_key_values input) to speed up sequential decoding.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=T


XLMProphetNetDecoder

rue is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, decoder_sequence_length, hidden_size).
Hidden-states of main stream of the decoder at the output of each layer plus the initial embedding outputs.
ngram_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, ngram * decoder_sequence_length, hidden_size).
Hidden-states of the predict stream of the decoder at the output of each layer plus the initial embedding
outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each


XLMProphetNetDecoder

layer) of shape (batch_size, num_attn_heads, decoder_sequence_length, decoder_sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
ngram_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_attn_heads, decoder_sequence_length, decoder_sequence_length).
Attentions weights of the predict stream of the decoder, after the attention softmax, used to compute the
weighted average in the
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_attn_heads, encoder_sequence_length, decoder_sequence_length).
Attentions weights of the c


XLMProphetNetDecoder

ross-attention layer of the decoder, after the attention softmax, used to
compute the weighted average in the
  The XLMProphetNetDecoder forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, XLMProphetNetDecoder
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("patrickvonplaten/xprophetnet-large-uncased-standalone")
>>> model = XLMProphetNetDecoder.from_pretrained(
...     "patrickvonplaten/xprophetnet-large-uncased-standalone", add_cross_attention=False
... )
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden


XLMProphetNetForConditionalGeneration

class transformers.XLMProphetNetForConditionalGeneration  < source > ( config: XLMProphetNetConfig  )   Parameters   config (XLMProphetNetConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The XLMProphetNet Model with a language modeling head. Can be used for sequence generation tasks.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) Original ProphetNet code can be found here. Checkpoints were converted
from original Fairseq checkpoints. For more information on the checkpoint conversion, please take a look at the
file convert_prophetnet_original_pytorch_checkpoint_to_pytorch.py. This model is a PyTorc


XLMProphetNetForConditionalGeneration

h torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matters related to general usage and
behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None decoder_input_ids: typing.Optional[torch.Tensor] = None decoder_attention_mask: typing.Optional[torch.BoolTensor] = None head_mask: typing.Optional[torch.Tensor] = None decoder_head_mask: typing.Optional[torch.Tensor] = None cross_attn_head_mask: typing.Optional[torch.Tensor] = None encoder_outputs: typing.Optional[torch.Tensor] = None past_key_values: typing.Optional[typing.Tuple[typing.Tuple[torch.Tensor]]] = None inputs_embeds: typing.Optional[torch.Tensor] = None decoder_inputs_embeds: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[


XLMProphetNetForConditionalGeneration

bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.xlm_prophetnet.modeling_xlm_prophetnet.XLMProphetNetSeq2SeqLMOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   decoder_input_ids (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Indices of decoder input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See P


XLMProphetNetForConditionalGeneration

reTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are decoder input IDs?
XLMProphetNet uses the eos_token_id as the starting token for decoder_input_ids generation. If
past_key_values is used, optionally only the last decoder_input_ids have to be input (see
past_key_values).   decoder_attention_mask (torch.BoolTensor of shape (batch_size, target_sequence_length), optional) —
Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also
be used by default.   head_mask (torch.Tensor of shape (encoder_layers, encoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   decoder_head_mask (torch.Tensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in


XLMProphetNetForConditionalGeneration

 [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   cross_attn_head_mask (torch.Tensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the cross-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   encoder_outputs (tuple(tuple(torch.FloatTensor), optional) —
Tuple consists of (last_hidden_state, optional: hidden_states, optional: attentions)
last_hidden_state of shape (batch_size, sequence_length, hidden_size), optional) is a sequence of
hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.   past_key_values (tuple(tuple(torch.FloatTensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)) —
Contains precomputed key and value hidden-states of the attention blocks. Can be used to speed up deco


XLMProphetNetForConditionalGeneration

ding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the sequence classification/regression loss. Indices shoul


XLMProphetNetForConditionalGeneration

d be in [-100, 0, ..., config.vocab_size - 1]. All labels set to -100 are ignored (masked), the loss is only computed for
labels in [0, ..., config.vocab_size]   Returns
transformers.models.xlm_prophetnet.modeling_xlm_prophetnet.XLMProphetNetSeq2SeqLMOutput or tuple(torch.FloatTensor)

A transformers.models.xlm_prophetnet.modeling_xlm_prophetnet.XLMProphetNetSeq2SeqLMOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (XLMProphetNetConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Language modeling loss.
logits (torch.FloatTensor of shape (batch_size, decoder_sequence_length, config.vocab_size)) — Prediction scores of the main stream language modeling head (scores for each vocabulary token before
SoftMax).
logits_ngram (torch.FloatTensor of shape (batch_size, ngram * decoder_sequence_length, config.vocab


XLMProphetNetForConditionalGeneration

_size)) — Prediction scores of the predict stream language modeling head (scores for each vocabulary token before
SoftMax).
past_key_values (List[torch.FloatTensor], optional, returned when use_cache=True is passed or when config.use_cache=True) — List of torch.FloatTensor of length config.n_layers, with each tensor of shape (2, batch_size, num_attn_heads, decoder_sequence_length, embed_size_per_head)).
Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be
used (see past_key_values input) to speed up sequential decoding.
decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, decoder_sequence_length, hidden_size).
Hidden-states of main stream of the decoder at the output of each layer plus the initial embe


XLMProphetNetForConditionalGeneration

dding outputs.
decoder_ngram_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, ngram * decoder_sequence_length, hidden_size).
Hidden-states of the predict stream of the decoder at the output of each layer plus the initial embedding
outputs.
decoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_attn_heads, decoder_sequence_length, decoder_sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
decoder_ngram_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or whe


XLMProphetNetForConditionalGeneration

n config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_attn_heads, decoder_sequence_length, decoder_sequence_length).
Attentions weights of the predict stream of the decoder, after the attention softmax, used to compute the
weighted average in the self-attention heads.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_attn_heads, encoder_sequence_length, decoder_sequence_length).
Attentions weights of the cross-attention layer of the decoder, after the attention softmax, used to
compute the weighted average in the
encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, encoder_sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(torch.Float


XLMProphetNetForConditionalGeneration

Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, encoder_sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the initial embedding outputs.
encoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_attn_heads, encoder_sequence_length, encoder_sequence_length). Attentions weights of the encoder, after the attention
softmax, used to compute the weighted average in the self-attention heads.
  The XLMProphetNetForConditionalGeneration forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
i


XLMProphetNetForConditionalGeneration

nstance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, XLMProphetNetForConditionalGeneration
>>> tokenizer = AutoTokenizer.from_pretrained("patrickvonplaten/xprophetnet-large-uncased-standalone")
>>> model = XLMProphetNetForConditionalGeneration.from_pretrained(
...     "patrickvonplaten/xprophetnet-large-uncased-standalone"
... )
>>> input_ids = tokenizer(
...     "Studies have been shown that owning a dog is good for you", return_tensors="pt"
... ).input_ids  # Batch size 1
>>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids  # Batch size 1
>>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids)
>>> logits_next_token = outputs.logits  # logits to predict next token as usual
>>> logits_ngram_next_tokens = outputs.logits_ngram  # logits to predict 2nd, 3rd, ... n


XLMProphetNetForCausalLM

class transformers.XLMProphetNetForCausalLM  < source > ( config: XLMProphetNetConfig  )   Parameters   config (XLMProphetNetConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The standalone decoder part of the XLMProphetNetModel with a lm head on top. The model can be used for causal language modeling.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) Original ProphetNet code can be found here. Checkpoints were converted
from original Fairseq checkpoints. For more information on the checkpoint conversion, please take a look at the
file convert_prophetnet_original_pytorch_checkpoint_to_


XLMProphetNetForCausalLM

pytorch.py. This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matters related to general usage and
behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None encoder_hidden_states: typing.Optional[torch.Tensor] = None encoder_attention_mask: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None cross_attn_head_mask: typing.Optional[torch.Tensor] = None past_key_values: typing.Optional[typing.Tuple[typing.Tuple[torch.Tensor]]] = None inputs_embeds: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.xlm_prophetnet.modeling_xlm_prophetnet.X


XLMProphetNetForCausalLM

LMProphetNetDecoderLMOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   head_mask (torch.Tensor of shape (encoder_layers, encoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   output_attentions (bool, optional) —
Whether o


XLMProphetNetForCausalLM

r not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   encoder_hidden_states (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if
the model is configured as a decoder.   encoder_attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in
the cross-attention if the model is configured as a decoder. Mask values selected in [0, 1]:   cross_attn_head_mask (torch.Tensor of shape (


XLMProphetNetForCausalLM

decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the cross-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   past_key_values (tuple(tuple(torch.FloatTensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)) —
Contains precomputed key and value hidden-states of the attention blocks. Can be used to speed up decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).
1 for tokens that are not masked,
0 for t


XLMProphetNetForCausalLM

okens that are masked.
   labels (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in
[-100, 0, ..., config.vocab_size] (see input_ids docstring) Tokens with indices set to -100 are
ignored (masked), the loss is only computed for the tokens with labels n [0, ..., config.vocab_size]   Returns
transformers.models.xlm_prophetnet.modeling_xlm_prophetnet.XLMProphetNetDecoderLMOutput or tuple(torch.FloatTensor)

A transformers.models.xlm_prophetnet.modeling_xlm_prophetnet.XLMProphetNetDecoderLMOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (XLMProphetNetConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Language modeling loss.
logits (torch.FloatTensor of shape (batch_


XLMProphetNetForCausalLM

size, decoder_sequence_length, config.vocab_size)) — Prediction scores of the main stream language modeling head (scores for each vocabulary token before
SoftMax).
logits_ngram (torch.FloatTensor of shape (batch_size, ngram * decoder_sequence_length, config.vocab_size)) — Prediction scores of the predict stream language modeling head (scores for each vocabulary token before
SoftMax).
past_key_values (List[torch.FloatTensor], optional, returned when use_cache=True is passed or when config.use_cache=True) — List of torch.FloatTensor of length config.n_layers, with each tensor of shape (2, batch_size, num_attn_heads, decoder_sequence_length, embed_size_per_head)).
Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be
used (see past_key_values input) to speed up sequential decoding.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden


XLMProphetNetForCausalLM

_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, decoder_sequence_length, hidden_size).
Hidden-states of main stream of the decoder at the output of each layer plus the initial embedding outputs.
ngram_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, ngram * decoder_sequence_length, hidden_size).
Hidden-states of the predict stream of the decoder at the output of each layer plus the initial embedding
outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_attn_heads, decoder_sequence_length,


XLMProphetNetForCausalLM

decoder_sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
ngram_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_attn_heads, decoder_sequence_length, decoder_sequence_length).
Attentions weights of the predict stream of the decoder, after the attention softmax, used to compute the
weighted average in the
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_attn_heads, encoder_sequence_length, decoder_sequence_length).
Attentions weights of the cross-attention layer of the decoder, after the attention softmax, used to
compute the weighted ave


XLMProphetNetForCausalLM

rage in the
  The XLMProphetNetForCausalLM forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, XLMProphetNetForCausalLM
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("patrickvonplaten/xprophetnet-large-uncased-standalone")
>>> model = XLMProphetNetForCausalLM.from_pretrained("patrickvonplaten/xprophetnet-large-uncased-standalone")
>>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder."
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> outputs = model(**inputs)
>>> logits = outputs.logits
>>> # Model can also be used with EncoderDecoder framework
>>> from transformer


XLMProphetNetForCausalLM

s import BertTokenizer, EncoderDecoderModel, AutoTokenizer
>>> import torch
>>> tokenizer_enc = BertTokenizer.from_pretrained("bert-large-uncased")
>>> tokenizer_dec = AutoTokenizer.from_pretrained("patrickvonplaten/xprophetnet-large-uncased-standalone")
>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained(
...     "bert-large-uncased", "patrickvonplaten/xprophetnet-large-uncased-standalone"
... )
>>> ARTICLE = (
...     "the us state department said wednesday it had received no "
...     "formal word from bolivia that it was expelling the us ambassador there "
...     "but said the charges made against him are `` baseless ."
... )
>>> input_ids = tokenizer_enc(ARTICLE, return_tensors="pt").input_ids
>>> labels = tokenizer_dec(
...     "us rejects charges against its ambassador in bolivia", return_tensors="pt"
... ).input_ids
>>> outputs = model(input_ids=input_ids, decoder_input_ids=labels[:, :-1], labels=labels[:, 1:])
>>> loss = outputs.loss


Overview

The PoolFormer model was proposed in MetaFormer is Actually What You Need for Vision  by Sea AI Labs. Instead of designing complicated token mixer to achieve SOTA performance, the target of this work is to demonstrate the competence of transformer models largely stem from the general architecture MetaFormer.
The abstract from the paper is the following:
Transformers have shown great potential in computer vision tasks. A common belief is their attention-based token mixer module contributes most to their competence. However, recent works show the attention-based module in transformers can be replaced by spatial MLPs and the resulted models still perform quite well. Based on this observation, we hypothesize that the general architecture of the transformers, instead of the specific token m


Overview

ixer module, is more essential to the model’s performance. To verify this, we deliberately replace the attention module in transformers with an embarrassingly simple spatial pooling operator to conduct only the most basic token mixing. Surprisingly, we observe that the derived model, termed as PoolFormer, achieves competitive performance on multiple computer vision tasks. For example, on ImageNet-1K, PoolFormer achieves 82.1% top-1 accuracy, surpassing well-tuned vision transformer/MLP-like baselines DeiT-B/ResMLP-B24 by 0.3%/1.1% accuracy with 35%/52% fewer parameters and 48%/60% fewer MACs. The effectiveness of PoolFormer verifies our hypothesis and urges us to initiate the concept of “MetaFormer”, a general architecture abstracted from transformers without specifying the token mixer


Overview

. Based on the extensive experiments, we argue that MetaFormer is the key player in achieving superior results for recent transformer and MLP-like models on vision tasks. This work calls for more future research dedicated to improving MetaFormer instead of focusing on the token mixer modules. Additionally, our proposed PoolFormer could serve as a starting baseline for future MetaFormer architecture design.
The figure below illustrates the architecture of PoolFormer. Taken from the original paper.
Tips:
PoolFormer has a hierarchical architecture, where instead of Attention, a simple Average Pooling layer is present. All checkpoints of the model can be found on the hub. One can use PoolFormerImageProcessor to prepare images for the model. As most models, PoolFormer comes in different siz


Overview

es, the details of which can be found in the table below.
| Model variant   | Depths        | Hidden sizes        |   Params (M) |   ImageNet-1k Top 1 |
|-----------------|---------------|---------------------|--------------|---------------------|
| s12             | [2, 2, 6, 2]  | [64, 128, 320, 512] |           12 |                77.2 |
| s24             | [4, 4, 12, 4] | [64, 128, 320, 512] |           21 |                80.3 |
| s36             | [6, 6, 18, 6] | [64, 128, 320, 512] |           31 |                81.4 |
| m36             | [6, 6, 18, 6] | [96, 192, 384, 768] |           56 |                82.1 |
| m48             | [8, 8, 24, 8] | [96, 192, 384, 768] |           73 |                82.5 |
This model was contributed by heytanay. The original code can be found he


Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with PoolFormer.
 Image Classification
PoolFormerForImageClassification is supported by this example script and notebook. See also: Image classification task guide
If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we’ll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


PoolFormerConfig

class transformers.PoolFormerConfig  < source > ( num_channels = 3 patch_size = 16 stride = 16 pool_size = 3 mlp_ratio = 4.0 depths = [2, 2, 6, 2] hidden_sizes = [64, 128, 320, 512] patch_sizes = [7, 3, 3, 3] strides = [4, 2, 2, 2] padding = [2, 1, 1, 1] num_encoder_blocks = 4 drop_path_rate = 0.0 hidden_act = 'gelu' use_layer_scale = True layer_scale_init_value = 1e-05 initializer_range = 0.02 **kwargs  )   Parameters   num_channels (int, optional, defaults to 3) —
The number of channels in the input image.   patch_size (int, optional, defaults to 16) —
The size of the input patch.   stride (int, optional, defaults to 16) —
The stride of the input patch.   pool_size (int, optional, defaults to 3) —
The size of the pooling window.   mlp_ratio (float, optional, defaults to 4.0) —
The ratio of the number of channels in the output of the MLP to the number of channels in the input.   depths (list, optional, defaults to [2, 2, 6, 2]) —
The depth of each


PoolFormerConfig

 encoder block.   hidden_sizes (list, optional, defaults to [64, 128, 320, 512]) —
The hidden sizes of each encoder block.   patch_sizes (list, optional, defaults to [7, 3, 3, 3]) —
The size of the input patch for each encoder block.   strides (list, optional, defaults to [4, 2, 2, 2]) —
The stride of the input patch for each encoder block.   padding (list, optional, defaults to [2, 1, 1, 1]) —
The padding of the input patch for each encoder block.   num_encoder_blocks (int, optional, defaults to 4) —
The number of encoder blocks.   drop_path_rate (float, optional, defaults to 0.0) —
The dropout rate for the dropout layers.   hidden_act (str, optional, defaults to "gelu") —
The activation function for the hidden layers.   use_layer_scale (bool, optional, defaults to True) —
Whether to use layer scale.   layer_scale_init_value (float, optional, defaults to 1e-05) —
The initial value for the layer scale.   initializer_range (float, optional, defaults


PoolFormerConfig

 to 0.02) —
The initializer range for the weights.    This is the configuration class to store the configuration of PoolFormerModel. It is used to instantiate a
PoolFormer model according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the PoolFormer
sail/poolformer_s12 architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import PoolFormerConfig, PoolFormerModel
>>> # Initializing a PoolFormer sail/poolformer_s12 style configuration
>>> configuration = PoolFormerConfig()
>>> # Initializing a model (with random weights) from the sail/poolformer_s12 style configuration
>>> model = PoolFormerModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.confi


PoolFormerFeatureExtractor

class transformers.PoolFormerFeatureExtractor  < source > ( *args **kwargs  )     __call__  < source > ( images **kwargs  )    Preprocess an image or a batch of images.


PoolFormerImageProcessor

class transformers.PoolFormerImageProcessor  < source > ( do_resize: bool = True size: typing.Dict[str, int] = None crop_pct: int = 0.9 resample: Resampling = <Resampling.BICUBIC: 3> do_center_crop: bool = True crop_size: typing.Dict[str, int] = None rescale_factor: typing.Union[int, float] = 0.00392156862745098 do_rescale: bool = True do_normalize: bool = True image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float], NoneType] = None **kwargs  )   Parameters   do_resize (bool, optional, defaults to True) —
Whether to resize the image’s (height, width) dimensions to the specified size. Can be overridden by
do_resize in the preprocess method.   size (Dict[str, int] optional, defaults to {"shortest_edge" -- 224}):
Size of the image after resizing. Can be overridden by size in the preprocess method. If crop_pct


PoolFormerImageProcessor

 is
unset:
size is {"height": h, "width": w}: the image is resized to (h, w).
size is {"shortest_edge": s}: the shortest edge of the image is resized to s whilst maintaining the
aspect ratio.
If crop_pct is set:
size is {"height": h, "width": w}: the image is resized to (int(floor(h/crop_pct)), int(floor(w/crop_pct)))
size is {"height": c, "width": c}: the shortest edge of the image is resized to int(floor(c/crop_pct)
whilst maintaining the aspect ratio.
size is {"shortest_edge": c}: the shortest edge of the image is resized to int(floor(c/crop_pct)
whilst maintaining the aspect ratio.
   crop_pct (float, optional, defaults to 0.9) —
Percentage of the image to crop from the center. Can be overridden by crop_pct in the preprocess
method.   resample (PILImageResampling, optional, defaults to Resampling.BICUBIC) —
Resampling filter to use if resizing the image. Can be ove


PoolFormerImageProcessor

rridden by resample in the preprocess method.   do_center_crop (bool, optional, defaults to True) —
Whether to center crop the image. If the input size is smaller than crop_size along any edge, the image
is padded with 0’s and then center cropped. Can be overridden by do_center_crop in the preprocess
method.   crop_size (Dict[str, int], optional, defaults to {"height" -- 224, "width": 224}):
Size of the image after applying center crop. Only has an effect if do_center_crop is set to True. Can
be overridden by the crop_size parameter in the preprocess method.   rescale_factor (int or float, optional, defaults to 1/255) —
Scale factor to use if rescaling the image. Can be overridden by the rescale_factor parameter in the
preprocess method.   do_rescale (bool, optional, defaults to True) —
Whether to rescale the image by the specified scale rescale_factor. Can be overridd


PoolFormerImageProcessor

en by the do_rescale
parameter in the preprocess method.   do_normalize (bool, optional, defaults to True) —
Controls whether to normalize the image. Can be overridden by the do_normalize parameter in the
preprocess method.   image_mean (float or List[float], optional, defaults to IMAGENET_STANDARD_MEAN) —
Mean to use if normalizing the image. This is a float or list of floats the length of the number of
channels in the image. Can be overridden by the image_mean parameter in the preprocess method.   image_std (float or List[float], optional, defaults to IMAGENET_STANDARD_STD) —
Standard deviation to use if normalizing the image. This is a float or list of floats the length of the
number of channels in the image. Can be overridden by the image_std parameter in the preprocess method.    Constructs a PoolFormer image processor.  preprocess  < source > ( images: typing.Uni


PoolFormerImageProcessor

on[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]] do_resize: bool = None size: typing.Dict[str, int] = None crop_pct: int = None resample: Resampling = None do_center_crop: bool = None crop_size: typing.Dict[str, int] = None do_rescale: bool = None rescale_factor: float = None do_normalize: bool = None image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float], NoneType] = None return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None data_format: ChannelDimension = <ChannelDimension.FIRST: 'channels_first'> input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None **kwargs  )   Parameters   images (Ima


PoolFormerImageProcessor

geInput) —
Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If
passing in images with pixel values between 0 and 1, set do_rescale=False.   do_resize (bool, optional, defaults to self.do_resize) —
Whether to resize the image.   size (Dict[str, int], optional, defaults to self.size) —
Size of the image after applying resize.   crop_pct (float, optional, defaults to self.crop_pct) —
Percentage of the image to crop. Only has an effect if do_resize is set to True.   resample (int, optional, defaults to self.resample) —
Resampling filter to use if resizing the image. This can be one of the enum PILImageResampling, Only
has an effect if do_resize is set to True.   do_center_crop (bool, optional, defaults to self.do_center_crop) —
Whether to center crop the image.   crop_size (Dict[str, int], optional, defaults to self.crop_siz


PoolFormerImageProcessor

e) —
Size of the image after applying center crop.   do_rescale (bool, optional, defaults to self.do_rescale) —
Whether to rescale the image values between [0 - 1].   rescale_factor (float, optional, defaults to self.rescale_factor) —
Rescale factor to rescale the image by if do_rescale is set to True.   do_normalize (bool, optional, defaults to self.do_normalize) —
Whether to normalize the image.   image_mean (float or List[float], optional, defaults to self.image_mean) —
Image mean.   image_std (float or List[float], optional, defaults to self.image_std) —
Image standard deviation.   return_tensors (str or TensorType, optional) —
The type of tensors to return. Can be one of:
Unset: Return a list of np.ndarray.
TensorType.TENSORFLOW or 'tf': Return a batch of type tf.Tensor.
TensorType.PYTORCH or 'pt': Return a batch of type torch.Tensor.
TensorType.NUMPY or 'np': Ret


PoolFormerImageProcessor

urn a batch of type np.ndarray.
TensorType.JAX or 'jax': Return a batch of type jax.numpy.ndarray.
   data_format (ChannelDimension or str, optional, defaults to ChannelDimension.FIRST) —
The channel dimension format for the output image. Can be one of:
ChannelDimension.FIRST: image in (num_channels, height, width) format.
ChannelDimension.LAST: image in (height, width, num_channels) format.
   input_data_format (ChannelDimension or str, optional) —
The channel dimension format for the input image. If unset, the channel dimension format is inferred
from the input image. Can be one of:
"channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
"channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
"none" or ChannelDimension.NONE: image in (height, width) format.
    Preprocess an image or batch of images.


PoolFormerModel

class transformers.PoolFormerModel  < source > ( config  )   Parameters   config (PoolFormerConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare PoolFormer Model transformer outputting raw hidden-states without any specific head on top.
This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: typing.Optional[torch.FloatTensor] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.BaseModelOutputWithNoAttention or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel


PoolFormerModel

values. Pixel values can be obtained using AutoImageProcessor. See
PoolFormerImageProcessor.call() for details.   Returns
transformers.modeling_outputs.BaseModelOutputWithNoAttention or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutputWithNoAttention or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (PoolFormerConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, num_channels, height, width)) — Sequence of hidden-states at the output of the last layer of the model.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, num_channels, height, width).
Hidden-states of the model at t


PoolFormerModel

he output of each layer plus the optional initial embedding outputs.
  The PoolFormerModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoImageProcessor, PoolFormerModel
>>> import torch
>>> from datasets import load_dataset
>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]
>>> image_processor = AutoImageProcessor.from_pretrained("sail/poolformer_s12")
>>> model = PoolFormerModel.from_pretrained("sail/poolformer_s12")
>>> inputs = image_processor(image, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1, 512, 7, 7


PoolFormerForImageClassification

class transformers.PoolFormerForImageClassification  < source > ( config  )   Parameters   config (PoolFormerConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    PoolFormer Model transformer with an image classification head on top This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.LongTensor] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.ImageClassifierOutputWithNoAttention


PoolFormerForImageClassification

or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
PoolFormerImageProcessor.call() for details.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the image classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-Entropy).   Returns
transformers.modeling_outputs.ImageClassifierOutputWithNoAttention or tuple(torch.FloatTensor)

A transformers.modeling_outputs.ImageClassifierOutputWithNoAttention or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the conf


PoolFormerForImageClassification

iguration (PoolFormerConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each stage) of shape (batch_size, num_channels, height, width). Hidden-states (also
called feature maps) of the model at the output of each stage.
  The PoolFormerForImageClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defi


PoolFormerForImageClassification

ned within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoImageProcessor, PoolFormerForImageClassification
>>> import torch
>>> from datasets import load_dataset
>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]
>>> image_processor = AutoImageProcessor.from_pretrained("sail/poolformer_s12")
>>> model = PoolFormerForImageClassification.from_pretrained("sail/poolformer_s12")
>>> inputs = image_processor(image, return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> # model predicts one of the 1000 ImageNet classes
>>> predicted_label = logits.argmax(-1).item()
>>> print(model.config.id2label[predicted_label])
tabby, tabby c


MaskFormer

This is a recently introduced model so the API hasn’t been tested extensively. There may be some bugs or slight
breaking changes to fix it in the future. If you see something strange, file a Github Issue.


Overview

The MaskFormer model was proposed in Per-Pixel Classification is Not All You Need for Semantic Segmentation by Bowen Cheng, Alexander G. Schwing, Alexander Kirillov. MaskFormer addresses semantic segmentation with a mask classification paradigm instead of performing classic pixel-level classification.
The abstract from the paper is the following:
Modern approaches typically formulate semantic segmentation as a per-pixel classification task, while instance-level segmentation is handled with an alternative mask classification. Our key insight: mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks in a unified manner using the exact same model, loss, and training procedure. Following this observation, we propose MaskFormer, a simple mask classification model which predicts a set of binary masks, each associated with a single global class label prediction. Overall, the proposed mask


Overview

classification-based method simplifies the landscape of effective approaches to semantic and panoptic segmentation tasks and shows excellent empirical results. In particular, we observe that MaskFormer outperforms per-pixel classification baselines when the number of classes is large. Our mask classification-based method outperforms both current state-of-the-art semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models.
Tips:
MaskFormer’s Transformer decoder is identical to the decoder of DETR. During training, the authors of DETR did find it helpful to use auxiliary losses in the decoder, especially to help the model output the correct number of objects of each class. If you set the parameter use_auxilary_loss of MaskFormerConfig to True, then prediction feedforward neural networks and Hungarian losses are added after each decoder layer (with the FFNs sharing parameters). If you want to train the model in


Overview

a distributed environment across multiple nodes, then one should update the
get_num_masks function inside in the MaskFormerLoss class of modeling_maskformer.py. When training on multiple nodes, this should be
set to the average number of target masks across all nodes, as can be seen in the original implementation here. One can use MaskFormerImageProcessor to prepare images for the model and optional targets for the model. To get the final segmentation, depending on the task, you can call post_process_semantic_segmentation() or post_process_panoptic_segmentation(). Both tasks can be solved using MaskFormerForInstanceSegmentation output, panoptic segmentation accepts an optional label_ids_to_fuse argument to fuse instances of the target object/s (e.g. sky) together.
The figure below illustrates the architecture of MaskFormer. Taken from the original paper.
This model was contributed by francesco. The original code can be found here.


Resources

Image Segmentation
All notebooks that illustrate inference as well as fine-tuning on custom data with MaskFormer can be found here.


MaskFormer specific outputs

class transformers.models.maskformer.modeling_maskformer.MaskFormerModelOutput  < source > ( encoder_last_hidden_state: typing.Optional[torch.FloatTensor] = None pixel_decoder_last_hidden_state: typing.Optional[torch.FloatTensor] = None transformer_decoder_last_hidden_state: typing.Optional[torch.FloatTensor] = None encoder_hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None pixel_decoder_hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None transformer_decoder_hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None attentions: typing.Optional[typing.Tuple[torch.FloatTensor]] = None  )   Parameters   encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Last hidden states (final feature map) of the last stage of the encoder model (backbone).   pixel_de


MaskFormer specific outputs

coder_last_hidden_state (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Last hidden states (final feature map) of the last stage of the pixel decoder model (FPN).   transformer_decoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) —
Last hidden states (final feature map) of the last stage of the transformer decoder model.   encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the encoder
model at the output of each stage.   pixel_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.outpu


MaskFormer specific outputs

t_hidden_states=True) —
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the pixel
decoder model at the output of each stage.   transformer_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, sequence_length, hidden_size). Hidden-states (also called feature maps) of the
transformer decoder at the output of each stage.   hidden_states tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor containing encoder_hidden_states, pixel_decoder_hidden_states an


MaskFormer specific outputs

d
decoder_hidden_states   attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) —
Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Attentions weights from Detr’s decoder after the attention softmax, used to compute the
weighted average in the self-attention heads.    Class for outputs of MaskFormerModel. This class returns all the needed hidden states to compute the logits.
 class transformers.models.maskformer.modeling_maskformer.MaskFormerForInstanceSegmentationOutput  < source > ( loss: typing.Optional[torch.FloatTensor] = None class_queries_logits: FloatTensor = None masks_queries_logits: FloatTensor = None auxiliary_logits: FloatTensor = None encoder_last_hidden_state: typing.Optional[torch.FloatTensor] = None pixel_decoder_last_hidden_state: typing.Optional[torch.Fl


MaskFormer specific outputs

oatTensor] = None transformer_decoder_last_hidden_state: typing.Optional[torch.FloatTensor] = None encoder_hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None pixel_decoder_hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None transformer_decoder_hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None attentions: typing.Optional[typing.Tuple[torch.FloatTensor]] = None  )   Parameters   loss (torch.Tensor, optional) —
The computed loss, returned when labels are present.   class_queries_logits (torch.FloatTensor) —
A tensor of shape (batch_size, num_queries, num_labels + 1) representing the proposed classes for each
query. Note the + 1 is needed because we incorporate the null class.   masks_queries_logits (torch.FloatTensor) —
A tensor of shape (batch_size, num_queries, height, width) representing


MaskFormer specific outputs

 the proposed masks for each
query.   encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Last hidden states (final feature map) of the last stage of the encoder model (backbone).   pixel_decoder_last_hidden_state (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Last hidden states (final feature map) of the last stage of the pixel decoder model (FPN).   transformer_decoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) —
Last hidden states (final feature map) of the last stage of the transformer decoder model.   encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, num_channels, height,


MaskFormer specific outputs

 width). Hidden-states (also called feature maps) of the encoder
model at the output of each stage.   pixel_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the pixel
decoder model at the output of each stage.   transformer_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, sequence_length, hidden_size). Hidden-states of the transformer decoder at the output
of each stage.   hidden_states tuple(torch


MaskFormer specific outputs

.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor containing encoder_hidden_states, pixel_decoder_hidden_states and
decoder_hidden_states.   attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) —
Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Attentions weights from Detr’s decoder after the attention softmax, used to compute the
weighted average in the self-attention heads.    Class for outputs of MaskFormerForInstanceSegmentation. This output can be directly passed to post_process_semantic_segmentation() or or
post_process_instance_segmentation() or
post_process_panoptic_segmentation() depending on the task. Please, see
[`~MaskFormerImageProcessor] for details regarding usa


MaskFormerConfig

class transformers.MaskFormerConfig  < source > ( fpn_feature_size: int = 256 mask_feature_size: int = 256 no_object_weight: float = 0.1 use_auxiliary_loss: bool = False backbone_config: typing.Optional[typing.Dict] = None decoder_config: typing.Optional[typing.Dict] = None init_std: float = 0.02 init_xavier_std: float = 1.0 dice_weight: float = 1.0 cross_entropy_weight: float = 1.0 mask_weight: float = 20.0 output_auxiliary_logits: typing.Optional[bool] = None **kwargs  )   Parameters   mask_feature_size (int, optional, defaults to 256) —
The masks’ features size, this value will also be used to specify the Feature Pyramid Network features’
size.   no_object_weight (float, optional, defaults to 0.1) —
Weight to apply to the null (no object) class.   use_auxiliary_loss(bool, optional, defaults to False) —
If True MaskFormerForInstanceSegmentationOutput will contain the auxiliary losses computed


MaskFormerConfig

 using the
logits from each decoder’s stage.   backbone_config (Dict, optional) —
The configuration passed to the backbone, if unset, the configuration corresponding to
swin-base-patch4-window12-384 will be used.   decoder_config (Dict, optional) —
The configuration passed to the transformer decoder model, if unset the base config for detr-resnet-50
will be used.   init_std (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   init_xavier_std (float, optional, defaults to 1) —
The scaling factor used for the Xavier initialization gain in the HM Attention map module.   dice_weight (float, optional, defaults to 1.0) —
The weight for the dice loss.   cross_entropy_weight (float, optional, defaults to 1.0) —
The weight for the cross entropy loss.   mask_weight (float, optional, defaults to 20.0) —
The weight for the


MaskFormerConfig

mask loss.   output_auxiliary_logits (bool, optional) —
Should the model output its auxiliary_logits or not.   Raises
ValueError

ValueError —
Raised if the backbone model type selected is not in ["swin"] or the decoder model type selected is not
in ["detr"]
 This is the configuration class to store the configuration of a MaskFormerModel. It is used to instantiate a
MaskFormer model according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the MaskFormer
facebook/maskformer-swin-base-ade architecture trained
on ADE20k-150. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information. Currently, MaskFormer only supports the Swin Transformer as backbone.  Examples:   Copied >>> from tran


MaskFormerConfig

sformers import MaskFormerConfig, MaskFormerModel
>>> # Initializing a MaskFormer facebook/maskformer-swin-base-ade configuration
>>> configuration = MaskFormerConfig()
>>> # Initializing a model (with random weights) from the facebook/maskformer-swin-base-ade style configuration
>>> model = MaskFormerModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config  from_backbone_and_decoder_configs  < source > ( backbone_config: PretrainedConfig decoder_config: PretrainedConfig **kwargs  ) → MaskFormerConfig  Parameters   backbone_config (PretrainedConfig) —
The backbone configuration.   decoder_config (PretrainedConfig) —
The transformer decoder configuration to use.   Returns
MaskFormerConfig

An instance of a configuration object
  Instantiate a MaskFormerConfig (or a derived class) from a pre-trained backbone model configuration and DETR model
configuration.


MaskFormerImageProcessor

class transformers.MaskFormerImageProcessor  < source > ( do_resize: bool = True size: typing.Dict[str, int] = None size_divisor: int = 32 resample: Resampling = <Resampling.BILINEAR: 2> do_rescale: bool = True rescale_factor: float = 0.00392156862745098 do_normalize: bool = True image_mean: typing.Union[float, typing.List[float]] = None image_std: typing.Union[float, typing.List[float]] = None ignore_index: typing.Optional[int] = None do_reduce_labels: bool = False **kwargs  )   Parameters   do_resize (bool, optional, defaults to True) —
Whether to resize the input to a certain size.   size (int, optional, defaults to 800) —
Resize the input to the given size. Only has an effect if do_resize is set to True. If size is a
sequence like (width, height), output size will be matched to this. If size is an int, smaller edge of
the image will be matched to this number. i.e, if height > width, then image will be rescaled to (size * height / width, size).   size_divisor (int, opti


MaskFormerImageProcessor

onal, defaults to 32) —
Some backbones need images divisible by a certain number. If not passed, it defaults to the value used in
Swin Transformer.   resample (int, optional, defaults to Resampling.BILINEAR) —
An optional resampling filter. This can be one of PIL.Image.Resampling.NEAREST,
PIL.Image.Resampling.BOX, PIL.Image.Resampling.BILINEAR, PIL.Image.Resampling.HAMMING,
PIL.Image.Resampling.BICUBIC or PIL.Image.Resampling.LANCZOS. Only has an effect if do_resize is set
to True.   do_rescale (bool, optional, defaults to True) —
Whether to rescale the input to a certain scale.   rescale_factor (float, optional, defaults to 1/ 255) —
Rescale the input by the given factor. Only has an effect if do_rescale is set to True.   do_normalize (bool, optional, defaults to True) —
Whether or not to normalize the input with mean and standard deviation.   image_mean (int, optional, defaults to [0.485, 0.456, 0.406]) —
The sequence of means for each channel, to be used when normalizin


MaskFormerImageProcessor

g images. Defaults to the ImageNet mean.   image_std (int, optional, defaults to [0.229, 0.224, 0.225]) —
The sequence of standard deviations for each channel, to be used when normalizing images. Defaults to the
ImageNet std.   ignore_index (int, optional) —
Label to be assigned to background pixels in segmentation maps. If provided, segmentation map pixels
denoted with 0 (background) will be replaced with ignore_index.   do_reduce_labels (bool, optional, defaults to False) —
Whether or not to decrement all label values of segmentation maps by 1. Usually used for datasets where 0
is used for background, and background itself is not included in all classes of a dataset (e.g. ADE20k).
The background label will be replaced by ignore_index.    Constructs a MaskFormer image processor. The image processor can be used to prepare image(s) and optional targets
for the model. This image processor inherits from BaseImageProcessor which contains most of the main methods. Users should


MaskFormerImageProcessor

refer to this superclass for more information regarding those methods.  preprocess  < source > ( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]] segmentation_maps: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')], NoneType] = None instance_id_to_semantic_id: typing.Union[typing.Dict[int, int], NoneType] = None do_resize: typing.Optional[bool] = None size: typing.Union[typing.Dict[str, int], NoneType] = None size_divisor: typing.Optional[int] = None resample: Resampling = None do_rescale: typing.Optional[bool] = None rescale_factor: typing.Optional[float] = None do_normalize: typing.Optional[bool] = None image_mean: typing.Union[float, typing.List[float], NoneType] = No


MaskFormerImageProcessor

ne image_std: typing.Union[float, typing.List[float], NoneType] = None ignore_index: typing.Optional[int] = None do_reduce_labels: typing.Optional[bool] = None return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None data_format: typing.Union[str, transformers.image_utils.ChannelDimension] = <ChannelDimension.FIRST: 'channels_first'> input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None **kwargs  )     encode_inputs  < source > ( pixel_values_list: typing.List[typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]]] segmentation_maps: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]] = None instance_id_to_sema


MaskFormerImageProcessor

ntic_id: typing.Union[typing.List[typing.Dict[int, int]], typing.Dict[int, int], NoneType] = None ignore_index: typing.Optional[int] = None reduce_labels: bool = False return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None  ) → BatchFeature  Parameters   pixel_values_list (List[ImageInput]) —
List of images (pixel values) to be padded. Each image should be a tensor of shape (channels, height, width).   segmentation_maps (ImageInput, optional) —
The corresponding semantic segmentation maps with the pixel-wise annotations.
(bool, optional, defaults to True):
Whether or not to pad images up to the largest image in a batch and create a pixel mask.
If left to the default, will return a pixel mask that is:
1 for pixels that are real (i.e. not masked),
0 for pixels that are padding (i.e. masked).
   instance_id_to_semantic_id (List[Dict[int, int]] or


MaskFormerImageProcessor

 Dict[int, int], optional) —
A mapping between object instance ids and class ids. If passed, segmentation_maps is treated as an
instance segmentation map where each pixel represents an instance id. Can be provided as a single
dictionary with a global/dataset-level mapping or as a list of dictionaries (one per image), to map
instance ids in each image separately.   return_tensors (str or TensorType, optional) —
If set, will return tensors instead of NumPy arrays. If set to 'pt', return PyTorch torch.Tensor
objects.   Returns
BatchFeature

A BatchFeature with the following fields:
pixel_values — Pixel values to be fed to a model.
pixel_mask — Pixel mask to be fed to a model (when =True or if pixel_mask is in
self.model_input_names).
mask_labels — Optional list of mask labels of shape (labels, height, width) to be fed to a model
(when annotations are provided).
class_labels — Optional list of class labels of shape (labels) to be fed to a model (when
annotations are provide


MaskFormerImageProcessor

d). They identify the labels of mask_labels, e.g. the label of
mask_labels[i][j] if class_labels[i][j].
  Pad images up to the largest image in a batch and create a corresponding pixel_mask. MaskFormer addresses semantic segmentation with a mask classification paradigm, thus input segmentation maps
will be converted to lists of binary masks and their respective labels. Let’s see an example, assuming
segmentation_maps = [[2,6,7,9]], the output will contain mask_labels = [[1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1]] (four binary masks) and class_labels = [2,6,7,9], the labels for
each mask.  post_process_semantic_segmentation  < source > ( outputs target_sizes: typing.Union[typing.List[typing.Tuple[int, int]], NoneType] = None  ) → List[torch.Tensor]  Parameters   outputs (MaskFormerForInstanceSegmentation) —
Raw outputs of the model.   target_sizes (List[Tuple[int, int]], optional) —
List of length (batch_size), where each list item (Tuple[int, int]]) corresponds to the request


MaskFormerImageProcessor

ed
final size (height, width) of each prediction. If left to None, predictions will not be resized.   Returns
List[torch.Tensor]

A list of length batch_size, where each item is a semantic segmentation map of shape (height, width)
corresponding to the target_sizes entry (if target_sizes is specified). Each entry of each
torch.Tensor correspond to a semantic class id.
  Converts the output of MaskFormerForInstanceSegmentation into semantic segmentation maps. Only supports
PyTorch.  post_process_instance_segmentation  < source > ( outputs threshold: float = 0.5 mask_threshold: float = 0.5 overlap_mask_area_threshold: float = 0.8 target_sizes: typing.Union[typing.List[typing.Tuple[int, int]], NoneType] = None return_coco_annotation: typing.Optional[bool] = False return_binary_maps: typing.Optional[bool] = False  ) → List[Dict]  Parameters   outputs (MaskFormerForInstanceSegmentation) —
Raw outputs of the model.   threshold (float, optional, defaults to 0.5) —
The probabili


MaskFormerImageProcessor

ty score threshold to keep predicted instance masks.   mask_threshold (float, optional, defaults to 0.5) —
Threshold to use when turning the predicted masks into binary values.   overlap_mask_area_threshold (float, optional, defaults to 0.8) —
The overlap mask area threshold to merge or discard small disconnected parts within each binary
instance mask.   target_sizes (List[Tuple], optional) —
List of length (batch_size), where each list item (Tuple[int, int]]) corresponds to the requested
final size (height, width) of each prediction. If left to None, predictions will not be resized.   return_coco_annotation (bool, optional, defaults to False) —
If set to True, segmentation maps are returned in COCO run-length encoding (RLE) format.   return_binary_maps (bool, optional, defaults to False) —
If set to True, segmentation maps are returned as a concatenated tensor of binary segmentation maps
(one per detected instance).   Returns
List[Dict]

A list of dictionaries, one per


MaskFormerImageProcessor

 image, each dictionary containing two keys:
segmentation — A tensor of shape (height, width) where each pixel represents a segment_id or
List[List] run-length encoding (RLE) of the segmentation map if return_coco_annotation is set to
True. Set to None if no mask if found above threshold.
segments_info — A dictionary that contains additional information on each segment.
id — An integer representing the segment_id.
label_id — An integer representing the label / semantic class id corresponding to segment_id.
score — Prediction score of segment with segment_id.
  Converts the output of MaskFormerForInstanceSegmentationOutput into instance segmentation predictions. Only
supports PyTorch.  post_process_panoptic_segmentation  < source > ( outputs threshold: float = 0.5 mask_threshold: float = 0.5 overlap_mask_area_threshold: float = 0.8 label_ids_to_fuse: typing.Optional[typing.Set[int]] = None target_sizes: typing.Union[typing.List[typing.Tuple[int, int]], NoneType] = None  ) →


MaskFormerImageProcessor

 List[Dict]  Parameters   outputs (MaskFormerForInstanceSegmentationOutput) —
The outputs from MaskFormerForInstanceSegmentation.   threshold (float, optional, defaults to 0.5) —
The probability score threshold to keep predicted instance masks.   mask_threshold (float, optional, defaults to 0.5) —
Threshold to use when turning the predicted masks into binary values.   overlap_mask_area_threshold (float, optional, defaults to 0.8) —
The overlap mask area threshold to merge or discard small disconnected parts within each binary
instance mask.   label_ids_to_fuse (Set[int], optional) —
The labels in this state will have all their instances be fused together. For instance we could say
there can only be one sky in an image, but several persons, so the label ID for sky would be in that
set, but not the one for person.   target_sizes (List[Tuple], optional) —
List of length (batch_size), where each list item (Tuple[int, int]]) corresponds to the requested
final size (height, widt


MaskFormerImageProcessor

h) of each prediction in batch. If left to None, predictions will not be
resized.   Returns
List[Dict]

A list of dictionaries, one per image, each dictionary containing two keys:
segmentation — a tensor of shape (height, width) where each pixel represents a segment_id, set
to None if no mask if found above threshold. If target_sizes is specified, segmentation is resized
to the corresponding target_sizes entry.
segments_info — A dictionary that contains additional information on each segment.
id — an integer representing the segment_id.
label_id — An integer representing the label / semantic class id corresponding to segment_id.
was_fused — a boolean, True if label_id was in label_ids_to_fuse, False otherwise.
Multiple instances of the same class / label were fused and assigned a single segment_id.
score — Prediction score of segment with segment_id.
  Converts the output of MaskFormerForInstanceSegmentationOutput into image panoptic segmentation
predictions. Only suppo


MaskFormerFeatureExtractor

class transformers.MaskFormerFeatureExtractor  < source > ( *args **kwargs  )     __call__  < source > ( images segmentation_maps = None **kwargs  )     encode_inputs  < source > ( pixel_values_list: typing.List[typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]]] segmentation_maps: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]] = None instance_id_to_semantic_id: typing.Union[typing.List[typing.Dict[int, int]], typing.Dict[int, int], NoneType] = None ignore_index: typing.Optional[int] = None reduce_labels: bool = False return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None input_data_format: typing.Union[str


MaskFormerFeatureExtractor

, transformers.image_utils.ChannelDimension, NoneType] = None  ) → BatchFeature  Parameters   pixel_values_list (List[ImageInput]) —
List of images (pixel values) to be padded. Each image should be a tensor of shape (channels, height, width).   segmentation_maps (ImageInput, optional) —
The corresponding semantic segmentation maps with the pixel-wise annotations.
(bool, optional, defaults to True):
Whether or not to pad images up to the largest image in a batch and create a pixel mask.
If left to the default, will return a pixel mask that is:
1 for pixels that are real (i.e. not masked),
0 for pixels that are padding (i.e. masked).
   instance_id_to_semantic_id (List[Dict[int, int]] or Dict[int, int], optional) —
A mapping between object instance ids and class ids. If passed, segmentation_maps is treated as an
instance segmentation map where each pixel represents an instance id. Can be provided as a single
dictionary with a global/datase


MaskFormerFeatureExtractor

t-level mapping or as a list of dictionaries (one per image), to map
instance ids in each image separately.   return_tensors (str or TensorType, optional) —
If set, will return tensors instead of NumPy arrays. If set to 'pt', return PyTorch torch.Tensor
objects.   Returns
BatchFeature

A BatchFeature with the following fields:
pixel_values — Pixel values to be fed to a model.
pixel_mask — Pixel mask to be fed to a model (when =True or if pixel_mask is in
self.model_input_names).
mask_labels — Optional list of mask labels of shape (labels, height, width) to be fed to a model
(when annotations are provided).
class_labels — Optional list of class labels of shape (labels) to be fed to a model (when
annotations are provided). They identify the labels of mask_labels, e.g. the label of
mask_labels[i][j] if class_labels[i][j].
  Pad images up to the largest image in a batch and create a corresponding pixel_mask. MaskFormer addresses semantic


MaskFormerFeatureExtractor

segmentation with a mask classification paradigm, thus input segmentation maps
will be converted to lists of binary masks and their respective labels. Let’s see an example, assuming
segmentation_maps = [[2,6,7,9]], the output will contain mask_labels = [[1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1]] (four binary masks) and class_labels = [2,6,7,9], the labels for
each mask.  post_process_semantic_segmentation  < source > ( outputs target_sizes: typing.Union[typing.List[typing.Tuple[int, int]], NoneType] = None  ) → List[torch.Tensor]  Parameters   outputs (MaskFormerForInstanceSegmentation) —
Raw outputs of the model.   target_sizes (List[Tuple[int, int]], optional) —
List of length (batch_size), where each list item (Tuple[int, int]]) corresponds to the requested
final size (height, width) of each prediction. If left to None, predictions will not be resized.   Returns
List[torch.Tensor]

A list of length batch_size, where each item is a se


MaskFormerFeatureExtractor

mantic segmentation map of shape (height, width)
corresponding to the target_sizes entry (if target_sizes is specified). Each entry of each
torch.Tensor correspond to a semantic class id.
  Converts the output of MaskFormerForInstanceSegmentation into semantic segmentation maps. Only supports
PyTorch.  post_process_instance_segmentation  < source > ( outputs threshold: float = 0.5 mask_threshold: float = 0.5 overlap_mask_area_threshold: float = 0.8 target_sizes: typing.Union[typing.List[typing.Tuple[int, int]], NoneType] = None return_coco_annotation: typing.Optional[bool] = False return_binary_maps: typing.Optional[bool] = False  ) → List[Dict]  Parameters   outputs (MaskFormerForInstanceSegmentation) —
Raw outputs of the model.   threshold (float, optional, defaults to 0.5) —
The probability score threshold to keep predicted instance masks.   mask_threshold (float, optional, defaults to 0.5) —
Threshold to use when turning the predicte


MaskFormerFeatureExtractor

d masks into binary values.   overlap_mask_area_threshold (float, optional, defaults to 0.8) —
The overlap mask area threshold to merge or discard small disconnected parts within each binary
instance mask.   target_sizes (List[Tuple], optional) —
List of length (batch_size), where each list item (Tuple[int, int]]) corresponds to the requested
final size (height, width) of each prediction. If left to None, predictions will not be resized.   return_coco_annotation (bool, optional, defaults to False) —
If set to True, segmentation maps are returned in COCO run-length encoding (RLE) format.   return_binary_maps (bool, optional, defaults to False) —
If set to True, segmentation maps are returned as a concatenated tensor of binary segmentation maps
(one per detected instance).   Returns
List[Dict]

A list of dictionaries, one per image, each dictionary containing two keys:
segmentation — A tensor of shape (height, width) where each pixel re


MaskFormerFeatureExtractor

presents a segment_id or
List[List] run-length encoding (RLE) of the segmentation map if return_coco_annotation is set to
True. Set to None if no mask if found above threshold.
segments_info — A dictionary that contains additional information on each segment.
id — An integer representing the segment_id.
label_id — An integer representing the label / semantic class id corresponding to segment_id.
score — Prediction score of segment with segment_id.
  Converts the output of MaskFormerForInstanceSegmentationOutput into instance segmentation predictions. Only
supports PyTorch.  post_process_panoptic_segmentation  < source > ( outputs threshold: float = 0.5 mask_threshold: float = 0.5 overlap_mask_area_threshold: float = 0.8 label_ids_to_fuse: typing.Optional[typing.Set[int]] = None target_sizes: typing.Union[typing.List[typing.Tuple[int, int]], NoneType] = None  ) → List[Dict]  Parameters   outputs (MaskFormerForInstanceSegmentationOutput) —


MaskFormerFeatureExtractor

The outputs from MaskFormerForInstanceSegmentation.   threshold (float, optional, defaults to 0.5) —
The probability score threshold to keep predicted instance masks.   mask_threshold (float, optional, defaults to 0.5) —
Threshold to use when turning the predicted masks into binary values.   overlap_mask_area_threshold (float, optional, defaults to 0.8) —
The overlap mask area threshold to merge or discard small disconnected parts within each binary
instance mask.   label_ids_to_fuse (Set[int], optional) —
The labels in this state will have all their instances be fused together. For instance we could say
there can only be one sky in an image, but several persons, so the label ID for sky would be in that
set, but not the one for person.   target_sizes (List[Tuple], optional) —
List of length (batch_size), where each list item (Tuple[int, int]]) corresponds to the requested
final size (height, width) of each prediction in batch. If left t


MaskFormerFeatureExtractor

o None, predictions will not be
resized.   Returns
List[Dict]

A list of dictionaries, one per image, each dictionary containing two keys:
segmentation — a tensor of shape (height, width) where each pixel represents a segment_id, set
to None if no mask if found above threshold. If target_sizes is specified, segmentation is resized
to the corresponding target_sizes entry.
segments_info — A dictionary that contains additional information on each segment.
id — an integer representing the segment_id.
label_id — An integer representing the label / semantic class id corresponding to segment_id.
was_fused — a boolean, True if label_id was in label_ids_to_fuse, False otherwise.
Multiple instances of the same class / label were fused and assigned a single segment_id.
score — Prediction score of segment with segment_id.
  Converts the output of MaskFormerForInstanceSegmentationOutput into image panoptic segmentation
predictions. Only supports P


MaskFormerModel

class transformers.MaskFormerModel  < source > ( config: MaskFormerConfig  )   Parameters   config (MaskFormerConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare MaskFormer Model outputting raw hidden-states without any specific head on top.
This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: Tensor pixel_mask: typing.Optional[torch.Tensor] = None output_hidden_states: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.maskformer.modeling_maskformer.MaskFormerModelOutput or tuple(torch.FloatTensor)  Parameters


MaskFormerModel

  pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
MaskFormerImageProcessor.call() for details.   pixel_mask (torch.LongTensor of shape (batch_size, height, width), optional) —
Mask to avoid performing attention on padding pixel values. Mask values selected in [0, 1]:
1 for pixels that are real (i.e. not masked),
0 for pixels that are padding (i.e. masked).
What are attention masks?   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of Detr’s decoder attention layers.   return_dict (bool, optional) —
Whether or not to return a ~MaskFormerModelOutput instead of a plain tuple.   Returns
transformers.models.maskformer.modeling_maskformer.MaskFormerModelOutput or tuple(tor


MaskFormerModel

ch.FloatTensor)

A transformers.models.maskformer.modeling_maskformer.MaskFormerModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (MaskFormerConfig) and inputs.
encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, num_channels, height, width)) — Last hidden states (final feature map) of the last stage of the encoder model (backbone).
pixel_decoder_last_hidden_state (torch.FloatTensor of shape (batch_size, num_channels, height, width)) — Last hidden states (final feature map) of the last stage of the pixel decoder model (FPN).
transformer_decoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Last hidden states (final feature map) of the last stage of the transformer decoder model.
encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or


MaskFormerModel

when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the encoder
model at the output of each stage.
pixel_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the pixel
decoder model at the output of each stage.
transformer_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size,


MaskFormerModel

sequence_length, hidden_size). Hidden-states (also called feature maps) of the
transformer decoder at the output of each stage.
hidden_states tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor containing encoder_hidden_states, pixel_decoder_hidden_states and
decoder_hidden_states
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Attentions weights from Detr’s decoder after the attention softmax, used to compute the
weighted average in the self-attention heads.
  The MaskFormerModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instea


MaskFormerModel

d of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoImageProcessor, MaskFormerModel
>>> from PIL import Image
>>> import requests
>>> # load MaskFormer fine-tuned on ADE20k semantic segmentation
>>> image_processor = AutoImageProcessor.from_pretrained("facebook/maskformer-swin-base-ade")
>>> model = MaskFormerModel.from_pretrained("facebook/maskformer-swin-base-ade")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = image_processor(image, return_tensors="pt")
>>> # forward pass
>>> outputs = model(**inputs)
>>> # the decoder of MaskFormer outputs hidden states of shape (batch_size, num_queries, hidden_size)
>>> transformer_decoder_last_hidden_state = outputs.transformer_decoder_last_hidden_state
>>> list(transformer_decoder_last_hidden_state.shape)
[1, 100, 2


MaskFormerForInstanceSegmentation

class transformers.MaskFormerForInstanceSegmentation  < source > ( config: MaskFormerConfig  )     forward  < source > ( pixel_values: Tensor mask_labels: typing.Optional[typing.List[torch.Tensor]] = None class_labels: typing.Optional[typing.List[torch.Tensor]] = None pixel_mask: typing.Optional[torch.Tensor] = None output_auxiliary_logits: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.maskformer.modeling_maskformer.MaskFormerForInstanceSegmentationOutput or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
MaskFormerImageProcessor.call() for details.   pixel_mask (torch.LongTensor of shape (batch_size, he


MaskFormerForInstanceSegmentation

ight, width), optional) —
Mask to avoid performing attention on padding pixel values. Mask values selected in [0, 1]:
1 for pixels that are real (i.e. not masked),
0 for pixels that are padding (i.e. masked).
What are attention masks?   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of Detr’s decoder attention layers.   return_dict (bool, optional) —
Whether or not to return a ~MaskFormerModelOutput instead of a plain tuple.   mask_labels (List[torch.Tensor], optional) —
List of mask labels of shape (num_labels, height, width) to be fed to a model   class_labels (List[torch.LongTensor], optional) —
list of target class labels of shape (num_labels, height, width) to be fed to a model. They identify the
lab


MaskFormerForInstanceSegmentation

els of mask_labels, e.g. the label of mask_labels[i][j] if class_labels[i][j].   Returns
transformers.models.maskformer.modeling_maskformer.MaskFormerForInstanceSegmentationOutput or tuple(torch.FloatTensor)

A transformers.models.maskformer.modeling_maskformer.MaskFormerForInstanceSegmentationOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (MaskFormerConfig) and inputs.
loss (torch.Tensor, optional) — The computed loss, returned when labels are present.
class_queries_logits (torch.FloatTensor) — A tensor of shape (batch_size, num_queries, num_labels + 1) representing the proposed classes for each
query. Note the + 1 is needed because we incorporate the null class.
masks_queries_logits (torch.FloatTensor) — A tensor of shape (batch_size, num_queries, height, width) representing th


MaskFormerForInstanceSegmentation

e proposed masks for each
query.
encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, num_channels, height, width)) — Last hidden states (final feature map) of the last stage of the encoder model (backbone).
pixel_decoder_last_hidden_state (torch.FloatTensor of shape (batch_size, num_channels, height, width)) — Last hidden states (final feature map) of the last stage of the pixel decoder model (FPN).
transformer_decoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Last hidden states (final feature map) of the last stage of the transformer decoder model.
encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, num_channels,


MaskFormerForInstanceSegmentation

 height, width). Hidden-states (also called feature maps) of the encoder
model at the output of each stage.
pixel_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the pixel
decoder model at the output of each stage.
transformer_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, sequence_length, hidden_size). Hidden-states of the transformer decoder at the output
of each stage.
hidd


MaskFormerForInstanceSegmentation

en_states tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor containing encoder_hidden_states, pixel_decoder_hidden_states and
decoder_hidden_states.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Attentions weights from Detr’s decoder after the attention softmax, used to compute the
weighted average in the self-attention heads.
  The MaskFormerForInstanceSegmentation forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of


MaskFormerForInstanceSegmentation

running the pre and post processing steps while
the latter silently ignores them. Examples:  Semantic segmentation example:   Copied >>> from transformers import AutoImageProcessor, MaskFormerForInstanceSegmentation
>>> from PIL import Image
>>> import requests
>>> # load MaskFormer fine-tuned on ADE20k semantic segmentation
>>> image_processor = AutoImageProcessor.from_pretrained("facebook/maskformer-swin-base-ade")
>>> model = MaskFormerForInstanceSegmentation.from_pretrained("facebook/maskformer-swin-base-ade")
>>> url = (
...     "https://huggingface.co/datasets/hf-internal-testing/fixtures_ade20k/resolve/main/ADE_val_00000001.jpg"
... )
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = image_processor(images=image, return_tensors="pt")
>>> outputs = model(**inputs)
>>> # model predicts class_queries_logits of shape `(batch_size, num_queries)`
>>> # and masks_queries_logi


MaskFormerForInstanceSegmentation

ts of shape `(batch_size, num_queries, height, width)`
>>> class_queries_logits = outputs.class_queries_logits
>>> masks_queries_logits = outputs.masks_queries_logits
>>> # you can pass them to image_processor for postprocessing
>>> predicted_semantic_map = image_processor.post_process_semantic_segmentation(
...     outputs, target_sizes=[image.size[::-1]]
... )[0]
>>> # we refer to the demo notebooks for visualization (see "Resources" section in the MaskFormer docs)
>>> list(predicted_semantic_map.shape)
[512, 683]  Panoptic segmentation example:   Copied >>> from transformers import AutoImageProcessor, MaskFormerForInstanceSegmentation
>>> from PIL import Image
>>> import requests
>>> # load MaskFormer fine-tuned on COCO panoptic segmentation
>>> image_processor = AutoImageProcessor.from_pretrained("facebook/maskformer-swin-base-coco")
>>> model = MaskFormerForInstanceSegmentation.from_pretraine


MaskFormerForInstanceSegmentation

d("facebook/maskformer-swin-base-coco")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = image_processor(images=image, return_tensors="pt")
>>> outputs = model(**inputs)
>>> # model predicts class_queries_logits of shape `(batch_size, num_queries)`
>>> # and masks_queries_logits of shape `(batch_size, num_queries, height, width)`
>>> class_queries_logits = outputs.class_queries_logits
>>> masks_queries_logits = outputs.masks_queries_logits
>>> # you can pass them to image_processor for postprocessing
>>> result = image_processor.post_process_panoptic_segmentation(outputs, target_sizes=[image.size[::-1]])[0]
>>> # we refer to the demo notebooks for visualization (see "Resources" section in the MaskFormer docs)
>>> predicted_panoptic_map = result["segmentation"]
>>> list(predicted_panoptic_map.shape)
[480, 640]


Model Details

Mistral-7B-v0.1 is a decoder-based LM with the following architectural choices:
Sliding Window Attention - Trained with 8k context length and fixed cache size, with a theoretical attention span of 128K tokens GQA (Grouped Query Attention) - allowing faster inference and lower cache size. Byte-fallback BPE tokenizer - ensures that characters are never mapped to out of vocabulary tokens.
We also provide an instruction fine-tuned model: Mistral-7B-Instruct-v0.1 which can be used for chat-based inference.
For more details please read our release blog post