MobileNetV2Model

onfiguration (MobileNetV2Config) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, num_channels, height, width)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — Last layer hidden-state after a pooling operation on the spatial dimensions.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, num_channels, height, width).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
  The MobileNetV2Model forward method, overrides the __call__ special method.


MobileNetV2Model

Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoImageProcessor, MobileNetV2Model
>>> import torch
>>> from datasets import load_dataset
>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]
>>> image_processor = AutoImageProcessor.from_pretrained("google/mobilenet_v2_1.0_224")
>>> model = MobileNetV2Model.from_pretrained("google/mobilenet_v2_1.0_224")
>>> inputs = image_processor(image, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1, 1280, 7, 7]


class transformers.MobileNetV2ForImageClassification  < source > ( config: MobileNetV2Config  )   Parameters   config (MobileNetV2Config) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    MobileNetV2 model with an image classification head on top (a linear layer on top of the pooled features), e.g. for
ImageNet. This model is a PyTorch torch.nn.Module subclass. Use it
as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: typing.Optional[torch.Tensor] = None output_hidden_states: typing.Optional[bool] = None labels: typing.Optional[torch.Tensor] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.ImageClassifierOutputWithNoAttention or tuple(torch.Floa


tTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
MobileNetV2ImageProcessor.call() for details.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the image classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss). If
config.num_labels > 1 a classification loss is computed (Cross-Entropy).   Returns
transformers.modeling_outputs.ImageClassifierOutputWithNoAttention or tuple(torch.FloatTensor)

A transformers.modeling_outputs.ImageClassifierOutputWithNoAtten


tion or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (MobileNetV2Config) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each stage) of shape (batch_size, num_channels, height, width). Hidden-states (also
called feature maps) of the model at the output of each stage.
  The MobileNetV2ForImageClassification forward method, overrides the __call__ s


pecial method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoImageProcessor, MobileNetV2ForImageClassification
>>> import torch
>>> from datasets import load_dataset
>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]
>>> image_processor = AutoImageProcessor.from_pretrained("google/mobilenet_v2_1.0_224")
>>> model = MobileNetV2ForImageClassification.from_pretrained("google/mobilenet_v2_1.0_224")
>>> inputs = image_processor(image, return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> # model predicts one of the 1000 ImageNet classes
>>> predicted_label = logits.argmax(-1).item()
>>> print(model.config.id2label[predicted_label])
tabby, tabby c


class transformers.MobileNetV2ForSemanticSegmentation  < source > ( config: MobileNetV2Config  )   Parameters   config (MobileNetV2Config) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    MobileNetV2 model with a semantic segmentation head on top, e.g. for Pascal VOC. This model is a PyTorch torch.nn.Module subclass. Use it
as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.Sem


anticSegmenterOutput or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
MobileNetV2ImageProcessor.call() for details.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size, height, width), optional) —
Ground truth semantic segmentation maps for computing the loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels > 1, a classification loss is computed (Cross-Entropy).   Returns
transformers.modeling_outputs.SemanticSegmenterOutput or tuple(torch.FloatTensor)


A transformers.modeling_outputs.SemanticSegmenterOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (MobileNetV2Config) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, config.num_labels, logits_height, logits_width)) — Classification scores for each pixel.
The logits returned do not necessarily have the same size as the pixel_values passed as inputs. This is
to avoid doing two interpolations and lose some quality when a user needs to resize the logits to the
original image size as post-processing. You should always check your logits shape and resize as needed.
hidden_states (tuple(torch.FloatTensor), optional, ret


urned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, patch_size, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, patch_size, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The MobileNetV2ForSemanticSegmentation forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined


within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoImageProcessor, MobileNetV2ForSemanticSegmentation
>>> from PIL import Image
>>> import requests
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> image_processor = AutoImageProcessor.from_pretrained("google/deeplabv3_mobilenet_v2_1.0_513")
>>> model = MobileNetV2ForSemanticSegmentation.from_pretrained("google/deeplabv3_mobilenet_v2_1.0_513")
>>> inputs = image_processor(images=image, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> # logits are of shape (batch_size, num_labels, height, width)
>>> logits = outputs.logit


Overview

The XLSR-Wav2Vec2 model was proposed in Unsupervised Cross-Lingual Representation Learning For Speech Recognition by Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, Michael
Auli.
The abstract from the paper is the following:
This paper presents XLSR which learns cross-lingual speech representations by pretraining a single model from the raw
waveform of speech in multiple languages. We build on wav2vec 2.0 which is trained by solving a contrastive task over
masked latent speech representations and jointly learns a quantization of the latents shared across languages. The
resulting model is fine-tuned on labeled data and experiments show that cross-lingual pretraining significantly
outperforms monolingual pretraining. On the CommonVoice benchmark, XLSR shows a relative phoneme error rate reduction
of 72% compared to the best known results. On BABEL, our


Overview

approach improves word error rate by 16% relative compared to
a comparable system. Our approach enables a single multilingual speech recognition model which is competitive to strong
individual models. Analysis shows that the latent discrete speech representations are shared across languages with
increased sharing for related languages. We hope to catalyze research in low-resource speech understanding by releasing
XLSR-53, a large model pretrained in 53 languages.
Tips:
XLSR-Wav2Vec2 is a speech model that accepts a float array corresponding to the raw waveform of the speech signal. XLSR-Wav2Vec2 model was trained using connectionist temporal classification (CTC) so the model output has to be
decoded using Wav2Vec2CTCTokenizer.
XLSR-Wav2Vec2’s architecture is based on the Wav2Vec2 model, so one can refer to Wav2Vec2’s documentation page.
The original code can be found here.


Overview

MatCha has been proposed in the paper MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering, from Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, Julian Martin Eisenschlos.
The abstract of the paper states the following:
Visual language data such as plots, charts, and infographics are ubiquitous in the human world. However, state-of-the-art vision-language models do not perform well on these data. We propose MatCha (Math reasoning and Chart derendering pretraining) to enhance visual language models’ capabilities in jointly modeling charts/plots and lang


Overview

uage data. Specifically, we propose several pretraining tasks that cover plot deconstruction and numerical reasoning which are the key capabilities in visual language modeling. We perform the MatCha pretraining starting from Pix2Struct, a recently proposed image-to-text visual language model. On standard benchmarks such as PlotQA and ChartQA, the MatCha model outperforms state-of-the-art methods by as much as nearly 20%. We also examine how well MatCha pretraining transfers to domains such as screenshots, textbook diagrams, and document figures and observe overall improvement, verifying the usefulness of MatCha pretraining on broader visual language tasks


Model description

MatCha is a model that is trained using Pix2Struct architecture. You can find more information about Pix2Struct in the Pix2Struct documentation.
MatCha is a Visual Question Answering subset of Pix2Struct architecture. It renders the input question on the image and predicts the answer.


Usage

Currently 6 checkpoints are available for MatCha:
google/matcha: the base MatCha model, used to fine-tune MatCha on downstream tasks google/matcha-chartqa: MatCha model fine-tuned on ChartQA dataset. It can be used to answer questions about charts. google/matcha-plotqa-v1: MatCha model fine-tuned on PlotQA dataset. It can be used to answer questions about plots. google/matcha-plotqa-v2: MatCha model fine-tuned on PlotQA dataset. It can be used to answer questions about plots. google/matcha-chart2text-statista: MatCha model fine-tuned on Statista dataset. google/matcha-chart2text-pew: MatCha model fine-tuned on Pew dataset.
The models finetuned on chart2text-pew and chart2text-statista are more suited for summarization, whereas the models finetuned on plotqa and chartqa are


Usage

 more suited for question answering.
You can use these models as follows (example on a ChatQA dataset):
  Copied from transformers import AutoProcessor, Pix2StructForConditionalGeneration
import requests
from PIL import Image
model = Pix2StructForConditionalGeneration.from_pretrained("google/matcha-chartqa").to(0)
processor = AutoProcessor.from_pretrained("google/matcha-chartqa")
url = "https://raw.githubusercontent.com/vis-nlp/ChartQA/main/ChartQA%20Dataset/val/png/20294671002019.png"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(images=image, text="Is the sum of all 4 places greater than Laos?", return_tensors="pt").to(0)
predictions = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(predictions[0], skip_special_tokens=True)


Fine-tuning

To fine-tune MatCha, refer to the pix2struct fine-tuning notebook. For Pix2Struct models, we have found out that fine-tuning the model with Adafactor and cosine learning rate scheduler leads to faste convergence:
  Copied from transformers.optimization import Adafactor, get_cosine_schedule_with_warmup
optimizer = Adafactor(self.parameters(), scale_parameter=False, relative_step=False, lr=0.01, weight_decay=1e-05)
scheduler = get_cosine_schedule_with_warmup(optimizer, num_warmup_steps=1000, num_training_steps=40000)


Overview

The ResNet model was proposed in Deep Residual Learning for Image Recognition by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. Our implementation follows the small changes made by Nvidia, we apply the stride=2 for downsampling in bottleneck’s 3x3 conv and not in the first 1x1. This is generally known as “ResNet v1.5”.
ResNet introduced residual connections, they allow to train networks with an unseen number of layers (up to 1000). ResNet won the 2015 ILSVRC & COCO competition, one important milestone in deep computer vision.
The abstract from the paper is the following:
Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are sub


Overview

stantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers.
The dept


Overview

h of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.
Tips:
One can use AutoImageProcessor to prepare images for the model.
The figure below illustrates the architecture of ResNet. Taken from the original paper.
This model was contributed by Francesco. The TensorFlow version of this model was added by amyeroberts. The original code can be found her


Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ResNet.
 Image Classification
ResNetForImageClassification is supported by this example script and notebook. See also: Image classification task guide
If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we’ll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


ResNetConfig

class transformers.ResNetConfig  < source > ( num_channels = 3 embedding_size = 64 hidden_sizes = [256, 512, 1024, 2048] depths = [3, 4, 6, 3] layer_type = 'bottleneck' hidden_act = 'relu' downsample_in_first_stage = False out_features = None out_indices = None **kwargs  )   Parameters   num_channels (int, optional, defaults to 3) —
The number of input channels.   embedding_size (int, optional, defaults to 64) —
Dimensionality (hidden size) for the embedding layer.   hidden_sizes (List[int], optional, defaults to [256, 512, 1024, 2048]) —
Dimensionality (hidden size) at each stage.   depths (List[int], optional, defaults to [3, 4, 6, 3]) —
Depth (number of layers) for each stage.   layer_type (str, optional, defaults to "bottleneck") —
The layer to use, it can be either "basic" (used for smaller models, like resnet-18 or resnet-34) or
"bottleneck" (used for larger models like resnet-50 and ab


ResNetConfig

ove).   hidden_act (str, optional, defaults to "relu") —
The non-linear activation function in each block. If string, "gelu", "relu", "selu" and "gelu_new"
are supported.   downsample_in_first_stage (bool, optional, defaults to False) —
If True, the first stage will downsample the inputs using a stride of 2.   out_features (List[str], optional) —
If used as backbone, list of features to output. Can be any of "stem", "stage1", "stage2", etc.
(depending on how many stages the model has). If unset and out_indices is set, will default to the
corresponding stages. If unset and out_indices is unset, will default to the last stage.   out_indices (List[int], optional) —
If used as backbone, list of indices of features to output. Can be any of 0, 1, 2, etc. (depending on how
many stages the model has). If unset and out_features is set, will default to the corresponding stages.
If unset and out_feature


ResNetConfig

s is unset, will default to the last stage.    This is the configuration class to store the configuration of a ResNetModel. It is used to instantiate an
ResNet model according to the specified arguments, defining the model architecture. Instantiating a configuration
with the defaults will yield a similar configuration to that of the ResNet
microsoft/resnet-50 architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import ResNetConfig, ResNetModel
>>> # Initializing a ResNet resnet-50 style configuration
>>> configuration = ResNetConfig()
>>> # Initializing a model (with random weights) from the resnet-50 style configuration
>>> model = ResNetModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.conf


ResNetModel

class transformers.ResNetModel  < source > ( config  )   Parameters   config (ResNetConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare ResNet model outputting raw features without any specific head on top.
This model is a PyTorch torch.nn.Module subclass. Use it
as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: Tensor output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.BaseModelOutputWithPoolingAndNoAttention or tuple(torch.FloatTensor)


ResNetModel

 Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
ConvNextImageProcessor.call() for details.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.BaseModelOutputWithPoolingAndNoAttention or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutputWithPoolingAndNoAttention or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (ResNetConfig) and i


ResNetModel

nputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, num_channels, height, width)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — Last layer hidden-state after a pooling operation on the spatial dimensions.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, num_channels, height, width).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
  The ResNetModel forward method, overrides the __call__ special method. Although


ResNetModel

 the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoImageProcessor, ResNetModel
>>> import torch
>>> from datasets import load_dataset
>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]
>>> image_processor = AutoImageProcessor.from_pretrained("microsoft/resnet-50")
>>> model = ResNetModel.from_pretrained("microsoft/resnet-50")
>>> inputs = image_processor(image, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1, 2048, 7, 7


ResNetForImageClassification

class transformers.ResNetForImageClassification  < source > ( config  )   Parameters   config (ResNetConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    ResNet Model with an image classification head on top (a linear layer on top of the pooled features), e.g. for
ImageNet. This model is a PyTorch torch.nn.Module subclass. Use it
as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.LongTensor] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.ImageClassifierOutputWithNoAttention or tupl


ResNetForImageClassification

e(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
ConvNextImageProcessor.call() for details.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the image classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels > 1 a classification loss is computed (Cross-Entropy).   Returns
transformers.modeling_outputs.ImageClassifierOutputWithNoAttention or tuple(torch.FloatTensor)

A transformers.modeling_outputs.ImageClassifierOutputWithNoAttention or a tuple of
torch.Float


ResNetForImageClassification

Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (ResNetConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each stage) of shape (batch_size, num_channels, height, width). Hidden-states (also
called feature maps) of the model at the output of each stage.
  The ResNetForImageClassification forward method, overrides the __call__ spec


ResNetForImageClassification

ial method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoImageProcessor, ResNetForImageClassification
>>> import torch
>>> from datasets import load_dataset
>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]
>>> image_processor = AutoImageProcessor.from_pretrained("microsoft/resnet-50")
>>> model = ResNetForImageClassification.from_pretrained("microsoft/resnet-50")
>>> inputs = image_processor(image, return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> # model predicts one of the 1000 ImageNet classes
>>> predicted_label = logits.argmax(-1).item()
>>> print(model.config.id2label[predicted_label])
tiger


TFResNetModel

class transformers.TFResNetModel  < source > ( *args **kwargs  )   Parameters   config (ResNetConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare ResNet model outputting raw features without any specific head on top.
This model is a TensorFlow
tf.keras.layers.Layer sub-class. Use it as a
regular TensorFlow Module and refer to the TensorFlow documentation for all matter related to general usage and
behavior.  call  < source > ( pixel_values: Tensor output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None training: bool = False  ) → transformers.modeling_tf_outputs.TFBaseMod


TFResNetModel

elOutputWithPoolingAndNoAttention or tuple(tf.Tensor)  Parameters   pixel_values (tf.Tensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
ConvNextImageProcessor.call() for details.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_tf_outputs.TFBaseModelOutputWithPoolingAndNoAttention or tuple(tf.Tensor)

A transformers.modeling_tf_outputs.TFBaseModelOutputWithPoolingAndNoAttention or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements


TFResNetModel

depending on the
configuration (ResNetConfig) and inputs.
last_hidden_state (tf.Tensor of shape (batch_size, num_channels, height, width)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (tf.Tensor of shape (batch_size, hidden_size)) — Last layer hidden-state after a pooling operation on the spatial dimensions.
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings, if the model has an embedding layer, + one for
the output of each layer) of shape (batch_size, num_channels, height, width).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
  The TFResNetModel forward method, overrides the _


TFResNetModel

_call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoImageProcessor, TFResNetModel
>>> from datasets import load_dataset
>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]
>>> image_processor = AutoImageProcessor.from_pretrained("microsoft/resnet-50")
>>> model = TFResNetModel.from_pretrained("microsoft/resnet-50")
>>> inputs = image_processor(image, return_tensors="tf")
>>> outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1, 2048, 7, 7


TFResNetForImageClassification

class transformers.TFResNetForImageClassification  < source > ( *args **kwargs  )   Parameters   config (ResNetConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    ResNet Model with an image classification head on top (a linear layer on top of the pooled features), e.g. for
ImageNet. This model is a TensorFlow
tf.keras.layers.Layer sub-class. Use it as a
regular TensorFlow Module and refer to the TensorFlow documentation for all matter related to general usage and
behavior.  call  < source > ( pixel_values: Tensor = None labels: Tensor = None output_hidden_states: bool = None return_dict: bool = None training: bool = False  ) → transformers.modeling_tf_outputs.TFImageClassifierOutputWithNoAttention or tuple(tf.Tensor)


TFResNetForImageClassification

 Parameters   pixel_values (tf.Tensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
ConvNextImageProcessor.call() for details.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (tf.Tensor of shape (batch_size,), optional) —
Labels for computing the image classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels > 1 a classification loss is computed (Cross-Entropy).   Returns
transformers.modeling_tf_outputs.TFImageClassifierOutputWithNoAttention or tuple(tf.Tensor)

A transformers.modeling_tf_outputs.TFImageClassifierOutputWithNoAttention or a tuple of tf.Tensor (if
return


TFResNetForImageClassification

_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (ResNetConfig) and inputs.
loss (tf.Tensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (tf.Tensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings, if the model has an embedding layer, + one for
the output of each stage) of shape (batch_size, num_channels, height, width). Hidden-states (also called
feature maps) of the model at the output of each stage.
  The TFResNetForImageClassification forward method, overrides the __call__ special method. Although t


TFResNetForImageClassification

he recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoImageProcessor, TFResNetForImageClassification
>>> import tensorflow as tf
>>> from datasets import load_dataset
>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]
>>> image_processor = AutoImageProcessor.from_pretrained("microsoft/resnet-50")
>>> model = TFResNetForImageClassification.from_pretrained("microsoft/resnet-50")
>>> inputs = image_processor(image, return_tensors="tf")
>>> logits = model(**inputs).logits
>>> # model predicts one of the 1000 ImageNet classes
>>> predicted_label = int(tf.math.argmax(logits, axis=-1))
>>> print(model.config.id2label[predicted_label])
tiger


FlaxResNetModel

class transformers.FlaxResNetModel  < source > ( config: ResNetConfig input_shape = (1, 224, 224, 3) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True **kwargs  )   Parameters   config (ResNetConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.   dtype (jax.numpy.dtype, optional, defaults to jax.numpy.float32) —
The data type of the computation. Can be one of jax.numpy.float32, jax.numpy.float16 (on GPUs) and
jax.numpy.bfloat16 (on TPUs).
This can be used to enable mixed-precision training or half-precision inference on GPUs or TPUs. If
specified all the computation will


FlaxResNetModel

be performed with the given dtype.
Note that this only specifies the dtype of the computation and does not influence the dtype of model
parameters.
If you wish to change the dtype of the model parameters, see to_fp16() and
to_bf16().    The bare ResNet model outputting raw features without any specific head on top. This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading, saving and converting weights from PyTorch models) This model is also a Flax Linen flax.linen.Module
subclass. Use it as a regular Flax linen Module and refer to the Flax documentation for all matter related to
general usage and behavior. Finally, this model supports inherent JAX features such as: Just-In-Time (JIT)


FlaxResNetModel

compilation Automatic Differentiation Vectorization Parallelization  __call__  < source > ( pixel_values params: dict = None train: bool = False output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_flax_outputs.FlaxBaseModelOutputWithPoolingAndNoAttention or tuple(torch.FloatTensor)  Returns
transformers.modeling_flax_outputs.FlaxBaseModelOutputWithPoolingAndNoAttention or tuple(torch.FloatTensor)

A transformers.modeling_flax_outputs.FlaxBaseModelOutputWithPoolingAndNoAttention or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.resnet.configuration_resnet.ResNetConfig'>) and inputs.
last_hi


FlaxResNetModel

dden_state (jnp.ndarray of shape (batch_size, num_channels, height, width)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (jnp.ndarray of shape (batch_size, hidden_size)) — Last layer hidden-state after a pooling operation on the spatial dimensions.
hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings, if the model has an embedding layer, + one
for the output of each layer) of shape (batch_size, num_channels, height, width). Hidden-states of the
model at the output of each layer plus the optional initial embedding outputs.
  The FlaxResNetPreTrainedModel forward method, overrides the __call__ special method.


FlaxResNetModel

Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoImageProcessor, FlaxResNetModel
>>> from PIL import Image
>>> import requests
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> image_processor = AutoImageProcessor.from_pretrained("microsoft/resnet-50")
>>> model = FlaxResNetModel.from_pretrained("microsoft/resnet-50")
>>> inputs = image_processor(images=image, return_tensors="np")
>>> outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_state


FlaxResNetForImageClassification

class transformers.FlaxResNetForImageClassification  < source > ( config: ResNetConfig input_shape = (1, 224, 224, 3) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True **kwargs  )   Parameters   config (ResNetConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.   dtype (jax.numpy.dtype, optional, defaults to jax.numpy.float32) —
The data type of the computation. Can be one of jax.numpy.float32, jax.numpy.float16 (on GPUs) and
jax.numpy.bfloat16 (on TPUs).
This can be used to enable mixed-precision training or half-precision inference on GPUs or TPUs. If
specified all the computation will b


FlaxResNetForImageClassification

e performed with the given dtype.
Note that this only specifies the dtype of the computation and does not influence the dtype of model
parameters.
If you wish to change the dtype of the model parameters, see to_fp16() and
to_bf16().    ResNet Model with an image classification head on top (a linear layer on top of the pooled features), e.g. for
ImageNet. This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading, saving and converting weights from PyTorch models) This model is also a Flax Linen flax.linen.Module
subclass. Use it as a regular Flax linen Module and refer to the Flax documentation for all matter related to
general usage and behavior. Finally, this model supports inherent JAX features such a


FlaxResNetForImageClassification

s: Just-In-Time (JIT) compilation Automatic Differentiation Vectorization Parallelization  __call__  < source > ( pixel_values params: dict = None train: bool = False output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_flax_outputs.FlaxImageClassifierOutputWithNoAttention or tuple(torch.FloatTensor)  Returns
transformers.modeling_flax_outputs.FlaxImageClassifierOutputWithNoAttention or tuple(torch.FloatTensor)

A transformers.modeling_flax_outputs.FlaxImageClassifierOutputWithNoAttention or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.resnet.configuration_resnet.ResNetConfig'>) and inputs.
logits (jnp.nda


FlaxResNetForImageClassification

rray of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when
config.output_hidden_states=True):
Tuple of jnp.ndarray (one for the output of the embeddings, if the model has an embedding layer, + one
for the output of each stage) of shape (batch_size, num_channels, height, width). Hidden-states (also
called feature maps) of the model at the output of each stage.
  The FlaxResNetPreTrainedModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post p


FlaxResNetForImageClassification

rocessing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoImageProcessor, FlaxResNetForImageClassification
>>> from PIL import Image
>>> import jax
>>> import requests
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> image_processor = AutoImageProcessor.from_pretrained("microsoft/resnet-50")
>>> model = FlaxResNetForImageClassification.from_pretrained("microsoft/resnet-50")
>>> inputs = image_processor(images=image, return_tensors="np")
>>> outputs = model(**inputs)
>>> logits = outputs.logits
>>> # model predicts one of the 1000 ImageNet classes
>>> predicted_class_idx = jax.numpy.argmax(logits, axis=-1)
>>> print("Predicted class:", model.config.id2label[predicted_class_idx.item(


Overview

The HerBERT model was proposed in KLEJ: Comprehensive Benchmark for Polish Language Understanding by Piotr Rybak, Robert Mroczkowski, Janusz Tracz, and
Ireneusz Gawlik. It is a BERT-based Language Model trained on Polish Corpora using only MLM objective with dynamic
masking of whole words.
The abstract from the paper is the following:
In recent years, a series of Transformer-based models unlocked major improvements in general natural language
understanding (NLU) tasks. Such a fast pace of research would not be possible without general NLU benchmarks, which
allow for a fair comparison of the proposed methods. However, such benchmarks are available only for a handful of
languages. To alleviate this issue, we introduce a comprehensive multi-task benchmark for the Polish language
understa


Overview

nding, accompanied by an online leaderboard. It consists of a diverse set of tasks, adopted from existing
datasets for named entity recognition, question-answering, textual entailment, and others. We also introduce a new
sentiment analysis task for the e-commerce domain, named Allegro Reviews (AR). To ensure a common evaluation scheme and
promote models that generalize to different NLU tasks, the benchmark includes datasets from varying domains and
applications. Additionally, we release HerBERT, a Transformer-based model trained specifically for the Polish language,
which has the best average performance and obtains the best results for three out of nine tasks. Finally, we provide an
extensive evaluation, including several standard baselines and recently proposed, multilingual Transfo


Overview

rmer-based
models.
Examples of use:
  Copied >>> from transformers import HerbertTokenizer, RobertaModel
>>> tokenizer = HerbertTokenizer.from_pretrained("allegro/herbert-klej-cased-tokenizer-v1")
>>> model = RobertaModel.from_pretrained("allegro/herbert-klej-cased-v1")
>>> encoded_input = tokenizer.encode("Kto ma lepszą sztukę, ma lepszy rząd – to jasne.", return_tensors="pt")
>>> outputs = model(encoded_input)
>>> # HerBERT can also be loaded using AutoTokenizer and AutoModel:
>>> import torch
>>> from transformers import AutoModel, AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("allegro/herbert-klej-cased-tokenizer-v1")
>>> model = AutoModel.from_pretrained("allegro/herbert-klej-cased-v1")
This model was contributed by rmroczkowski. The original code can be found
here.


HerbertTokenizer

class transformers.HerbertTokenizer  < source > ( vocab_file merges_file tokenizer_file = None cls_token = '<s>' unk_token = '<unk>' pad_token = '<pad>' mask_token = '<mask>' sep_token = '</s>' bos_token = '<s>' do_lowercase_and_remove_accent = False additional_special_tokens = ['<special0>', '<special1>', '<special2>', '<special3>', '<special4>', '<special5>', '<special6>', '<special7>', '<special8>', '<special9>'] lang2id = None id2lang = None **kwargs  )    Construct a BPE tokenizer for HerBERT. Peculiarities: uses BERT’s pre-tokenizer: BaseTokenizer splits tokens on spaces, and also on punctuation. Each occurrence of a
punctuation character will be treated separately. Such pretokenized input is BPE subtokenized This tokenizer inherits from XLMTokenizer which


HerbertTokenizer

contains most of the methods. Users should refer to the
superclass for more information regarding methods.  build_inputs_with_special_tokens  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None  ) → List[int]  Parameters   token_ids_0 (List[int]) —
List of IDs to which the special tokens will be added.   token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.   Returns
List[int]

List of input IDs with the appropriate special tokens.
  Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. An XLM sequence has the following format: single sequence: <s> X </s> pair of sequences: <s> A </s> B </s>  convert_tokens


HerbertTokenizer

_to_string  < source > ( tokens  )    Converts a sequence of tokens (string) in a single string.  create_token_type_ids_from_sequences  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None  ) → List[int]  Parameters   token_ids_0 (List[int]) —
List of IDs.   token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.   Returns
List[int]

List of token type IDs according to the given sequence(s).
  Create a mask from the two sequences passed to be used in a sequence-pair classification task. An XLM sequence  pair mask has the following format:   Copied 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
| first sequence    | second sequence | If token_ids_1 is None, this method only returns the first porti


HerbertTokenizer

on of the mask (0s).  get_special_tokens_mask  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None already_has_special_tokens: bool = False  ) → List[int]  Parameters   token_ids_0 (List[int]) —
List of IDs.   token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.   already_has_special_tokens (bool, optional, defaults to False) —
Whether or not the token list is already formatted with special tokens for the model.   Returns
List[int]

A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
  Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer prepare_for_model method.


HerbertTokenizerFast

class transformers.HerbertTokenizerFast  < source > ( vocab_file = None merges_file = None tokenizer_file = None cls_token = '<s>' unk_token = '<unk>' pad_token = '<pad>' mask_token = '<mask>' sep_token = '</s>' **kwargs  )   Parameters   vocab_file (str) —
Path to the vocabulary file.   merges_file (str) —
Path to the merges file.    Construct a “Fast” BPE tokenizer for HerBERT (backed by HuggingFace’s tokenizers library). Peculiarities: uses BERT’s pre-tokenizer: BertPreTokenizer splits tokens on spaces, and also on punctuation. Each occurrence of
a punctuation character will be treated separately. This tokenizer inherits from PreTrainedTokenizer which contains most of the methods. Users should refer to the
superclass for more information regarding methods.  build_inputs_with_special_tokens  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None  ) → List[int]  Parameters   token_


HerbertTokenizerFast

ids_0 (List[int]) —
List of IDs to which the special tokens will be added.   token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.   Returns
List[int]

List of input IDs with the appropriate special tokens.
  Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. An HerBERT, like BERT sequence has the following format: single sequence: <s> X </s> pair of sequences: <s> A </s> B </s>  create_token_type_ids_from_sequences  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None  ) → List[int]  Parameters   token_ids_0 (List[int]) —
List of IDs.   token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.   Returns
List[int]

List of token type IDs according to the given sequence(s).
  Create a mask from the two sequences passed to be used in a seq


HerbertTokenizerFast

uence-pair classification task. HerBERT, like  BERT sequence pair mask has the following format:   Copied 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
| first sequence    | second sequence |  get_special_tokens_mask  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None already_has_special_tokens: bool = False  ) → List[int]  Parameters   token_ids_0 (List[int]) —
List of IDs.   token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.   already_has_special_tokens (bool, optional, defaults to False) —
Whether or not the token list is already formatted with special tokens for the model.   Returns
List[int]

A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
  Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer prepare_for_model method


Overview

FLAN-T5 was released in the paper Scaling Instruction-Finetuned Language Models - it is an enhanced version of T5 that has been finetuned in a mixture of tasks.
One can directly use FLAN-T5 weights without finetuning the model:
  Copied >>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
>>> model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-small")
>>> tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-small")
>>> inputs = tokenizer("A step by step recipe to make bolognese pasta:", return_tensors="pt")
>>> outputs = model.generate(**inputs)
>>> print(tokenizer.


Overview

batch_decode(outputs, skip_special_tokens=True))
['Pour a cup of bolognese into a large bowl and add the pasta']
FLAN-T5 includes the same improvements as T5 version 1.1 (see here for the full details of the model’s improvements.)
Google has released the following variants:
google/flan-t5-small google/flan-t5-base google/flan-t5-large google/flan-t5-xl google/flan-t5-xxl.
One can refer to T5’s documentation page for all tips, code examples and notebooks. As well as the FLAN-T5 model card for more details regarding training and evaluation of the model.
The original checkpoints can be found here


Overview

The Mask2Former model was proposed in Masked-attention Mask Transformer for Universal Image Segmentation by Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, Rohit Girdhar. Mask2Former is a unified framework for panoptic, instance and semantic segmentation and features significant performance and efficiency improvements over MaskFormer.
The abstract from the paper is the following:
Image segmentation groups pixels with different semantics, e.g., category or instance membership. Each choice
of semantics defines a task. While only the semantics of each task differ, current research focuses on designing specialized architectures for each task. We present Masked-attent


Overview

ion Mask Transformer (Mask2Former), a new architecture capable of addressing any image segmentation task (panoptic, instance or semantic). Its key components include masked attention, which extracts localized features by constraining cross-attention within predicted mask regions. In addition to reducing the research effort by at least three times, it outperforms the best specialized architectures by a significant margin on four popular datasets. Most notably, Mask2Former sets a new state-of-the-art for panoptic segmentation (57.8 PQ on COCO), instance segmentation (50.1 AP on COCO) and semantic segmentation (57.7 mIoU on ADE20K).
Tips:
Mask2Former uses the same preprocessing and pos


Overview

tprocessing steps as MaskFormer. Use Mask2FormerImageProcessor or AutoImageProcessor to prepare images and optional targets for the model. To get the final segmentation, depending on the task, you can call post_process_semantic_segmentation() or post_process_instance_segmentation() or post_process_panoptic_segmentation(). All three tasks can be solved using Mask2FormerForUniversalSegmentation output, panoptic segmentation accepts an optional label_ids_to_fuse argument to fuse instances of the target object/s (e.g. sky) together.
Mask2Former architecture. Taken from the original paper.
This model was contributed by Shivalika Singh and Alara Dirik. The original code can be found here.


Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Mask2Former.
Demo notebooks regarding inference + fine-tuning Mask2Former on custom data can be found here.
If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.
The resource should ideally demonstrate something new instead of duplicating an existing resource.


MaskFormer specific outputs

class transformers.models.mask2former.modeling_mask2former.Mask2FormerModelOutput  < source > ( encoder_last_hidden_state: FloatTensor = None pixel_decoder_last_hidden_state: FloatTensor = None transformer_decoder_last_hidden_state: FloatTensor = None encoder_hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None pixel_decoder_hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None transformer_decoder_hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None transformer_decoder_intermediate_states: typing.Tuple[torch.FloatTensor] = None masks_queries_logits: typing.Tuple[torch.FloatTensor] = None attentions: typing.Optional[typing.Tuple[torch.FloatTensor]] = None  )   Parameters   encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, num_channels, height, width), optional) —
Last hidden states (final feature map) of the last s


MaskFormer specific outputs

tage of the encoder model (backbone). Returned when
output_hidden_states=True is passed.   encoder_hidden_states (tuple(torch.FloatTensor), optional) —
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the encoder
model at the output of each stage. Returned when output_hidden_states=True is passed.   pixel_decoder_last_hidden_state (torch.FloatTensor of shape (batch_size, num_channels, height, width), optional) —
Last hidden states (final feature map) of the last stage of the pixel decoder model.   pixel_decoder_hidden_states (tuple(torch.FloatTensor), , optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the outp


MaskFormer specific outputs

ut of each stage) of
shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the pixel
decoder model at the output of each stage. Returned when output_hidden_states=True is passed.   transformer_decoder_last_hidden_state (tuple(torch.FloatTensor)) —
Final output of the transformer decoder (batch_size, sequence_length, hidden_size).   transformer_decoder_hidden_states (tuple(torch.FloatTensor), optional) —
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, sequence_length, hidden_size). Hidden-states (also called feature maps) of the
transformer decoder at the output of each stage. Returned when output_hidden_states=True is passed.   transformer_decoder_intermediate_states (tuple(torch.FloatTensor) of shape (num_queries, 1, hidden_size)) —
Intermediate decoder activations, i.e


MaskFormer specific outputs

. the output of each decoder layer, each of them gone through a
layernorm.   masks_queries_logits (tuple(torch.FloatTensor) of shape (batch_size, num_queries, height, width)) —
Mask Predictions from each layer in the transformer decoder.   attentions (tuple(tuple(torch.FloatTensor)), optional, returned when output_attentions=True is passed) —
Tuple of tuple(torch.FloatTensor) (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Self attentions weights from transformer decoder.    Class for outputs of Mask2FormerModel. This class returns all the needed hidden states to compute the logits.
 class transformers.models.mask2former.modeling_mask2former.Mask2FormerForUniversalSegmentationOutput  < source > ( loss: typing.Optional[torch.FloatTensor] = None class_queries_logits: FloatTensor = None masks_queries_logits: FloatTensor = None auxiliary_logits:


MaskFormer specific outputs

typing.Union[typing.List[typing.Dict[str, torch.FloatTensor]], NoneType] = None encoder_last_hidden_state: FloatTensor = None pixel_decoder_last_hidden_state: FloatTensor = None transformer_decoder_last_hidden_state: FloatTensor = None encoder_hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None pixel_decoder_hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None transformer_decoder_hidden_states: typing.Optional[torch.FloatTensor] = None attentions: typing.Optional[typing.Tuple[torch.FloatTensor]] = None  )   Parameters   loss (torch.Tensor, optional) —
The computed loss, returned when labels are present.   class_queries_logits (torch.FloatTensor) —
A tensor of shape (batch_size, num_queries, num_labels + 1) representing the proposed classes for each
query. Note the + 1 is needed because we incorporate the null class.   masks_queries_logits (torch.


MaskFormer specific outputs

FloatTensor) —
A tensor of shape (batch_size, num_queries, height, width) representing the proposed masks for each
query.   auxiliary_logits (List[Dict(str, torch.FloatTensor)], optional) —
List of class and mask predictions from each layer of the transformer decoder.   encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Last hidden states (final feature map) of the last stage of the encoder model (backbone).   encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the encoder
model at the output of each stage.   pixel_decoder_last_hidden_state (t


MaskFormer specific outputs

orch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Last hidden states (final feature map) of the last stage of the pixel decoder model.   pixel_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the pixel
decoder model at the output of each stage.   transformer_decoder_last_hidden_state (tuple(torch.FloatTensor)) —
Final output of the transformer decoder (batch_size, sequence_length, hidden_size).   transformer_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of


MaskFormer specific outputs

 torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, sequence_length, hidden_size). Hidden-states (also called feature maps) of the
transformer decoder at the output of each stage.   attentions (tuple(tuple(torch.FloatTensor)), optional, returned when output_attentions=True is passed or when config.output_attentions=True) —
Tuple of tuple(torch.FloatTensor) (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Self and Cross Attentions weights from transformer decoder.    Class for outputs of Mask2FormerForUniversalSegmentationOutput. This output can be directly passed to post_process_semantic_segmentation() or
post_process_instance_segmentation() or
post_process_panoptic_segmentation() to compute final segmentation maps. Please, see
[`~Mask2FormerImageProcessor] for details regarding us


Mask2FormerConfig

class transformers.Mask2FormerConfig  < source > ( backbone_config: typing.Optional[typing.Dict] = None feature_size: int = 256 mask_feature_size: int = 256 hidden_dim: int = 256 encoder_feedforward_dim: int = 1024 activation_function: str = 'relu' encoder_layers: int = 6 decoder_layers: int = 10 num_attention_heads: int = 8 dropout: float = 0.0 dim_feedforward: int = 2048 pre_norm: bool = False enforce_input_projection: bool = False common_stride: int = 4 ignore_value: int = 255 num_queries: int = 100 no_object_weight: float = 0.1 class_weight: float = 2.0 mask_weight: float = 5.0 dice_weight: float = 5.0 train_num_points: int = 12544 oversample_ratio: float = 3.0 importance_sample_ratio: float = 0.75 init_std: float = 0.02 init_xavier_std: float = 1.0 use_auxiliary_loss: bool = True feature_strides: typing.List[int] = [4, 8, 16, 32] output_auxiliary_logits: bool = None **kwargs  )   Parameters   backbone_config (PretrainedConfig or dict,


Mask2FormerConfig

 optional, defaults to SwinConfig()) —
The configuration of the backbone model. If unset, the configuration corresponding to
swin-base-patch4-window12-384 will be used.   feature_size (int, optional, defaults to 256) —
The features (channels) of the resulting feature maps.   mask_feature_size (int, optional, defaults to 256) —
The masks’ features size, this value will also be used to specify the Feature Pyramid Network features’
size.   hidden_dim (int, optional, defaults to 256) —
Dimensionality of the encoder layers.   encoder_feedforward_dim (int, optional, defaults to 1024) —
Dimension of feedforward network for deformable detr encoder used as part of pixel decoder.   encoder_layers (int, optional, defaults to 6) —
Number of layers in the deformable detr encoder used as part of pixel decoder.   decoder_layers (int, optional, defaults to 10) —
Number of layers in the Transformer decoder.   num_attention_heads (int, optional, defaults to


Mask2FormerConfig

 8) —
Number of attention heads for each attention layer.   dropout (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the embeddings, encoder.   dim_feedforward (int, optional, defaults to 2048) —
Feature dimension in feedforward network for transformer decoder.   pre_norm (bool, optional, defaults to False) —
Whether to use pre-LayerNorm or not for transformer decoder.   enforce_input_projection (bool, optional, defaults to False) —
Whether to add an input projection 1x1 convolution even if the input channels and hidden dim are identical
in the Transformer decoder.   common_stride (int, optional, defaults to 4) —
Parameter used for determining number of FPN levels used as part of pixel decoder.   ignore_value (int, optional, defaults to 255) —
Category id to be ignored during training.   num_queries (int, optional, defaults to 100) —
Number of queries for the decoder.   no_object_weight (int, o


Mask2FormerConfig

ptional, defaults to 0.1) —
The weight to apply to the null (no object) class.   class_weight (int, optional, defaults to 2.0) —
The weight for the cross entropy loss.   mask_weight (int, optional, defaults to 5.0) —
The weight for the mask loss.   dice_weight (int, optional, defaults to 5.0) —
The weight for the dice loss.   train_num_points (str or function, optional, defaults to 12544) —
Number of points used for sampling during loss calculation.   oversample_ratio (float, optional, defaults to 3.0) —
Oversampling parameter used for calculating no. of sampled points   importance_sample_ratio (float, optional, defaults to 0.75) —
Ratio of points that are sampled via importance sampling.   init_std (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   init_xavier_std (`float“, optional, defaults to 1.0) —
The scaling factor used for the Xavier initializatio


Mask2FormerConfig

n gain in the HM Attention map module.   use_auxiliary_loss (boolean``, *optional*, defaults to True) -- If True Mask2FormerForUniversalSegmentationOutput` will contain the auxiliary losses computed using
the logits from each decoder’s stage.   feature_strides (List[int], optional, defaults to [4, 8, 16, 32]) —
Feature strides corresponding to features generated from backbone network.   output_auxiliary_logits (bool, optional) —
Should the model output its auxiliary_logits or not.    This is the configuration class to store the configuration of a Mask2FormerModel. It is used to instantiate a
Mask2Former model according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the Mask2Former
facebook/mask2former-swin-small-coco-instance
architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model output


Mask2FormerConfig

s. Read the
documentation from PretrainedConfig for more information. Currently, Mask2Former only supports the Swin Transformer as backbone.  Examples:   Copied >>> from transformers import Mask2FormerConfig, Mask2FormerModel
>>> # Initializing a Mask2Former facebook/mask2former-swin-small-coco-instance configuration
>>> configuration = Mask2FormerConfig()
>>> # Initializing a model (with random weights) from the facebook/mask2former-swin-small-coco-instance style configuration
>>> model = Mask2FormerModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config  from_backbone_config  < source > ( backbone_config: PretrainedConfig **kwargs  ) → Mask2FormerConfig  Parameters   backbone_config (PretrainedConfig) —
The backbone configuration.   Returns
Mask2FormerConfig

An instance of a configuration object
  Instantiate a Mask2FormerConfig (or a derived class) from a pre-trained backbone model configuration


Mask2FormerModel

class transformers.Mask2FormerModel  < source > ( config: Mask2FormerConfig  )   Parameters   config (Mask2FormerConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare Mask2Former Model outputting raw hidden-states without any specific head on top.
This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: Tensor pixel_mask: typing.Optional[torch.Tensor] = None output_hidden_states: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.mask2former.modeling_mask2former.Mask2FormerModelOutput or tuple(torch.FloatTensor)  Parameters   pix


Mask2FormerModel

el_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
AutoImageProcessor.preprocess for details.   pixel_mask (torch.LongTensor of shape (batch_size, height, width), optional) —
Mask to avoid performing attention on padding pixel values. Mask values selected in [0, 1]:
1 for pixels that are real (i.e. not masked),
0 for pixels that are padding (i.e. masked).
What are attention masks?   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of Detr’s decoder attention layers.   return_dict (bool, optional) —
Whether or not to return a ~Mask2FormerModelOutput instead of a plain tuple.   Returns
transformers.models.mask2former.modeling_mask2former.Mask2FormerModelOutput or tuple(torch.FloatTensor)


Mask2FormerModel

A transformers.models.mask2former.modeling_mask2former.Mask2FormerModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (Mask2FormerConfig) and inputs.
encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, num_channels, height, width), optional) — Last hidden states (final feature map) of the last stage of the encoder model (backbone). Returned when
output_hidden_states=True is passed.
encoder_hidden_states (tuple(torch.FloatTensor), optional) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the encoder
model at the output of each stage. Returned when output_hidden_states=True is passed.
pixel_decoder_last_hidden_state (torch.FloatTensor of shape (batch_size, num_channels, height, width), optional) — La


Mask2FormerModel

st hidden states (final feature map) of the last stage of the pixel decoder model.
pixel_decoder_hidden_states (tuple(torch.FloatTensor), , optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the pixel
decoder model at the output of each stage. Returned when output_hidden_states=True is passed.
transformer_decoder_last_hidden_state (tuple(torch.FloatTensor)) — Final output of the transformer decoder (batch_size, sequence_length, hidden_size).
transformer_decoder_hidden_states (tuple(torch.FloatTensor), optional) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, sequence_length, hidden_size). Hidden-states (also called feature maps) of the
transformer decoder at


Mask2FormerModel

the output of each stage. Returned when output_hidden_states=True is passed.
transformer_decoder_intermediate_states (tuple(torch.FloatTensor) of shape (num_queries, 1, hidden_size)) — Intermediate decoder activations, i.e. the output of each decoder layer, each of them gone through a
layernorm.
masks_queries_logits (tuple(torch.FloatTensor) of shape (batch_size, num_queries, height, width))
Mask Predictions from each layer in the transformer decoder.
attentions (tuple(tuple(torch.FloatTensor)), optional, returned when output_attentions=True is passed) — Tuple of tuple(torch.FloatTensor) (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Self attentions weights from transformer decoder.
Mask2FormerModelOutput
  The Mask2FormerModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former


Mask2FormerModel

takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> import torch
>>> from PIL import Image
>>> import requests
>>> from transformers import AutoImageProcessor, Mask2FormerModel
>>> # load image
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> # load image preprocessor and Mask2FormerModel trained on COCO instance segmentation dataset
>>> image_processor = AutoImageProcessor.from_pretrained("facebook/mask2former-swin-small-coco-instance")
>>> model = Mask2FormerModel.from_pretrained("facebook/mask2former-swin-small-coco-instance")
>>> inputs = image_processor(image, return_tensors="pt")
>>> # forward pass
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> # model outputs last hidden states of shape (batch_size, num_queries, hidden_size)
>>> print(outputs.transformer_decoder_last_hidden_state.shape)
torch.Size([1, 100, 256


class transformers.Mask2FormerForUniversalSegmentation  < source > ( config: Mask2FormerConfig  )   Parameters   config (Mask2FormerConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The Mask2Former Model with heads on top for instance/semantic/panoptic segmentation.
This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: Tensor mask_labels: typing.Optional[typing.List[torch.Tensor]] = None class_labels: typing.Optional[typing.List[torch.Tensor]] = None pixel_mask: typing.Optional[torch.Tensor] = None output_hidden_states: typing.Optional[bool] = None output_auxiliary_logits: typing.Optional[bool] = None output_attentions


: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.mask2former.modeling_mask2former.Mask2FormerForUniversalSegmentationOutput or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
AutoImageProcessor.preprocess for details.   pixel_mask (torch.LongTensor of shape (batch_size, height, width), optional) —
Mask to avoid performing attention on padding pixel values. Mask values selected in [0, 1]:
1 for pixels that are real (i.e. not masked),
0 for pixels that are padding (i.e. masked).
What are attention masks?   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of Detr’s decoder attention layers.


return_dict (bool, optional) —
Whether or not to return a ~Mask2FormerModelOutput instead of a plain tuple.   mask_labels (List[torch.Tensor], optional) —
List of mask labels of shape (num_labels, height, width) to be fed to a model   class_labels (List[torch.LongTensor], optional) —
list of target class labels of shape (num_labels, height, width) to be fed to a model. They identify the
labels of mask_labels, e.g. the label of mask_labels[i][j] if class_labels[i][j].   Returns
transformers.models.mask2former.modeling_mask2former.Mask2FormerForUniversalSegmentationOutput or tuple(torch.FloatTensor)

A transformers.models.mask2former.modeling_mask2former.Mask2FormerForUniversalSegmentationOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (Mask2FormerConfig) and inputs.
loss (torch.Tensor, optional) — The computed loss, returned when labels are present.
class_quer


ies_logits (torch.FloatTensor) — A tensor of shape (batch_size, num_queries, num_labels + 1) representing the proposed classes for each
query. Note the + 1 is needed because we incorporate the null class.
masks_queries_logits (torch.FloatTensor) — A tensor of shape (batch_size, num_queries, height, width) representing the proposed masks for each
query.
auxiliary_logits (List[Dict(str, torch.FloatTensor)], optional) — List of class and mask predictions from each layer of the transformer decoder.
encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, num_channels, height, width)) — Last hidden states (final feature map) of the last stage of the encoder model (backbone).
encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, num_channels, height, wid


th). Hidden-states (also called feature maps) of the encoder
model at the output of each stage.
pixel_decoder_last_hidden_state (torch.FloatTensor of shape (batch_size, num_channels, height, width)) — Last hidden states (final feature map) of the last stage of the pixel decoder model.
pixel_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the pixel
decoder model at the output of each stage.
transformer_decoder_last_hidden_state (tuple(torch.FloatTensor)) — Final output of the transformer decoder (batch_size, sequence_length, hidden_size).
transformer_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_h


idden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, sequence_length, hidden_size). Hidden-states (also called feature maps) of the
transformer decoder at the output of each stage.
attentions (tuple(tuple(torch.FloatTensor)), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tuple(torch.FloatTensor) (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Self and Cross Attentions weights from transformer decoder.
Mask2FormerUniversalSegmentationOutput
  The Mask2FormerForUniversalSegmentation forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them. E


xamples:  Instance segmentation example:   Copied >>> from transformers import AutoImageProcessor, Mask2FormerForUniversalSegmentation
>>> from PIL import Image
>>> import requests
>>> import torch
>>> # Load Mask2Former trained on COCO instance segmentation dataset
>>> image_processor = AutoImageProcessor.from_pretrained("facebook/mask2former-swin-small-coco-instance")
>>> model = Mask2FormerForUniversalSegmentation.from_pretrained(
...     "facebook/mask2former-swin-small-coco-instance"
... )
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = image_processor(image, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> # Model predicts class_queries_logits of shape `(batch_size, num_queries)`
>>> # and masks_queries_logits of shape `(batch_size, num_queries, height, width)`
>>> class_queries_logits = outputs.class_queries_logits
>>> masks_queries_logits = ou


tputs.masks_queries_logits
>>> # Perform post-processing to get instance segmentation map
>>> pred_instance_map = image_processor.post_process_semantic_segmentation(
...     outputs, target_sizes=[image.size[::-1]]
... )[0]
>>> print(pred_instance_map.shape)
torch.Size([480, 640])  Semantic segmentation example:   Copied >>> from transformers import AutoImageProcessor, Mask2FormerForUniversalSegmentation
>>> from PIL import Image
>>> import requests
>>> import torch
>>> # Load Mask2Former trained on ADE20k semantic segmentation dataset
>>> image_processor = AutoImageProcessor.from_pretrained("facebook/mask2former-swin-small-ade-semantic")
>>> model = Mask2FormerForUniversalSegmentation.from_pretrained("facebook/mask2former-swin-small-ade-semantic")
>>> url = (
...     "https://huggingface.co/datasets/hf-internal-testing/fixtures_ade20k/resolve/main/ADE_val_00000001.jpg"
... )
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = image_processor(image, return


_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> # Model predicts class_queries_logits of shape `(batch_size, num_queries)`
>>> # and masks_queries_logits of shape `(batch_size, num_queries, height, width)`
>>> class_queries_logits = outputs.class_queries_logits
>>> masks_queries_logits = outputs.masks_queries_logits
>>> # Perform post-processing to get semantic segmentation map
>>> pred_semantic_map = image_processor.post_process_semantic_segmentation(
...     outputs, target_sizes=[image.size[::-1]]
... )[0]
>>> print(pred_semantic_map.shape)
torch.Size([512, 683])  Panoptic segmentation example:   Copied >>> from transformers import AutoImageProcessor, Mask2FormerForUniversalSegmentation
>>> from PIL import Image
>>> import requests
>>> import torch
>>> # Load Mask2Former trained on CityScapes panoptic segmentation dataset
>>> image_processor = AutoImageProcessor.from_pretrained("facebook/mask2former-swin-small-cityscapes-panoptic")
>>> model


= Mask2FormerForUniversalSegmentation.from_pretrained(
...     "facebook/mask2former-swin-small-cityscapes-panoptic"
... )
>>> url = "https://cdn-media.huggingface.co/Inference-API/Sample-results-on-the-Cityscapes-dataset-The-above-images-show-how-our-method-can-handle.png"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = image_processor(image, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> # Model predicts class_queries_logits of shape `(batch_size, num_queries)`
>>> # and masks_queries_logits of shape `(batch_size, num_queries, height, width)`
>>> class_queries_logits = outputs.class_queries_logits
>>> masks_queries_logits = outputs.masks_queries_logits
>>> # Perform post-processing to get panoptic segmentation map
>>> pred_panoptic_map = image_processor.post_process_panoptic_segmentation(
...     outputs, target_sizes=[image.size[::-1]]
... )[0]["segmentation"]
>>> print(pred_panoptic_map.shape)
torch.Size([338,


class transformers.Mask2FormerImageProcessor  < source > ( do_resize: bool = True size: typing.Dict[str, int] = None size_divisor: int = 32 resample: Resampling = <Resampling.BILINEAR: 2> do_rescale: bool = True rescale_factor: float = 0.00392156862745098 do_normalize: bool = True image_mean: typing.Union[float, typing.List[float]] = None image_std: typing.Union[float, typing.List[float]] = None ignore_index: typing.Optional[int] = None reduce_labels: bool = False **kwargs  )   Parameters   do_resize (bool, optional, defaults to True) —
Whether to resize the input to a certain size.   size (int, optional, defaults to 800) —
Resize the input to the given size. Only has an effect if do_resize is set to True. If size is a
sequence like (width, height), output size will be matched to this. If size is an int, smaller edge of
the image will be matched to this number. i.e, if height > width, then image will be rescaled to (size * height / width, size).   size_divisor (int, optional, defaults


 to 32) —
Some backbones need images divisible by a certain number. If not passed, it defaults to the value used in
Swin Transformer.   resample (int, optional, defaults to Resampling.BILINEAR) —
An optional resampling filter. This can be one of PIL.Image.Resampling.NEAREST,
PIL.Image.Resampling.BOX, PIL.Image.Resampling.BILINEAR, PIL.Image.Resampling.HAMMING,
PIL.Image.Resampling.BICUBIC or PIL.Image.Resampling.LANCZOS. Only has an effect if do_resize is set
to True.   do_rescale (bool, optional, defaults to True) —
Whether to rescale the input to a certain scale.   rescale_factor (float, optional, defaults to 1/ 255) —
Rescale the input by the given factor. Only has an effect if do_rescale is set to True.   do_normalize (bool, optional, defaults to True) —
Whether or not to normalize the input with mean and standard deviation.   image_mean (int, optional, defaults to [0.485, 0.456, 0.406]) —
The sequence of means for each channel, to be used when normalizing images. Defaults to the


ImageNet mean.   image_std (int, optional, defaults to [0.229, 0.224, 0.225]) —
The sequence of standard deviations for each channel, to be used when normalizing images. Defaults to the
ImageNet std.   ignore_index (int, optional) —
Label to be assigned to background pixels in segmentation maps. If provided, segmentation map pixels
denoted with 0 (background) will be replaced with ignore_index.   reduce_labels (bool, optional, defaults to False) —
Whether or not to decrement all label values of segmentation maps by 1. Usually used for datasets where 0
is used for background, and background itself is not included in all classes of a dataset (e.g. ADE20k).
The background label will be replaced by ignore_index.    Constructs a Mask2Former image processor. The image processor can be used to prepare image(s) and optional targets
for the model. This image processor inherits from BaseImageProcessor which contains most of the main methods. Users should
refer to this superclass for more inform


ation regarding those methods.  preprocess  < source > ( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]] segmentation_maps: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')], NoneType] = None instance_id_to_semantic_id: typing.Union[typing.Dict[int, int], NoneType] = None do_resize: typing.Optional[bool] = None size: typing.Union[typing.Dict[str, int], NoneType] = None size_divisor: typing.Optional[int] = None resample: Resampling = None do_rescale: typing.Optional[bool] = None rescale_factor: typing.Optional[float] = None do_normalize: typing.Optional[bool] = None image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float]


, NoneType] = None ignore_index: typing.Optional[int] = None reduce_labels: typing.Optional[bool] = None return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None data_format: typing.Union[str, transformers.image_utils.ChannelDimension] = <ChannelDimension.FIRST: 'channels_first'> input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None **kwargs  )     encode_inputs  < source > ( pixel_values_list: typing.List[typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]]] segmentation_maps: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]] = None instance_id_to_semantic_id: typing.Union[typing.List[typing.Dict[int, int]], typing.Di


ct[int, int], NoneType] = None ignore_index: typing.Optional[int] = None reduce_labels: bool = False return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None  ) → BatchFeature  Parameters   pixel_values_list (List[ImageInput]) —
List of images (pixel values) to be padded. Each image should be a tensor of shape (channels, height, width).   segmentation_maps (ImageInput, optional) —
The corresponding semantic segmentation maps with the pixel-wise annotations.
(bool, optional, defaults to True):
Whether or not to pad images up to the largest image in a batch and create a pixel mask.
If left to the default, will return a pixel mask that is:
1 for pixels that are real (i.e. not masked),
0 for pixels that are padding (i.e. masked).
   instance_id_to_semantic_id (List[Dict[int, int]] or Dict[int, int], optional) —
A mapping between object instance ids and class id


s. If passed, segmentation_maps is treated as an
instance segmentation map where each pixel represents an instance id. Can be provided as a single
dictionary with a global/dataset-level mapping or as a list of dictionaries (one per image), to map
instance ids in each image separately.   return_tensors (str or TensorType, optional) —
If set, will return tensors instead of NumPy arrays. If set to 'pt', return PyTorch torch.Tensor
objects.   input_data_format (ChannelDimension or str, optional) —
The channel dimension format of the input image. If not provided, it will be inferred.   Returns
BatchFeature

A BatchFeature with the following fields:
pixel_values — Pixel values to be fed to a model.
pixel_mask — Pixel mask to be fed to a model (when =True or if pixel_mask is in
self.model_input_names).
mask_labels — Optional list of mask labels of shape (labels, height, width) to be fed to a model
(when annotations are provided).
class_labels — Optional list of class labels of shape (labe


ls) to be fed to a model (when
annotations are provided). They identify the labels of mask_labels, e.g. the label of
mask_labels[i][j] if class_labels[i][j].
  Pad images up to the largest image in a batch and create a corresponding pixel_mask. Mask2Former addresses semantic segmentation with a mask classification paradigm, thus input segmentation maps
will be converted to lists of binary masks and their respective labels. Let’s see an example, assuming
segmentation_maps = [[2,6,7,9]], the output will contain mask_labels = [[1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1]] (four binary masks) and class_labels = [2,6,7,9], the labels for
each mask.  post_process_semantic_segmentation  < source > ( outputs target_sizes: typing.Union[typing.List[typing.Tuple[int, int]], NoneType] = None  ) → List[torch.Tensor]  Parameters   outputs (Mask2FormerForUniversalSegmentation) —
Raw outputs of the model.   target_sizes (List[Tuple[int, int]], optional) —
List of length (batch_size), where each list item


(Tuple[int, int]]) corresponds to the requested
final size (height, width) of each prediction. If left to None, predictions will not be resized.   Returns
List[torch.Tensor]

A list of length batch_size, where each item is a semantic segmentation map of shape (height, width)
corresponding to the target_sizes entry (if target_sizes is specified). Each entry of each
torch.Tensor correspond to a semantic class id.
  Converts the output of Mask2FormerForUniversalSegmentation into semantic segmentation maps. Only supports
PyTorch.  post_process_instance_segmentation  < source > ( outputs threshold: float = 0.5 mask_threshold: float = 0.5 overlap_mask_area_threshold: float = 0.8 target_sizes: typing.Union[typing.List[typing.Tuple[int, int]], NoneType] = None return_coco_annotation: typing.Optional[bool] = False return_binary_maps: typing.Optional[bool] = False  ) → List[Dict]  Parameters   outputs (Mask2FormerForUniversalSegmentation) —
Raw outputs of the model.   threshold (float, optio


nal, defaults to 0.5) —
The probability score threshold to keep predicted instance masks.   mask_threshold (float, optional, defaults to 0.5) —
Threshold to use when turning the predicted masks into binary values.   overlap_mask_area_threshold (float, optional, defaults to 0.8) —
The overlap mask area threshold to merge or discard small disconnected parts within each binary
instance mask.   target_sizes (List[Tuple], optional) —
List of length (batch_size), where each list item (Tuple[int, int]]) corresponds to the requested
final size (height, width) of each prediction. If left to None, predictions will not be resized.   return_coco_annotation (bool, optional, defaults to False) —
If set to True, segmentation maps are returned in COCO run-length encoding (RLE) format.   return_binary_maps (bool, optional, defaults to False) —
If set to True, segmentation maps are returned as a concatenated tensor of binary segmentation maps
(one per detected instance).   Returns
List[Dict]

A list


 of dictionaries, one per image, each dictionary containing two keys:
segmentation — A tensor of shape (height, width) where each pixel represents a segment_id or
List[List] run-length encoding (RLE) of the segmentation map if return_coco_annotation is set to
True. Set to None if no mask if found above threshold.
segments_info — A dictionary that contains additional information on each segment.
id — An integer representing the segment_id.
label_id — An integer representing the label / semantic class id corresponding to segment_id.
score — Prediction score of segment with segment_id.
  Converts the output of Mask2FormerForUniversalSegmentationOutput into instance segmentation predictions.
Only supports PyTorch.  post_process_panoptic_segmentation  < source > ( outputs threshold: float = 0.5 mask_threshold: float = 0.5 overlap_mask_area_threshold: float = 0.8 label_ids_to_fuse: typing.Optional[typing.Set[int]] = None target_sizes: typing.Union[typing.List[typing.Tuple[int, int]], NoneTy


pe] = None  ) → List[Dict]  Parameters   outputs (Mask2FormerForUniversalSegmentationOutput) —
The outputs from Mask2FormerForUniversalSegmentation.   threshold (float, optional, defaults to 0.5) —
The probability score threshold to keep predicted instance masks.   mask_threshold (float, optional, defaults to 0.5) —
Threshold to use when turning the predicted masks into binary values.   overlap_mask_area_threshold (float, optional, defaults to 0.8) —
The overlap mask area threshold to merge or discard small disconnected parts within each binary
instance mask.   label_ids_to_fuse (Set[int], optional) —
The labels in this state will have all their instances be fused together. For instance we could say
there can only be one sky in an image, but several persons, so the label ID for sky would be in that
set, but not the one for person.   target_sizes (List[Tuple], optional) —
List of length (batch_size), where each list item (Tuple[int, int]]) corresponds to the requested
final size (heigh


t, width) of each prediction in batch. If left to None, predictions will not be
resized.   Returns
List[Dict]

A list of dictionaries, one per image, each dictionary containing two keys:
segmentation — a tensor of shape (height, width) where each pixel represents a segment_id, set
to None if no mask if found above threshold. If target_sizes is specified, segmentation is resized
to the corresponding target_sizes entry.
segments_info — A dictionary that contains additional information on each segment.
id — an integer representing the segment_id.
label_id — An integer representing the label / semantic class id corresponding to segment_id.
was_fused — a boolean, True if label_id was in label_ids_to_fuse, False otherwise.
Multiple instances of the same class / label were fused and assigned a single segment_id.
score — Prediction score of segment with segment_id.
  Converts the output of Mask2FormerForUniversalSegmentationOutput into image panoptic segmentation
predictions. Only supports


Overview

The EncoderDecoderModel can be used to initialize a sequence-to-sequence model with any
pretrained autoencoding model as the encoder and any pretrained autoregressive model as the decoder.
The effectiveness of initializing sequence-to-sequence models with pretrained checkpoints for sequence generation tasks
was shown in Leveraging Pre-trained Checkpoints for Sequence Generation Tasks by
Sascha Rothe, Shashi Narayan, Aliaksei Severyn.
After such an EncoderDecoderModel has been trained/fine-tuned, it can be saved/loaded just like
any other models (see the examples for more information).
An application of this architecture could be to leverage two pretrained BertModel as the encoder
and decoder for a summarization model as was shown in: Text Summarization with Pretrained Encoders by Yang Liu and Mirella Lapata.