TFSegformerForSemanticSegmentation

manticSegmenterOutput or tuple(tf.Tensor)

A transformers.modeling_tf_outputs.TFSemanticSegmenterOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (SegformerConfig) and inputs.
loss (tf.Tensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (tf.Tensor of shape (batch_size, config.num_labels, logits_height, logits_width)) — Classification scores for each pixel.
The logits returned do not necessarily have the same size as the pixel_values passed as inputs. This is
to avoid doing two interpolations and lose some quality when a user needs to resize the logits to the
original image size as post-processing. You should always check your logits shape and resize as needed.
hidden_states (tuple(tf.Tensor), optiona


TFSegformerForSemanticSegmentation

l, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings, if the model has an embedding layer, + one for
the output of each layer) of shape (batch_size, patch_size, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, patch_size, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The TFSegformerForSemanticSegmentation forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one sh


TFSegformerForSemanticSegmentation

ould call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoImageProcessor, TFSegformerForSemanticSegmentation
>>> from PIL import Image
>>> import requests
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> image_processor = AutoImageProcessor.from_pretrained("nvidia/segformer-b0-finetuned-ade-512-512")
>>> model = TFSegformerForSemanticSegmentation.from_pretrained("nvidia/segformer-b0-finetuned-ade-512-512")
>>> inputs = image_processor(images=image, return_tensors="tf")
>>> outputs = model(**inputs, training=False)
>>> # logits are of shape (batch_size, num_labels, height/4, width/4)
>>> logits = outputs.logits
>>> list(logits.shape)
[1, 150, 128, 1


Overview

The VideoMAE model was proposed in VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training by Zhan Tong, Yibing Song, Jue Wang, Limin Wang.
VideoMAE extends masked auto encoders (MAE) to video, claiming state-of-the-art performance on several video classification benchmarks.
The abstract from the paper is the following:
Pre-training video transformers on extra large-scale datasets is generally required to achieve premier performance on relatively small datasets. In this paper, we show that video masked autoencoders (VideoMAE) are data-efficient learners for self-supervised video pre-training (SSVP). We are inspired by the recent ImageMAE and pro


Overview

pose customized video tube masking and reconstruction. These simple designs turn out to be effective for overcoming information leakage caused by the temporal correlation during video reconstruction. We obtain three important findings on SSVP: (1) An extremely high proportion of masking ratio (i.e., 90% to 95%) still yields favorable performance of VideoMAE. The temporally redundant video content enables higher masking ratio than that of images. (2) VideoMAE achieves impressive results on very small datasets (i.e., around 3k-4k videos) without using any extra data. This is partially ascribed to the challenging task of video reconstruction to enforce high-level structure learning. (3) V


Overview

ideoMAE shows that data quality is more important than data quantity for SSVP. Domain shift between pre-training and target datasets are important issues in SSVP. Notably, our VideoMAE with the vanilla ViT backbone can achieve 83.9% on Kinects-400, 75.3% on Something-Something V2, 90.8% on UCF101, and 61.1% on HMDB51 without using any extra data.
Tips:
One can use VideoMAEImageProcessor to prepare videos for the model. It will resize + normalize all frames of a video for you. VideoMAEForPreTraining includes the decoder on top for self-supervised pre-training.
VideoMAE pre-training. Taken from the original paper.
This model was contributed by nielsr.
The original code can be found here.


Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with VideoMAE. If
you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we’ll
review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.
Video classification
A notebook that shows how
to fine-tune a VideoMAE model on a custom dataset. Video classification task guide A 🤗 Space showing how to perform inference with a video classification model.


VideoMAEConfig

class transformers.VideoMAEConfig  < source > ( image_size = 224 patch_size = 16 num_channels = 3 num_frames = 16 tubelet_size = 2 hidden_size = 768 num_hidden_layers = 12 num_attention_heads = 12 intermediate_size = 3072 hidden_act = 'gelu' hidden_dropout_prob = 0.0 attention_probs_dropout_prob = 0.0 initializer_range = 0.02 layer_norm_eps = 1e-12 qkv_bias = True use_mean_pooling = True decoder_num_attention_heads = 6 decoder_hidden_size = 384 decoder_num_hidden_layers = 4 decoder_intermediate_size = 1536 norm_pix_loss = True **kwargs  )   Parameters   image_size (int, optional, defaults to 224) —
The size (resolution) of each image.   patch_size (int, optional, defaults to 16) —
The size (resolution) of each patch.   num_channels (int, optional, defaults to 3) —
The number of input channels.   num_frames (int, optional, defaults to 16) —
The number of frames in each video.   tubelet_size (int, optional, defaults to 2) —
The number of tubelets.   hidden_si


VideoMAEConfig

ze (int, optional, defaults to 768) —
Dimensionality of the encoder layers and the pooler layer.   num_hidden_layers (int, optional, defaults to 12) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 12) —
Number of attention heads for each attention layer in the Transformer encoder.   intermediate_size (int, optional, defaults to 3072) —
Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.   hidden_act (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "selu" and "gelu_new" are supported.   hidden_dropout_prob (float, optional, defaults to 0.0) —
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.   attention_probs_dropout_prob (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.


VideoMAEConfig

 initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   layer_norm_eps (float, optional, defaults to 1e-12) —
The epsilon used by the layer normalization layers.   qkv_bias (bool, optional, defaults to True) —
Whether to add a bias to the queries, keys and values.   use_mean_pooling (bool, optional, defaults to True) —
Whether to mean pool the final hidden states instead of using the final hidden state of the [CLS] token.   decoder_num_attention_heads (int, optional, defaults to 6) —
Number of attention heads for each attention layer in the decoder.   decoder_hidden_size (int, optional, defaults to 384) —
Dimensionality of the decoder.   decoder_num_hidden_layers (int, optional, defaults to 4) —
Number of hidden layers in the decoder.   decoder_intermediate_size (int, optional, defaults to 1536) —
Dimensionality of the “intermediate” (i.e., feed-forward) laye


VideoMAEConfig

r in the decoder.   norm_pix_loss (bool, optional, defaults to True) —
Whether to normalize the target patch pixels.    This is the configuration class to store the configuration of a VideoMAEModel. It is used to instantiate a
VideoMAE model according to the specified arguments, defining the model architecture. Instantiating a configuration
with the defaults will yield a similar configuration to that of the VideoMAE
MCG-NJU/videomae-base architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import VideoMAEConfig, VideoMAEModel
>>> # Initializing a VideoMAE videomae-base style configuration
>>> configuration = VideoMAEConfig()
>>> # Randomly initializing a model from the configuration
>>> model = VideoMAEModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.confi


VideoMAEFeatureExtractor

class transformers.VideoMAEFeatureExtractor  < source > ( *args **kwargs  )     __call__  < source > ( images **kwargs  )    Preprocess an image or a batch of images.


VideoMAEImageProcessor

class transformers.VideoMAEImageProcessor  < source > ( do_resize: bool = True size: typing.Dict[str, int] = None resample: Resampling = <Resampling.BILINEAR: 2> do_center_crop: bool = True crop_size: typing.Dict[str, int] = None do_rescale: bool = True rescale_factor: typing.Union[int, float] = 0.00392156862745098 do_normalize: bool = True image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float], NoneType] = None **kwargs  )   Parameters   do_resize (bool, optional, defaults to True) —
Whether to resize the image’s (height, width) dimensions to the specified size. Can be overridden by the
do_resize parameter in the preprocess method.   size (Dict[str, int] optional, defaults to {"shortest_edge" -- 224}):
Size of the output image after resizing. The shortest edge of the image will be resized to
size["shortest_edge"] w


VideoMAEImageProcessor

hile maintaining the aspect ratio of the original image. Can be overriden by
size in the preprocess method.   resample (PILImageResampling, optional, defaults to Resampling.BILINEAR) —
Resampling filter to use if resizing the image. Can be overridden by the resample parameter in the
preprocess method.   do_center_crop (bool, optional, defaults to True) —
Whether to center crop the image to the specified crop_size. Can be overridden by the do_center_crop
parameter in the preprocess method.   crop_size (Dict[str, int], optional, defaults to {"height" -- 224, "width": 224}):
Size of the image after applying the center crop. Can be overridden by the crop_size parameter in the
preprocess method.   do_rescale (bool, optional, defaults to True) —
Whether to rescale the image by the specified scale rescale_factor. Can be overridden by the do_rescale
parameter in the preprocess method.


VideoMAEImageProcessor

 rescale_factor (int or float, optional, defaults to 1/255) —
Defines the scale factor to use if rescaling the image. Can be overridden by the rescale_factor parameter
in the preprocess method.   do_normalize (bool, optional, defaults to True) —
Whether to normalize the image. Can be overridden by the do_normalize parameter in the preprocess
method.   image_mean (float or List[float], optional, defaults to IMAGENET_STANDARD_MEAN) —
Mean to use if normalizing the image. This is a float or list of floats the length of the number of
channels in the image. Can be overridden by the image_mean parameter in the preprocess method.   image_std (float or List[float], optional, defaults to IMAGENET_STANDARD_STD) —
Standard deviation to use if normalizing the image. This is a float or list of floats the length of the
number of channels in the image. Can be overridden by the image_std parame


VideoMAEImageProcessor

ter in the preprocess method.    Constructs a VideoMAE image processor.  preprocess  < source > ( videos: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]] do_resize: bool = None size: typing.Dict[str, int] = None resample: Resampling = None do_center_crop: bool = None crop_size: typing.Dict[str, int] = None do_rescale: bool = None rescale_factor: float = None do_normalize: bool = None image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float], NoneType] = None return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None data_format: ChannelDimension = <ChannelDimension.FIRST: 'channels_first'> input_data_format: typing.Union[str, transformers.


VideoMAEImageProcessor

image_utils.ChannelDimension, NoneType] = None **kwargs  )   Parameters   images (ImageInput) —
Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If
passing in images with pixel values between 0 and 1, set do_rescale=False.   do_resize (bool, optional, defaults to self.do_resize) —
Whether to resize the image.   size (Dict[str, int], optional, defaults to self.size) —
Size of the image after applying resize.   resample (PILImageResampling, optional, defaults to self.resample) —
Resampling filter to use if resizing the image. This can be one of the enum PILImageResampling, Only
has an effect if do_resize is set to True.   do_center_crop (bool, optional, defaults to self.do_centre_crop) —
Whether to centre crop the image.   crop_size (Dict[str, int], optional, defaults to self.crop_size) —
Size of the image after applying the centre


VideoMAEImageProcessor

crop.   do_rescale (bool, optional, defaults to self.do_rescale) —
Whether to rescale the image values between [0 - 1].   rescale_factor (float, optional, defaults to self.rescale_factor) —
Rescale factor to rescale the image by if do_rescale is set to True.   do_normalize (bool, optional, defaults to self.do_normalize) —
Whether to normalize the image.   image_mean (float or List[float], optional, defaults to self.image_mean) —
Image mean.   image_std (float or List[float], optional, defaults to self.image_std) —
Image standard deviation.   return_tensors (str or TensorType, optional) —
The type of tensors to return. Can be one of:
Unset: Return a list of np.ndarray.
TensorType.TENSORFLOW or 'tf': Return a batch of type tf.Tensor.
TensorType.PYTORCH or 'pt': Return a batch of type torch.Tensor.
TensorType.NUMPY or 'np': Return a batch of type np.ndarray.
TensorType.JAX or 'jax'


VideoMAEImageProcessor

: Return a batch of type jax.numpy.ndarray.
   data_format (ChannelDimension or str, optional, defaults to ChannelDimension.FIRST) —
The channel dimension format for the output image. Can be one of:
ChannelDimension.FIRST: image in (num_channels, height, width) format.
ChannelDimension.LAST: image in (height, width, num_channels) format.
Unset: Use the inferred channel dimension format of the input image.
   input_data_format (ChannelDimension or str, optional) —
The channel dimension format for the input image. If unset, the channel dimension format is inferred
from the input image. Can be one of:
"channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
"channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
"none" or ChannelDimension.NONE: image in (height, width) format.
    Preprocess an image or batch of ima


VideoMAEModel

class transformers.VideoMAEModel  < source > ( config  )   Parameters   config (VideoMAEConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare VideoMAE Model transformer outputting raw hidden-states without any specific head on top.
This model is a PyTorch torch.nn.Module subclass. Use it
as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: FloatTensor bool_masked_pos: typing.Optional[torch.BoolTensor] = None head_mask: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → t


VideoMAEModel

ransformers.modeling_outputs.BaseModelOutput or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_frames, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
VideoMAEImageProcessor.call() for details.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whet


VideoMAEModel

her or not to return a ModelOutput instead of a plain tuple.   bool_masked_pos (torch.BoolTensor of shape (batch_size, sequence_length), optional) —
Boolean masked positions. Indicates which patches are masked (1) and which aren’t (0). Each video in the
batch must have the same number of masked patches. If None, then all patches are considered. Sequence
length is (num_frames // tubelet_size) * (image_size // patch_size) ** 2.   Returns
transformers.modeling_outputs.BaseModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (VideoMAEConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
hid


VideoMAEModel

den_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The VideoMAEModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs t


VideoMAEModel

o be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> import av
>>> import numpy as np
>>> from transformers import AutoImageProcessor, VideoMAEModel
>>> from huggingface_hub import hf_hub_download
>>> np.random.seed(0)
>>> def read_video_pyav(container, indices):
...     '''
...     Decode the video with PyAV decoder.
...     Args:
...         container (`av.container.input.InputContainer`): PyAV container.
...         indices (`List[int]`): List of frame indices to decode.
...     Returns:
...         result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3).
...     '''
...     frames = []
...     container.seek(0)
...     start_index = indices[0]
...     end_index = indices[-1]
...     for i, fr


VideoMAEModel

ame in enumerate(container.decode(video=0)):
...         if i > end_index:
...             break
...         if i >= start_index and i in indices:
...             frames.append(frame)
...     return np.stack([x.to_ndarray(format="rgb24") for x in frames])
>>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len):
...     '''
...     Sample a given number of frame indices from the video.
...     Args:
...         clip_len (`int`): Total number of frames to sample.
...         frame_sample_rate (`int`): Sample every n-th frame.
...         seg_len (`int`): Maximum allowed index of sample's last frame.
...     Returns:
...         indices (`List[int]`): List of sampled frame indices
...     '''
...     converted_len = int(clip_len * frame_sample_rate)
...     end_idx = np.random.randint(converted_len, seg_len)
...     start_idx = end_idx - converted_len
...     indices = np.linspace(start_idx, end_id


VideoMAEModel

x, num=clip_len)
...     indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64)
...     return indices
>>> # video clip consists of 300 frames (10 seconds at 30 FPS)
>>> file_path = hf_hub_download(
...     repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset"
... )
>>> container = av.open(file_path)
>>> # sample 16 frames
>>> indices = sample_frame_indices(clip_len=16, frame_sample_rate=1, seg_len=container.streams.video[0].frames)
>>> video = read_video_pyav(container, indices)
>>> image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base")
>>> model = VideoMAEModel.from_pretrained("MCG-NJU/videomae-base")
>>> # prepare video for the model
>>> inputs = image_processor(list(video), return_tensors="pt")
>>> # forward pass
>>> outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1, 1568, 76


VideoMAEForPreTraining

class transformers.VideoMAEForPreTraining  < source > ( config  )   Parameters   config (VideoMAEConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The VideoMAE Model transformer with the decoder on top for self-supervised pre-training.
This model is a PyTorch torch.nn.Module subclass. Use it
as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: FloatTensor bool_masked_pos: BoolTensor head_mask: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.videomae.modeling_video


VideoMAEForPreTraining

mae.VideoMAEForPreTrainingOutput or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_frames, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
VideoMAEImageProcessor.call() for details.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a


VideoMAEForPreTraining

ModelOutput instead of a plain tuple.   bool_masked_pos (torch.BoolTensor of shape (batch_size, sequence_length)) —
Boolean masked positions. Indicates which patches are masked (1) and which aren’t (0). Each video in the
batch must have the same number of masked patches. Sequence length is (num_frames // tubelet_size) * (image_size // patch_size) ** 2.   Returns
transformers.models.videomae.modeling_videomae.VideoMAEForPreTrainingOutput or tuple(torch.FloatTensor)

A transformers.models.videomae.modeling_videomae.VideoMAEForPreTrainingOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (VideoMAEConfig) and inputs.
loss (torch.FloatTensor of shape (1,)) — Pixel reconstruction loss.
logits (torch.FloatTensor of shape (batch_size, patch_size ** 2 * num_channels)) — Pixel reconstruction logits.
hidden_states


VideoMAEForPreTraining

 (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size). Hidden-states of the model at the output of each layer
plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Attentions weights after the attention softmax, used to compute the weighted average in
the self-attention heads.
  The VideoMAEForPreTraining forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Modu


VideoMAEForPreTraining

le
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoImageProcessor, VideoMAEForPreTraining
>>> import numpy as np
>>> import torch
>>> num_frames = 16
>>> video = list(np.random.randint(0, 256, (num_frames, 3, 224, 224)))
>>> image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base")
>>> model = VideoMAEForPreTraining.from_pretrained("MCG-NJU/videomae-base")
>>> pixel_values = image_processor(video, return_tensors="pt").pixel_values
>>> num_patches_per_frame = (model.config.image_size // model.config.patch_size) ** 2
>>> seq_length = (num_frames // model.config.tubelet_size) * num_patches_per_frame
>>> bool_masked_pos = torch.randint(0, 2, (1, seq_length)).bool()
>>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos)
>>> loss = outputs.


VideoMAEForVideoClassification

class transformers.VideoMAEForVideoClassification  < source > ( config  )   Parameters   config (VideoMAEConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    VideoMAE Model transformer with a video classification head on top (a linear layer on top of the average pooled hidden
states of all tokens) e.g. for ImageNet.
This model is a PyTorch torch.nn.Module subclass. Use it
as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dic


VideoMAEForVideoClassification

t: typing.Optional[bool] = None  ) → transformers.modeling_outputs.ImageClassifierOutput or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_frames, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
VideoMAEImageProcessor.call() for details.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or n


VideoMAEForVideoClassification

ot to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the image classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-Entropy).   Returns
transformers.modeling_outputs.ImageClassifierOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.ImageClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (VideoMAEConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, config.num_labels)) — Classification (or regression if con


VideoMAEForVideoClassification

fig.num_labels==1) scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each stage) of shape (batch_size, sequence_length, hidden_size). Hidden-states
(also called feature maps) of the model at the output of each stage.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, patch_size, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The VideoMAEForVideoClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defin


VideoMAEForVideoClassification

ed within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> import av
>>> import torch
>>> import numpy as np
>>> from transformers import AutoImageProcessor, VideoMAEForVideoClassification
>>> from huggingface_hub import hf_hub_download
>>> np.random.seed(0)
>>> def read_video_pyav(container, indices):
...     '''
...     Decode the video with PyAV decoder.
...     Args:
...         container (`av.container.input.InputContainer`): PyAV container.
...         indices (`List[int]`): List of frame indices to decode.
...     Returns:
...         result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3).
...     '''
...     frames = []
...     container.seek(0)
...     start_index = indices[0]
...     end_index = indices[-1]
...     for i, frame in enumerate(container.d


VideoMAEForVideoClassification

ecode(video=0)):
...         if i > end_index:
...             break
...         if i >= start_index and i in indices:
...             frames.append(frame)
...     return np.stack([x.to_ndarray(format="rgb24") for x in frames])
>>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len):
...     '''
...     Sample a given number of frame indices from the video.
...     Args:
...         clip_len (`int`): Total number of frames to sample.
...         frame_sample_rate (`int`): Sample every n-th frame.
...         seg_len (`int`): Maximum allowed index of sample's last frame.
...     Returns:
...         indices (`List[int]`): List of sampled frame indices
...     '''
...     converted_len = int(clip_len * frame_sample_rate)
...     end_idx = np.random.randint(converted_len, seg_len)
...     start_idx = end_idx - converted_len
...     indices = np.linspace(start_idx, end_idx, num=clip_len)
...     indices = np.clip(indices, start_idx, end_idx - 1).asty


VideoMAEForVideoClassification

pe(np.int64)
...     return indices
>>> # video clip consists of 300 frames (10 seconds at 30 FPS)
>>> file_path = hf_hub_download(
...     repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset"
... )
>>> container = av.open(file_path)
>>> # sample 16 frames
>>> indices = sample_frame_indices(clip_len=16, frame_sample_rate=1, seg_len=container.streams.video[0].frames)
>>> video = read_video_pyav(container, indices)
>>> image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics")
>>> model = VideoMAEForVideoClassification.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics")
>>> inputs = image_processor(list(video), return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
...     logits = outputs.logits
>>> # model predicts one of the 400 Kinetics-400 classes
>>> predicted_label = logits.argmax(-1).item()
>>> print(model.config.id2label[predicted_label])
eating spagh


Overview

DePlot was proposed in the paper DePlot: One-shot visual language reasoning by plot-to-table translation from Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, Yasemin Altun.
The abstract of the paper states the following:
Visual language such as charts and plots is ubiquitous in the human world. Comprehending plots and charts requires strong reasoning skills. Prior state-of-the-art (SOTA) models require at least tens of thousands of training examples and their reasoning capabilities are still much limited, especially on complex human-written queries. This paper presents the first one-shot solution to visual language reasoning. We decompose the challenge of visual language reasoning into two steps: (1) p


Overview

lot-to-text translation, and (2) reasoning over the translated text. The key in this method is a modality conversion module, named as DePlot, which translates the image of a plot or chart to a linearized table. The output of DePlot can then be directly used to prompt a pretrained large language model (LLM), exploiting the few-shot reasoning capabilities of LLMs. To obtain DePlot, we standardize the plot-to-table task by establishing unified task formats and metrics, and train DePlot end-to-end on this task. DePlot can then be used off-the-shelf together with LLMs in a plug-and-play fashion. Compared with a SOTA model finetuned on more than >28k data points, DePlot+LLM with just one-shot prompting achieves a 24.0% improvement over finetuned SOTA on human-written queries from the task of chart QA


Model description

DePlot is a model that is trained using Pix2Struct architecture. You can find more information about Pix2Struct in the Pix2Struct documentation.
DePlot is a Visual Question Answering subset of Pix2Struct architecture. It renders the input question on the image and predicts the answer.


Usage

Currently one checkpoint is available for DePlot:
google/deplot: DePlot fine-tuned on ChartQA dataset
  Copied from transformers import AutoProcessor, Pix2StructForConditionalGeneration
import requests
from PIL import Image
model = Pix2StructForConditionalGeneration.from_pretrained("google/deplot")
processor = AutoProcessor.from_pretrained("google/deplot")
url = "https://raw.githubusercontent.com/vis-nlp/ChartQA/main/ChartQA%20Dataset/val/png/5090.png"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(images=image, text="Generate underlying data table of the figure below:", return_tensors="pt")
predictions = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(predictions[0], skip_special_tokens=True))


Fine-tuning

To fine-tune DePlot, refer to the pix2struct fine-tuning notebook. For Pix2Struct models, we have found out that fine-tuning the model with Adafactor and cosine learning rate scheduler leads to faster convergence:
  Copied from transformers.optimization import Adafactor, get_cosine_schedule_with_warmup
optimizer = Adafactor(self.parameters(), scale_parameter=False, relative_step=False, lr=0.01, weight_decay=1e-05)
scheduler = get_cosine_schedule_with_warmup(optimizer, num_warmup_steps=1000, num_training_steps=40000)


Overview

The ALIGN model was proposed in Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision by Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, Tom Duerig. ALIGN is a multi-modal vision and language model. It can be used for image-text similarity and for zero-shot image classification. ALIGN features a dual-encoder architecture with EfficientNet as its vision encoder and BERT as its text encoder, and learns to align visual and text representations with contrastive learning. Unlike previous work, ALIGN leverages a massive noisy dataset and shows that the scale of the corpus can be used to achieve SOTA representations with a simple recipe.
The abstract from the paper is the following:
Pre-trained representations are b


Overview

ecoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as ImageNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-pr


Overview

ocessing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries


Usage

ALIGN uses EfficientNet to get visual features and BERT to get the text features. Both the text and visual features are then projected to a latent space with identical dimension. The dot product between the projected image and text features is then used as a similarity score.
AlignProcessor wraps EfficientNetImageProcessor and BertTokenizer into a single instance to both encode the text and preprocess the images. The following example shows how to get the image-text similarity scores using AlignProcessor and AlignModel.
  Copied import requests
import torch
from PIL import Image
from transformers import AlignProcessor, AlignModel
processor = AlignProcessor.from_pretrained("kakaobrain/align-base")
model = AlignM


Usage

odel.from_pretrained("kakaobrain/align-base")
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
candidate_labels = ["an image of a cat", "an image of a dog"]
inputs = processor(text=candidate_labels, images=image, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)
# this is the image-text similarity score
logits_per_image = outputs.logits_per_image
# we can take the softmax to get the label probabilities
probs = logits_per_image.softmax(dim=1)
print(probs)
This model was contributed by Alara Dirik.
The original code is not released, this implementation is based on the Kakao Brain implementation based on the original paper.


Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ALIGN.
A blog post on ALIGN and the COYO-700M dataset. A zero-shot image classification demo. Model card of kakaobrain/align-base model.
If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it. The resource should ideally demonstrate something new instead of duplicating an existing resource.


AlignConfig

class transformers.AlignConfig  < source > ( text_config = None vision_config = None projection_dim = 640 temperature_init_value = 1.0 initializer_range = 0.02 **kwargs  )   Parameters   text_config (dict, optional) —
Dictionary of configuration options used to initialize AlignTextConfig.   vision_config (dict, optional) —
Dictionary of configuration options used to initialize AlignVisionConfig.   projection_dim (int, optional, defaults to 640) —
Dimentionality of text and vision projection layers.   temperature_init_value (float, optional, defaults to 1.0) —
The inital value of the temperature paramter. Default is used as per the original ALIGN implementation.   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.


AlignConfig

 kwargs (optional) —
Dictionary of keyword arguments.    AlignConfig is the configuration class to store the configuration of a AlignModel. It is used to
instantiate a ALIGN model according to the specified arguments, defining the text model and vision model configs.
Instantiating a configuration with the defaults will yield a similar configuration to that of the ALIGN
kakaobrain/align-base architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import AlignConfig, AlignModel
>>> # Initializing a AlignConfig with kakaobrain/align-base style configuration
>>> configuration = AlignConfig()
>>> # Initializing a AlignModel (with random weights) from the kaka


AlignConfig

obrain/align-base style configuration
>>> model = AlignModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config
>>> # We can also initialize a AlignConfig from a AlignTextConfig and a AlignVisionConfig
>>> from transformers import AlignTextConfig, AlignVisionConfig
>>> # Initializing ALIGN Text and Vision configurations
>>> config_text = AlignTextConfig()
>>> config_vision = AlignVisionConfig()
>>> config = AlignConfig.from_text_vision_configs(config_text, config_vision)  from_text_vision_configs  < source > ( text_config: AlignTextConfig vision_config: AlignVisionConfig **kwargs  ) → AlignConfig  Returns
AlignConfig

An instance of a configuration object
  Instantiate a AlignConfig (or a derived class) from align text model configuration and align vision model
configuration.


AlignTextConfig

class transformers.AlignTextConfig  < source > ( vocab_size = 30522 hidden_size = 768 num_hidden_layers = 12 num_attention_heads = 12 intermediate_size = 3072 hidden_act = 'gelu' hidden_dropout_prob = 0.1 attention_probs_dropout_prob = 0.1 max_position_embeddings = 512 type_vocab_size = 2 initializer_range = 0.02 layer_norm_eps = 1e-12 pad_token_id = 0 position_embedding_type = 'absolute' use_cache = True **kwargs  )   Parameters   vocab_size (int, optional, defaults to 30522) —
Vocabulary size of the Align Text model. Defines the number of different tokens that can be represented by
the inputs_ids passed when calling AlignTextModel.   hidden_size (int, optional, defaults to 768) —
Dimensionality of the encoder layers and the pooler layer.   num_hidden_layers (int, optional, defaults to 12) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 12) —
Number of attention heads for each attention layer in the Transformer encoder.


AlignTextConfig

  intermediate_size (int, optional, defaults to 3072) —
Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.   hidden_act (str or Callable, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "silu" and "gelu_new" are supported.   hidden_dropout_prob (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.   attention_probs_dropout_prob (float, optional, defaults to 0.1) —
The dropout ratio for the attention probabilities.   max_position_embeddings (int, optional, defaults to 512) —
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).   type_vocab_size (int, optional, defaults to 2) —
The vocabulary size of the token_type_ids passed when calling AlignTextModel.   initializer_r


AlignTextConfig

ange (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   layer_norm_eps (float, optional, defaults to 1e-12) —
The epsilon used by the layer normalization layers.   pad_token_id (int, optional, defaults to 0) —
Padding token id.   position_embedding_type (str, optional, defaults to "absolute") —
Type of position embedding. Choose one of "absolute", "relative_key", "relative_key_query". For
positional embeddings use "absolute". For more information on "relative_key", please refer to
Self-Attention with Relative Position Representations (Shaw et al.).
For more information on "relative_key_query", please refer to Method 4 in Improve Transformer Models
with Better Relative Position Embeddings (Huang et al.).   use_cache (bool, optional, defaults to True) —
Whether or not the model should return the last key/values attentions (not used by all models). Only
relevant if config.is_decoder=True.    Thi


AlignTextConfig

s is the configuration class to store the configuration of a AlignTextModel. It is used to instantiate a
ALIGN text encoder according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the text encoder of the ALIGN
kakaobrain/align-base architecture. The default values here are
copied from BERT. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import AlignTextConfig, AlignTextModel
>>> # Initializing a AlignTextConfig with kakaobrain/align-base style configuration
>>> configuration = AlignTextConfig()
>>> # Initializing a AlignTextModel (with random weights) from the kakaobrain/align-base style configuration
>>> model = AlignTextModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config


AlignVisionConfig

class transformers.AlignVisionConfig  < source > ( num_channels: int = 3 image_size: int = 600 width_coefficient: float = 2.0 depth_coefficient: float = 3.1 depth_divisor: int = 8 kernel_sizes: typing.List[int] = [3, 3, 5, 3, 5, 5, 3] in_channels: typing.List[int] = [32, 16, 24, 40, 80, 112, 192] out_channels: typing.List[int] = [16, 24, 40, 80, 112, 192, 320] depthwise_padding: typing.List[int] = [] strides: typing.List[int] = [1, 2, 2, 2, 1, 2, 1] num_block_repeats: typing.List[int] = [1, 2, 2, 3, 3, 4, 1] expand_ratios: typing.List[int] = [1, 6, 6, 6, 6, 6, 6] squeeze_expansion_ratio: float = 0.25 hidden_act: str = 'swish' hidden_dim: int = 2560 pooling_type: str = 'mean' initializer_range: float = 0.02 batch_norm_eps: float = 0.001 batch_norm_momentum: float = 0.99 drop_connect_rate: float = 0.2 **kwargs  )   Parameters   num_channels (int, option


AlignVisionConfig

al, defaults to 3) —
The number of input channels.   image_size (int, optional, defaults to 600) —
The input image size.   width_coefficient (float, optional, defaults to 2.0) —
Scaling coefficient for network width at each stage.   depth_coefficient (float, optional, defaults to 3.1) —
Scaling coefficient for network depth at each stage.   depth_divisor int, optional, defaults to 8) —
A unit of network width.   kernel_sizes (List[int], optional, defaults to [3, 3, 5, 3, 5, 5, 3]) —
List of kernel sizes to be used in each block.   in_channels (List[int], optional, defaults to [32, 16, 24, 40, 80, 112, 192]) —
List of input channel sizes to be used in each block for convolutional layers.   out_channels (List[int], optional, defaults to [16, 24, 40, 80, 112, 192, 320]) —
List of output channel sizes to be used in each block for convolutional layers.   d


AlignVisionConfig

epthwise_padding (List[int], optional, defaults to []) —
List of block indices with square padding.   strides (List[int], optional, defaults to [1, 2, 2, 2, 1, 2, 1]) —
List of stride sizes to be used in each block for convolutional layers.   num_block_repeats (List[int], optional, defaults to [1, 2, 2, 3, 3, 4, 1]) —
List of the number of times each block is to repeated.   expand_ratios (List[int], optional, defaults to [1, 6, 6, 6, 6, 6, 6]) —
List of scaling coefficient of each block.   squeeze_expansion_ratio (float, optional, defaults to 0.25) —
Squeeze expansion ratio.   hidden_act (str or function, optional, defaults to "silu") —
The non-linear activation function (function or string) in each block. If string, "gelu", "relu",
"selu", “gelu_new”, “silu”and“mish”` are supported.   hiddem_dim (int, optional, defaults to 1280) —
The hidden dimensio


AlignVisionConfig

n of the layer before the classification head.   pooling_type (str or function, optional, defaults to "mean") —
Type of final pooling to be applied before the dense classification head. Available options are ["mean",
"max"]   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   batch_norm_eps (float, optional, defaults to 1e-3) —
The epsilon used by the batch normalization layers.   batch_norm_momentum (float, optional, defaults to 0.99) —
The momentum used by the batch normalization layers.   drop_connect_rate (float, optional, defaults to 0.2) —
The drop rate for skip connections.    This is the configuration class to store the configuration of a AlignVisionModel. It is used to instantiate a
ALIGN vision encoder according to the specified arguments,


AlignVisionConfig

 defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the vision encoder of the ALIGN
kakaobrain/align-base architecture. The default values are copied
from EfficientNet (efficientnet-b7) Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import AlignVisionConfig, AlignVisionModel
>>> # Initializing a AlignVisionConfig with kakaobrain/align-base style configuration
>>> configuration = AlignVisionConfig()
>>> # Initializing a AlignVisionModel (with random weights) from the kakaobrain/align-base style configuration
>>> model = AlignVisionModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.confi


AlignProcessor

class transformers.AlignProcessor  < source > ( image_processor tokenizer  )   Parameters   image_processor (EfficientNetImageProcessor) —
The image processor is a required input.   tokenizer ([BertTokenizer, BertTokenizerFast]) —
The tokenizer is a required input.    Constructs an ALIGN processor which wraps EfficientNetImageProcessor and
BertTokenizer/BertTokenizerFast into a single processor that interits both the image processor and
tokenizer functionalities. See the __call__() and decode() for more
information.  batch_decode  < source > ( *args **kwargs  )    This method forwards all its arguments to BertTokenizerFast’s batch_decode(). Please
refer to the docstring of this method for more information.  decode  < source > ( *args **kwargs  )    This method forwards all its arguments to BertTokenizerFast’s decode(). Please refer to
the docstring of this method for more information.


AlignModel

class transformers.AlignModel  < source > ( config: AlignConfig  )   Parameters   config (AlignConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None pixel_values: typing.Optional[torch.FloatTensor] = None attention_mask: typing.Optional[torch.Tensor] = None token_


AlignModel

type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None return_loss: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  )   Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?
attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional):
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that


AlignModel

 are masked.
What are attention masks?
position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional):
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?
token_type_ids (torch.LongTensor of shape ({0}), optional):
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?
head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional):
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
inputs_embeds (torch.FloatTensor of shape ({0}, hidden_size), optional):
Optionally, instead of passing input_ids you can choose to directly


AlignModel

pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.
pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)):
Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using
AutoImageProcessor. See EfficientNetImageProcessor.call() for details.
return_loss (bool, optional):
Whether or not to return the contrastive loss.
output_attentions (bool, optional):
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.
output_hidden_states (bool, optional):
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.
return_dict (bool, optional):
Whether or not to return a ModelOutput instead of a plain tuple.    Th


AlignModel

e AlignModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  get_text_features  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → text_features (torch.FloatTensor of shape (batch_size, output_dim)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequenc


AlignModel

e_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   token_type_ids (torch.LongTensor of shape ({0}), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0


AlignModel

, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape ({0}, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of


AlignModel

 all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
text_features (torch.FloatTensor of shape (batch_size, output_dim)

The text embeddings obtained by
applying the projection layer to the pooled output of AlignTextModel.
  The AlignModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoTokenizer, AlignModel
>>> model = AlignModel.from_pretrained("kakaobrain/align-base")
>>> tokenizer = AutoTokenizer.from_pretrained("kakaobrain/align-base")
>>> inputs = tokenizer(["a photo of a cat", "a photo of


AlignModel

a dog"], padding=True, return_tensors="pt")
>>> text_features = model.get_text_features(**inputs)  get_image_features  < source > ( pixel_values: typing.Optional[torch.FloatTensor] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → image_features (torch.FloatTensor of shape (batch_size, output_dim)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using
AutoImageProcessor. See EfficientNetImageProcessor.call() for details.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
image_features (torch.FloatTensor of shape (ba


AlignModel

tch_size, output_dim)

The image embeddings obtained by
applying the projection layer to the pooled output of AlignVisionModel.
  The AlignModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, AlignModel
>>> model = AlignModel.from_pretrained("kakaobrain/align-base")
>>> processor = AutoProcessor.from_pretrained("kakaobrain/align-base")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt")
>>> image_features = model.get_image_features(**


AlignTextModel

class transformers.AlignTextModel  < source > ( config: AlignTextConfig add_pooling_layer: bool = True  )   Parameters   config (AlignConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The text model from ALIGN without any head or projection on top.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] =


AlignTextModel

None attention_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optiona


AlignTextModel

l) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   token_type_ids (torch.LongTensor of shape ({0}), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not mas


AlignTextModel

ked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape ({0}, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions or tuple(torch.FloatTensor)

A transformers.mo


AlignTextModel

deling_outputs.BaseModelOutputWithPoolingAndCrossAttentions or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.align.configuration_align.AlignTextConfig'>) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing
through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns
the classification token after processing through a linear layer and a tanh activation function. The linear
layer weights are trained from the next sentence prediction (classi


AlignTextModel

fication) objective during pretraining.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attenti


AlignTextModel

ons=True and config.add_cross_attention=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and optionally if
config.is_encoder_decoder=True 2 additional tensors of shape (batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if
config.is_e


AlignTextModel

ncoder_decoder=True in the cross-attention blocks) that can be used (see past_key_values
input) to speed up sequential decoding.
  The AlignTextModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoTokenizer, AlignTextModel
>>> model = AlignTextModel.from_pretrained("kakaobrain/align-base")
>>> tokenizer = AutoTokenizer.from_pretrained("kakaobrain/align-base")
>>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_state = outputs.last_hidden_state
>>> pooled_output = outputs.pooler_output  # pooled (EOS token) sta


AlignVisionModel

class transformers.AlignVisionModel  < source > ( config: AlignVisionConfig  )   Parameters   config (AlignConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The vision model from ALIGN without any head or projection on top.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( pixel_values: typing.Optional[torch.FloatTensor] = None output_hidden_states: ty


AlignVisionModel

ping.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.BaseModelOutputWithPoolingAndNoAttention or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using
AutoImageProcessor. See EfficientNetImageProcessor.call() for details.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.BaseModelOutputWithPoolingAndNoAttention or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutputWithPoolingAndNoAttention or a tuple of
torch.FloatTensor (if return_d


AlignVisionModel

ict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.align.configuration_align.AlignVisionConfig'>) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, num_channels, height, width)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — Last layer hidden-state after a pooling operation on the spatial dimensions.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, num_channels, height, width).
Hidden-states of the model at the output of each layer plus the optional initial embeddin


AlignVisionModel

g outputs.
  The AlignVisionModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, AlignVisionModel
>>> model = AlignVisionModel.from_pretrained("kakaobrain/align-base")
>>> processor = AutoProcessor.from_pretrained("kakaobrain/align-base")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_state = outputs.last_hidden_state
>>> pooled_output = outputs.pooler_output  # pooled CLS state


Overview

The Donut model was proposed in OCR-free Document Understanding Transformer by
Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, Seunghyun Park.
Donut consists of an image Transformer encoder and an autoregressive text Transformer decoder to perform document understanding
tasks such as document image classification, form understanding and visual question answering.
The abstract from the paper is the following:
Understanding document images (e.g., invoices) is a core but challenging task since it requires complex functions such as reading text and a holistic understanding of the document. Current Visual Document Understanding (VDU) meth


Overview

ods outsource the task of reading text to off-the-shelf Optical Character Recognition (OCR) engines and focus on the understanding task with the OCR outputs. Although such OCR-based approaches have shown promising performance, they suffer from 1) high computational costs for using OCR; 2) inflexibility of OCR models on languages or types of document; 3) OCR error propagation to the subsequent process. To address these issues, in this paper, we introduce a novel OCR-free VDU model named Donut, which stands for Document understanding transformer. As the first step in OCR-free VDU research, we propose a simple architecture (i.e., Transformer) with a pre-training objective (i.e., cross-entropy loss). Donut is conc


Overview

eptually simple yet effective. Through extensive experiments and analyses, we show a simple OCR-free VDU model, Donut, achieves state-of-the-art performances on various VDU tasks in terms of both speed and accuracy. In addition, we offer a synthetic data generator that helps the model pre-training to be flexible in various languages and domains.
Donut high-level overview. Taken from the original paper.
This model was contributed by nielsr. The original code can be found
here.
Tips:
The quickest way to get started with Donut is by checking the tutorial
notebooks, which show how to use the model
at inference time as well as fine-tuning on custom data. Donut is always used within the VisionEncoderDecoder framewor


Inference

Donut’s VisionEncoderDecoder model accepts images as input and makes use of
generate() to autoregressively generate text given the input image.
The DonutImageProcessor class is responsible for preprocessing the input image and
[XLMRobertaTokenizer/XLMRobertaTokenizerFast] decodes the generated target tokens to the target string. The
DonutProcessor wraps DonutImageProcessor and [XLMRobertaTokenizer/XLMRobertaTokenizerFast]
into a single instance to both extract the input features and decode the predicted token ids.
Step-by-step Document Image Classification
  Copied >>> import re
>>> from transformers import DonutProcessor, VisionEncoderDecoderModel
>>> from datasets import load_dataset
>>> import torch
>>> processor = DonutProcessor.from_pretrained("naver-clova-ix/donut-base-finetuned-rvlcdip")
>>> model = VisionEncoderDecoderModel.from_pretrained("naver-clova-ix/donut-base-finetuned-rvlcdip")
>>> device = "cuda" if torch.cuda.is_available() else "cpu"


Inference

>>> model.to(device)
>>> # load document image
>>> dataset = load_dataset("hf-internal-testing/example-documents", split="test")
>>> image = dataset[1]["image"]
>>> # prepare decoder inputs
>>> task_prompt = "<s_rvlcdip>"
>>> decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors="pt").input_ids
>>> pixel_values = processor(image, return_tensors="pt").pixel_values
>>> outputs = model.generate(
...     pixel_values.to(device),
...     decoder_input_ids=decoder_input_ids.to(device),
...     max_length=model.decoder.config.max_position_embeddings,
...     pad_token_id=processor.tokenizer.pad_token_id,
...     eos_token_id=processor.tokenizer.eos_token_id,
...     use_cache=True,
...     bad_words_ids=[[processor.tokenizer.unk_token_id]],
...     return_dict_in_generate=True,
... )
>>> sequence = processor.batch_decode(outputs.sequences)[0]
>>> sequence = sequence.replace(processor.tokenizer.eos_token, "").replace(proc


Inference

essor.tokenizer.pad_token, "")
>>> sequence = re.sub(r"<.*?>", "", sequence, count=1).strip()  # remove first task start token
>>> print(processor.token2json(sequence))
{'class': 'advertisement'}
Step-by-step Document Parsing
  Copied >>> import re
>>> from transformers import DonutProcessor, VisionEncoderDecoderModel
>>> from datasets import load_dataset
>>> import torch
>>> processor = DonutProcessor.from_pretrained("naver-clova-ix/donut-base-finetuned-cord-v2")
>>> model = VisionEncoderDecoderModel.from_pretrained("naver-clova-ix/donut-base-finetuned-cord-v2")
>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> model.to(device)
>>> # load document image
>>> dataset = load_dataset("hf-internal-testing/example-documents", split="test")
>>> image = dataset[2]["image"]
>>> # prepare decoder inputs
>>> task_prompt = "<s_cord-v2>"
>>> decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors="pt").input_ids
>


Inference

>> pixel_values = processor(image, return_tensors="pt").pixel_values
>>> outputs = model.generate(
...     pixel_values.to(device),
...     decoder_input_ids=decoder_input_ids.to(device),
...     max_length=model.decoder.config.max_position_embeddings,
...     pad_token_id=processor.tokenizer.pad_token_id,
...     eos_token_id=processor.tokenizer.eos_token_id,
...     use_cache=True,
...     bad_words_ids=[[processor.tokenizer.unk_token_id]],
...     return_dict_in_generate=True,
... )
>>> sequence = processor.batch_decode(outputs.sequences)[0]
>>> sequence = sequence.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "")
>>> sequence = re.sub(r"<.*?>", "", sequence, count=1).strip()  # remove first task start token
>>> print(processor.token2json(sequence))
{'menu': {'nm': 'CINNAMON SUGAR', 'unitprice': '17,000', 'cnt': '1 x', 'price': '17,000'}, 'sub_total': {'subtotal_price': '17,000'}, 'total': {'total_price': '17,000',


Inference

 'cashprice': '20,000', 'changeprice': '3,000'}}
Step-by-step Document Visual Question Answering (DocVQA)
  Copied >>> import re
>>> from transformers import DonutProcessor, VisionEncoderDecoderModel
>>> from datasets import load_dataset
>>> import torch
>>> processor = DonutProcessor.from_pretrained("naver-clova-ix/donut-base-finetuned-docvqa")
>>> model = VisionEncoderDecoderModel.from_pretrained("naver-clova-ix/donut-base-finetuned-docvqa")
>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> model.to(device)
>>> # load document image from the DocVQA dataset
>>> dataset = load_dataset("hf-internal-testing/example-documents", split="test")
>>> image = dataset[0]["image"]
>>> # prepare decoder inputs
>>> task_prompt = "<s_docvqa><s_question>{user_input}</s_question><s_answer>"
>>> question = "When is the coffee break?"
>>> prompt = task_prompt.replace("{user_input}", question)
>>> decoder_input_ids = processor.tokenizer(prompt, add_special_t


Inference

okens=False, return_tensors="pt").input_ids
>>> pixel_values = processor(image, return_tensors="pt").pixel_values
>>> outputs = model.generate(
...     pixel_values.to(device),
...     decoder_input_ids=decoder_input_ids.to(device),
...     max_length=model.decoder.config.max_position_embeddings,
...     pad_token_id=processor.tokenizer.pad_token_id,
...     eos_token_id=processor.tokenizer.eos_token_id,
...     use_cache=True,
...     bad_words_ids=[[processor.tokenizer.unk_token_id]],
...     return_dict_in_generate=True,
... )
>>> sequence = processor.batch_decode(outputs.sequences)[0]
>>> sequence = sequence.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "")
>>> sequence = re.sub(r"<.*?>", "", sequence, count=1).strip()  # remove first task start token
>>> print(processor.token2json(sequence))
{'question': 'When is the coffee break?', 'answer': '11-14 to 11:39 a.m.'}
See the model hub to look for Donut checkpoints.


DonutSwinConfig

class transformers.DonutSwinConfig  < source > ( image_size = 224 patch_size = 4 num_channels = 3 embed_dim = 96 depths = [2, 2, 6, 2] num_heads = [3, 6, 12, 24] window_size = 7 mlp_ratio = 4.0 qkv_bias = True hidden_dropout_prob = 0.0 attention_probs_dropout_prob = 0.0 drop_path_rate = 0.1 hidden_act = 'gelu' use_absolute_embeddings = False initializer_range = 0.02 layer_norm_eps = 1e-05 **kwargs  )   Parameters   image_size (int, optional, defaults to 224) —
The size (resolution) of each image.   patch_size (int, optional, defaults to 4) —
The size (resolution) of each patch.   num_channels (int, optional, defaults to 3) —
The number of input channels.   embed_dim (int, optional, defaults to 96) —
Dimensionality of patch embedding.   depths (list(int), optional, default


DonutSwinConfig

s to [2, 2, 6, 2]) —
Depth of each layer in the Transformer encoder.   num_heads (list(int), optional, defaults to [3, 6, 12, 24]) —
Number of attention heads in each layer of the Transformer encoder.   window_size (int, optional, defaults to 7) —
Size of windows.   mlp_ratio (float, optional, defaults to 4.0) —
Ratio of MLP hidden dimensionality to embedding dimensionality.   qkv_bias (bool, optional, defaults to True) —
Whether or not a learnable bias should be added to the queries, keys and values.   hidden_dropout_prob (float, optional, defaults to 0.0) —
The dropout probability for all fully connected layers in the embeddings and encoder.   attention_probs_dropout_prob (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.   drop_path


DonutSwinConfig

_rate (float, optional, defaults to 0.1) —
Stochastic depth rate.   hidden_act (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder. If string, "gelu", "relu",
"selu" and "gelu_new" are supported.   use_absolute_embeddings (bool, optional, defaults to False) —
Whether or not to add absolute position embeddings to the patch embeddings.   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   layer_norm_eps (float, optional, defaults to 1e-05) —
The epsilon used by the layer normalization layers.    This is the configuration class to store the configuration of a DonutSwinModel. It is used to instantiate a
Don


DonutSwinConfig

ut model according to the specified arguments, defining the model architecture. Instantiating a configuration
with the defaults will yield a similar configuration to that of the Donut
naver-clova-ix/donut-base architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import DonutSwinConfig, DonutSwinModel
>>> # Initializing a Donut naver-clova-ix/donut-base style configuration
>>> configuration = DonutSwinConfig()
>>> # Randomly initializing a model from the naver-clova-ix/donut-base style configuration
>>> model = DonutSwinModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.con


DonutImageProcessor

class transformers.DonutImageProcessor  < source > ( do_resize: bool = True size: typing.Dict[str, int] = None resample: Resampling = <Resampling.BILINEAR: 2> do_thumbnail: bool = True do_align_long_axis: bool = False do_pad: bool = True do_rescale: bool = True rescale_factor: typing.Union[int, float] = 0.00392156862745098 do_normalize: bool = True image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float], NoneType] = None **kwargs  )   Parameters   do_resize (bool, optional, defaults to True) —
Whether to resize the image’s (height, width) dimensions to the specified size. Can be overridden by
do_resize in the preprocess method.   size (Dict[str, int] optional, defaults to {"shortest_edge" -- 224}):
Size of the image after resizing. The shortest edge of the image is resized to size[“shortest_edge”], with
the longest edge resized to keep the input aspect ratio. Can be overridden by size in the preprocess
method.   resample


DonutImageProcessor

 (PILImageResampling, optional, defaults to PILImageResampling.BILINEAR) —
Resampling filter to use if resizing the image. Can be overridden by resample in the preprocess method.   do_thumbnail (bool, optional, defaults to True) —
Whether to resize the image using thumbnail method.   do_align_long_axis (bool, optional, defaults to False) —
Whether to align the long axis of the image with the long axis of size by rotating by 90 degrees.   do_pad (bool, optional, defaults to True) —
Whether to pad the image. If random_padding is set to True in preprocess, each image is padded with a
random amont of padding on each size, up to the largest image size in the batch. Otherwise, all images are
padded to the largest image size in the batch.   do_rescale (bool, optional, defaults to True) —
Whether to rescale the image by the specified scale rescale_factor. Can be overridden by do_rescale in
the preprocess method.   rescale_factor (int or float, optional, defaults to 1/255) —
Scale factor to


DonutImageProcessor

use if rescaling the image. Can be overridden by rescale_factor in the preprocess
method.
do_normalize —
Whether to normalize the image. Can be overridden by do_normalize in the preprocess method.   image_mean (float or List[float], optional, defaults to IMAGENET_STANDARD_MEAN) —
Mean to use if normalizing the image. This is a float or list of floats the length of the number of
channels in the image. Can be overridden by the image_mean parameter in the preprocess method.   image_std (float or List[float], optional, defaults to IMAGENET_STANDARD_STD) —
Image standard deviation.    Constructs a Donut image processor.  preprocess  < source > ( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]] do_resize: bool = None size: typing.Dict[str, int] = None resample: Resampling = None do_thumbnail: bool = None do_align_long_axis: bool = N


DonutImageProcessor

one do_pad: bool = None random_padding: bool = False do_rescale: bool = None rescale_factor: float = None do_normalize: bool = None image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float], NoneType] = None return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None data_format: typing.Optional[transformers.image_utils.ChannelDimension] = <ChannelDimension.FIRST: 'channels_first'> input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None **kwargs  )   Parameters   images (ImageInput) —
Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If
passing in images with pixel values between 0 and 1, set do_rescale=False.   do_resize (bool, optional, defaults to self.do_resize) —
Whether to resize the image.   size (Dict[str, int], optional, defaults to self.size) —
Size of the image after resizing. Shortest edge of t


DonutImageProcessor

he image is resized to min(size[“height”],
size[“width”]) with the longest edge resized to keep the input aspect ratio.   resample (int, optional, defaults to self.resample) —
Resampling filter to use if resizing the image. This can be one of the enum PILImageResampling. Only
has an effect if do_resize is set to True.   do_thumbnail (bool, optional, defaults to self.do_thumbnail) —
Whether to resize the image using thumbnail method.   do_align_long_axis (bool, optional, defaults to self.do_align_long_axis) —
Whether to align the long axis of the image with the long axis of size by rotating by 90 degrees.   do_pad (bool, optional, defaults to self.do_pad) —
Whether to pad the image. If random_padding is set to True, each image is padded with a random
amont of padding on each size, up to the largest image size in the batch. Otherwise, all images are
padded to the largest image size in the batch.   random_padding (bool, optional, defaults to self.random_padding) —
Whether to use random


DonutImageProcessor

 padding when padding the image. If True, each image in the batch with be padded
with a random amount of padding on each side up to the size of the largest image in the batch.   do_rescale (bool, optional, defaults to self.do_rescale) —
Whether to rescale the image pixel values.   rescale_factor (float, optional, defaults to self.rescale_factor) —
Rescale factor to rescale the image by if do_rescale is set to True.   do_normalize (bool, optional, defaults to self.do_normalize) —
Whether to normalize the image.   image_mean (float or List[float], optional, defaults to self.image_mean) —
Image mean to use for normalization.   image_std (float or List[float], optional, defaults to self.image_std) —
Image standard deviation to use for normalization.   return_tensors (str or TensorType, optional) —
The type of tensors to return. Can be one of:
Unset: Return a list of np.ndarray.
TensorType.TENSORFLOW or 'tf': Return a batch of type tf.Tensor.
TensorType.PYTORCH or 'pt': Return a batch of


DonutImageProcessor

 type torch.Tensor.
TensorType.NUMPY or 'np': Return a batch of type np.ndarray.
TensorType.JAX or 'jax': Return a batch of type jax.numpy.ndarray.
   data_format (ChannelDimension or str, optional, defaults to ChannelDimension.FIRST) —
The channel dimension format for the output image. Can be one of:
ChannelDimension.FIRST: image in (num_channels, height, width) format.
ChannelDimension.LAST: image in (height, width, num_channels) format.
Unset: defaults to the channel dimension format of the input image.
   input_data_format (ChannelDimension or str, optional) —
The channel dimension format for the input image. If unset, the channel dimension format is inferred
from the input image. Can be one of:
"channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
"channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
"none" or ChannelDimension.NONE: image in (height, width) format.
    Preprocess an image or batch of images


DonutFeatureExtractor

class transformers.DonutFeatureExtractor  < source > ( *args **kwargs  )     __call__  < source > ( images **kwargs  )    Preprocess an image or a batch of images.


DonutProcessor

class transformers.DonutProcessor  < source > ( image_processor = None tokenizer = None **kwargs  )   Parameters   image_processor (DonutImageProcessor, optional) —
An instance of DonutImageProcessor. The image processor is a required input.   tokenizer ([XLMRobertaTokenizer/XLMRobertaTokenizerFast], optional) —
An instance of [XLMRobertaTokenizer/XLMRobertaTokenizerFast]. The tokenizer is a required input.    Constructs a Donut processor which wraps a Donut image processor and an XLMRoBERTa tokenizer into a single
processor. DonutProcessor offers all the functionalities of DonutImageProcessor and
[XLMRobertaTokenizer/XLMRobertaTokenizerFast]. See the call() and
decode() for more information.  __call__  < source > ( *args **kwargs  )    When used in normal mode, this method forwards all its arguments to AutoImageProcessor’s
__call__() and returns its output. If used in the context
as_target_processor() this method forwards all i


DonutProcessor

ts arguments to DonutTokenizer’s
~DonutTokenizer.__call__. Please refer to the doctsring of the above two methods for more information.  from_pretrained  < source > ( pretrained_model_name_or_path: typing.Union[str, os.PathLike] cache_dir: typing.Union[str, os.PathLike, NoneType] = None force_download: bool = False local_files_only: bool = False token: typing.Union[bool, str, NoneType] = None revision: str = 'main' **kwargs  )   Parameters   pretrained_model_name_or_path (str or os.PathLike) —
This can be either:
a string, the model id of a pretrained feature_extractor hosted inside a model repo on
huggingface.co. Valid model ids can be located at the root-level, like bert-base-uncased, or
namespaced under a user or organization name, like dbmdz/bert-base-german-cased.
a path to a directory containing a feature extractor file saved using the
save_pretrained() method, e.g., ./my_model_directory/.
a path or url to a saved feature


DonutProcessor

extractor JSON file, e.g.,
./my_model_directory/preprocessor_config.json.
**kwargs —
Additional keyword arguments passed along to both
from_pretrained() and
~tokenization_utils_base.PreTrainedTokenizer.from_pretrained.
    Instantiate a processor associated with a pretrained model. This class method is simply calling the feature extractor
from_pretrained(), image processor
ImageProcessingMixin and the tokenizer
~tokenization_utils_base.PreTrainedTokenizer.from_pretrained methods. Please refer to the docstrings of the
methods above for more information.  save_pretrained  < source > ( save_directory push_to_hub: bool = False **kwargs  )   Parameters   save_directory (str or os.PathLike) —
Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will
be created if it does not exist).   push_to_hub (bool, optional, defaults to False) —
Whether or not to push your model to the Hugging Face mode


DonutProcessor

l hub after saving it. You can specify the
repository you want to push to with repo_id (will default to the name of save_directory in your
namespace).   kwargs (Dict[str, Any], optional) —
Additional key word arguments passed along to the push_to_hub() method.    Saves the attributes of this processor (feature extractor, tokenizer…) in the specified directory so that it
can be reloaded using the from_pretrained() method. This class method is simply calling save_pretrained() and
save_pretrained(). Please refer to the docstrings of the
methods above for more information.  batch_decode  < source > ( *args **kwargs  )    This method forwards all its arguments to DonutTokenizer’s batch_decode(). Please refer
to the docstring of this method for more information.  decode  < source > ( *args **kwargs  )    This method forwards all its arguments to DonutTokenizer’s decode(). Please refer to the
docstring of this method for more informati


DonutSwinModel

class transformers.DonutSwinModel  < source > ( config add_pooling_layer = True use_mask_token = False  )   Parameters   config (DonutSwinConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare Donut Swin Model transformer outputting raw hidden-states without any specific head on top.
This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: typing.Optional[torch.FloatTensor] = None bool_masked_pos: typing.Optional[torch.BoolTensor] = None head_mask: typing.Optional[torch


DonutSwinModel

.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.donut.modeling_donut_swin.DonutSwinModelOutput or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
DonutImageProcessor.call() for details.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention laye