FlaxMarianMTModel

t_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the decoder at the output of each layer plus the initial embedding outputs.
decoder_attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
cross_attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to


FlaxMarianMTModel

 compute the
weighted average in the cross-attention heads.
encoder_last_hidden_state (jnp.ndarray of shape (batch_size, sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the initial embedding outputs.
encoder_attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the encoder, after the attention softmax, used to compute the weighted aver


FlaxMarianMTModel

age in the
self-attention heads.
  The FlaxMarianPreTrainedModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, FlaxMarianMTModel
>>> model = FlaxMarianMTModel.from_pretrained("Helsinki-NLP/opus-mt-en-de")
>>> tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de")
>>> text = "My friends are cool but they eat too many carbs."
>>> input_ids = tokenizer(text, max_length=64, return_tensors="jax").input_ids
>>> sequences = model.generate(input_ids, max_length=64, num_beams=2).sequences
>>> outputs = tokenizer.batch_decode(sequences, skip_special_tokens=True)
>>> # should give *Meine Freunde sind cool, aber sie essen zu viele Kohlenhydrate


Overview

The TVLT model was proposed in TVLT: Textless Vision-Language Transformer
by Zineng Tang, Jaemin Cho, Yixin Nie, Mohit Bansal (the first three authors contributed equally). The Textless Vision-Language Transformer (TVLT) is a model that uses raw visual and audio inputs for vision-and-language representation learning, without using text-specific modules such as tokenization or automatic speech recognition (ASR). It can perform various audiovisual and vision-language tasks like retrieval, question answering, etc.
The abstract from the paper is the following:
In this work, we present the Textless Vision-Language Transformer (TVLT), where homogeneous transformer blocks take raw visual and audio inputs for vision-and-language representation learning with minimal modality-specific design, and do not use text-specific modules such as tokenization or


Overview

automatic speech recognition (ASR). TVLT is trained by reconstructing masked patches of continuous video frames and audio spectrograms (masked autoencoding) and contrastive modeling to align video and audio. TVLT attains performance comparable to its text-based counterpart on various multimodal tasks, such as visual question answering, image retrieval, video retrieval, and multimodal sentiment analysis, with 28x faster inference speed and only 1/3 of the parameters. Our findings suggest the possibility of learning compact and efficient visual-linguistic representations from low-level visual and audio signals without assuming the prior existence of text.
Tips:
TVLT is a model that takes both pixel_values and audio_values as input. One can use TvltProcessor to prepare data for the model.
This processor wraps an image processor (for the image/vid


Overview

eo modality) and an audio feature extractor (for the audio modality) into one. TVLT is trained with images/videos and audios of various sizes: the authors resize and crop the input images/videos to 224 and limit the length of audio spectrogram to 2048. To make batching of videos and audios possible, the authors use a pixel_mask that indicates which pixels are real/padding and audio_mask that indicates which audio values are real/padding. The design of TVLT is very similar to that of a standard Vision Transformer (ViT) and masked autoencoder (MAE) as in ViTMAE. The difference is that the model includes embedding layers for the audio modality. The PyTorch version of this model is only available in torch 1.10 and higher.
TVLT architecture. Taken from the original paper.
The original code can be found here. This model was contributed by Zineng Tan


TvltConfig

class transformers.TvltConfig  < source > ( image_size = 224 spectrogram_length = 2048 frequency_length = 128 image_patch_size = [16, 16] audio_patch_size = [16, 16] num_image_channels = 3 num_audio_channels = 1 num_frames = 8 hidden_size = 768 num_hidden_layers = 12 num_attention_heads = 12 intermediate_size = 3072 hidden_act = 'gelu' hidden_dropout_prob = 0.0 attention_probs_dropout_prob = 0.0 initializer_range = 0.02 layer_norm_eps = 1e-06 qkv_bias = True use_mean_pooling = False decoder_num_attention_heads = 16 decoder_hidden_size = 512 decoder_num_hidden_layers = 8 decoder_intermediate_size = 2048 pixel_mask_ratio = 0.75 audio_mask_ratio = 0.15 audio_mask_type = 'frame-level' task_matching = True task_mae = True loss_type = 'classification' **kwargs  )   Parameters   image_size (int, optional, defaults to 224) —
The size (


TvltConfig

resolution) of each image.   spectrogram_length (int, optional, defaults to 2048) —
The time length of each audio spectrogram.   frequency_length (int, optional, defaults to 128) —
The frequency length of audio spectrogram.   image_patch_size (List[int], optional, defaults to [16, 16]) —
The size (resolution) of each image patch.   audio_patch_size (List[int], optional, defaults to [16, 16]) —
The size (resolution) of each audio patch.   num_image_channels (int, optional, defaults to 3) —
The number of input image channels.   num_audio_channels (int, optional, defaults to 1) —
The number of input audio channels.   num_frames (int, optional, defaults to 8) —
The maximum number of frames for an input video.   hidden_size (int, optional, defaults to 768) —
Dimensionality of the encoder layers and the pooler layer.   num_hidden_lay


TvltConfig

ers (int, optional, defaults to 12) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 12) —
Number of attention heads for each attention layer in the Transformer encoder.   intermediate_size (int, optional, defaults to 3072) —
Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.   hidden_act (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "selu" and "gelu_new" are supported.   hidden_dropout_prob (float, optional, defaults to 0.0) —
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.   attention_probs_dropout_prob (float, optional, defaults to 0.0) —
The dropout ratio for the attenti


TvltConfig

on probabilities.   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   layer_norm_eps (float, optional, defaults to 1e-06) —
The epsilon used by the layer normalization layers.   qkv_bias (bool, optional, defaults to True) —
Whether to add a bias to the queries, keys and values.   use_mean_pooling (bool, optional, defaults to False) —
Whether to mean pool the final hidden states instead of using the final hidden state of the [CLS] token.   decoder_num_attention_heads (int, optional, defaults to 16) —
Number of attention heads for each attention layer in the decoder.   decoder_hidden_size (int, optional, defaults to 512) —
Dimensionality of the decoder.   decoder_num_hidden_layers (int, optional, defaults to 8) —
Number of hid


TvltConfig

den layers in the decoder.   decoder_intermediate_size (int, optional, defaults to 2048) —
Dimensionality of the “intermediate” (i.e., feed-forward) layer in the decoder.   pixel_mask_ratio (float, optional, defaults to 0.75) —
Image patch masking ratio.   audio_mask_ratio (float, optional, defaults to 0.15) —
Audio patch masking ratio.   audio_mask_type (str, optional, defaults to "frame-level") —
Audio patch masking type, choose between “frame-level” and “patch-level”.   task_matching (bool, optional, defaults to True) —
Whether to use vision audio matching task in pretraining.   task_mae (bool, optional, defaults to True) —
Whether to use the masked auto-encoder (MAE) in pretraining.   loss_type (str, optional, defaults to "classification") —
Loss types including regression and classification.    This is the configuration cl


TvltConfig

ass to store the configuration of a TvltModel. It is used to instantiate a TVLT
model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
defaults will yield a similar configuration to that of the TVLT
ZinengTang/tvlt-base architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import TvltConfig, TvltModel
>>> # # Initializing a TVLT ZinengTang/tvlt-base style configuration
>>> configuration = TvltConfig()
>>> # # Initializing a model (with random weights) from the ZinengTang/tvlt-base style configuration
>>> model = TvltModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.con


TvltProcessor

class transformers.TvltProcessor  < source > ( image_processor feature_extractor  )   Parameters   image_processor (TvltImageProcessor) —
An instance of TvltImageProcessor. The image processor is a required input.   feature_extractor (TvltFeatureExtractor) —
An instance of TvltFeatureExtractor. The feature extractor is a required input.    Constructs a TVLT processor which wraps a TVLT image processor and TVLT feature extractor into a single processor. TvltProcessor offers all the functionalities of TvltImageProcessor and TvltFeatureExtractor. See the
docstring of call() for more information.  __call__  < source > ( images = None audio = None images_mixed = None sampling_rate = None mask_audio = False mask_pixel = False *args **kwargs  )    Forwards the images argument to TvltImageProcessor’s preprocess() and the audio
argument to TvltFeatureExtractor’s call(). Please refer to the docstring of the
above two methods for more information.


TvltImageProcessor

class transformers.TvltImageProcessor  < source > ( do_resize: bool = True size: typing.Dict[str, int] = None patch_size: typing.List[int] = [16, 16] num_frames: int = 8 resample: Resampling = <Resampling.BILINEAR: 2> do_center_crop: bool = True crop_size: typing.Dict[str, int] = None do_rescale: bool = True rescale_factor: typing.Union[int, float] = 0.00392156862745098 do_normalize: bool = True image_mean: typing.Union[float, typing.List[float], NoneType] = [0.5, 0.5, 0.5] image_std: typing.Union[float, typing.List[float], NoneType] = [0.5, 0.5, 0.5] init_mask_generator = False **kwargs  )   Parameters   do_resize (bool, optional, defaults to True) —
Whether to resize the image’s (height, width) dimensions to the specified size. Can be overridden by the
do_resize parameter in the preprocess method.   size (Dict[str, int] optional, defaults to {"shortest_edge" -- 224}):
Size of the output image after resizing. The shortest edge of


TvltImageProcessor

the image will be resized to
size["shortest_edge"] while maintaining the aspect ratio of the original image. Can be overriden by
size in the preprocess method.   patch_size (List[int] optional, defaults to [16,16]) —
The patch size of image patch embedding.   num_frames (int optional, defaults to 8) —
The maximum number of video frames.   resample (PILImageResampling, optional, defaults to PILImageResampling.BILINEAR) —
Resampling filter to use if resizing the image. Can be overridden by the resample parameter in the
preprocess method.   do_center_crop (bool, optional, defaults to True) —
Whether to center crop the image to the specified crop_size. Can be overridden by the do_center_crop
parameter in the preprocess method.   crop_size (Dict[str, int], optional, defaults to {"height" -- 224, "width": 224}):
Size of the image after applying the center crop. Can be overridden by the crop_size parameter in the
preprocess method.   do_r


TvltImageProcessor

escale (bool, optional, defaults to True) —
Whether to rescale the image by the specified scale rescale_factor. Can be overridden by the do_rescale
parameter in the preprocess method.   rescale_factor (int or float, optional, defaults to 1/255) —
Defines the scale factor to use if rescaling the image. Can be overridden by the rescale_factor parameter
in the preprocess method.   do_normalize (bool, optional, defaults to True) —
Whether to normalize the image. Can be overridden by the do_normalize parameter in the preprocess
method.   image_mean (float or List[float], optional, defaults to IMAGENET_STANDARD_MEAN) —
Mean to use if normalizing the image. This is a float or list of floats the length of the number of
channels in the image. Can be overridden by the image_mean parameter in the preprocess method.   image_std (float or List[float], optional, defaults to IMAGENET_STANDARD_STD) —
Standard deviation to use if normalizing the im


TvltImageProcessor

age. This is a float or list of floats the length of the
number of channels in the image. Can be overridden by the image_std parameter in the preprocess method.    Constructs a TVLT image processor. This processor can be used to prepare either videos or images for the model by converting images to 1-frame videos.  preprocess  < source > ( videos: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]] do_resize: bool = None size: typing.Dict[str, int] = None patch_size: typing.List[int] = None num_frames: int = None resample: Resampling = None do_center_crop: bool = None crop_size: typing.Dict[str, int] = None do_rescale: bool = None rescale_factor: float = None do_normalize: bool = None image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typi


TvltImageProcessor

ng.List[float], NoneType] = None is_mixed: bool = False return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None data_format: ChannelDimension = <ChannelDimension.FIRST: 'channels_first'> input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None **kwargs  ) → BatchFeature  Parameters   videos (ImageInput) —
Images or videos to preprocess. Expects a single or batch of frames with pixel values ranging from 0 to
255. If passing in frames with pixel values between 0 and 1, set do_rescale=False.   do_resize (bool, optional, defaults to self.do_resize) —
Whether to resize the image.   size (Dict[str, int], optional, defaults to self.size) —
Size of the image after applying resize.   patch_size (List[int] optional, defaults to self.patch_size) —
The patch size of image patch embedding.   num_frames (int optional, defaults to self.num_frames) —
The maximum number of vi


TvltImageProcessor

deo frames.   resample (PILImageResampling, optional, defaults to self.resample) —
Resampling filter to use if resizing the image. This can be one of the enum PILImageResampling, Only
has an effect if do_resize is set to True.   do_center_crop (bool, optional, defaults to self.do_centre_crop) —
Whether to centre crop the image.   crop_size (Dict[str, int], optional, defaults to self.crop_size) —
Size of the image after applying the centre crop.   do_rescale (bool, optional, defaults to self.do_rescale) —
Whether to rescale the image values between [0 - 1].   rescale_factor (float, optional, defaults to self.rescale_factor) —
Rescale factor to rescale the image by if do_rescale is set to True.   do_normalize (bool, optional, defaults to self.do_normalize) —
Whether to normalize the image.   image_mean (float or List[float], optional, defaults to self.image_mean) —
Image mean.   image_std (float or List[float], optional, defaults to


TvltImageProcessor

self.image_std) —
Image standard deviation.   is_mixed (bool, optional) —
If the input video has negative samples.   return_tensors (str or TensorType, optional) —
The type of tensors to return. Can be one of:
Unset: Return a list of np.ndarray.
TensorType.TENSORFLOW or 'tf': Return a batch of type tf.Tensor.
TensorType.PYTORCH or 'pt': Return a batch of type torch.Tensor.
TensorType.NUMPY or 'np': Return a batch of type np.ndarray.
TensorType.JAX or 'jax': Return a batch of type jax.numpy.ndarray.
   data_format (ChannelDimension or str, optional, defaults to ChannelDimension.FIRST) —
The channel dimension format for the output image. Can be one of:
ChannelDimension.FIRST: image in (num_channels, height, width) format.
ChannelDimension.LAST: image in (height, width, num_channels) format.
Unset: Use the inferred channel dimension format of the input image.
   input_data_format (ChannelDimension or str, optional) —
The channel dimen


TvltImageProcessor

sion format for the input image. If unset, the channel dimension format is inferred
from the input image. Can be one of:
"channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
"channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
"none" or ChannelDimension.NONE: image in (height, width) format.
   Returns
BatchFeature

A BatchFeature with the following fields:
pixel_values — Pixel values to be fed to a model, of shape (batch_size, num_channels, height,
width).
pixel_mask — Pixel masks to be fed to a model, of shape (batch_size, num_pixel_patches).
pixel_values_mixed — Pixel values with both postive or negative to be fed to a model, of shape
(batch_size, num_channels, height, width).
pixel_mask_mixed — Pixel masks with both postive or negative to be fed to a model, of shape
(batch_size, num_pixel_patches).
  Preprocess an videos or image or batch of videos or


TvltFeatureExtractor

class transformers.TvltFeatureExtractor  < source > ( spectrogram_length = 2048 num_channels = 1 patch_size = [16, 16] feature_size = 128 sampling_rate = 44100 hop_length_to_sampling_rate = 86 n_fft = 2048 padding_value = 0.0 **kwargs  )   Parameters   spectrogram_length (Dict[str, int] optional, defaults to 2048) —
The time length of each audio spectrogram.   num_channels (int optional, defaults to 1) —
Number of audio channels.   patch_size (List[int] optional, defaults to [16, 16]) —
The patch size of audio patch embedding.   feature_size (int, optional, defaults to 128) —
The frequency length of audio spectrogram.   sampling_rate (int, optional, defaults to 44100) —
The sampling rate at which the audio files should be digitalized expressed in Hertz (Hz).   hop_length_to_sampling_rate (int, optional, defaults to 86) —
Hop length is length of the overlaping windows for the STFT used to obtain the M


TvltFeatureExtractor

el Frequency coefficients.
For example, with sampling rate 44100, the hop length is 512, with 44100 / 512 = 86   n_fft (int, optional, defaults to 2048) —
Size of the Fourier transform.   padding_value (float, optional, defaults to 0.0) —
Padding value used to pad the audio. Should correspond to silences.    Constructs a TVLT audio feature extractor. This feature extractor can be used to prepare audios for the model. This feature extractor inherits from FeatureExtractionMixin which contains most of the main methods. Users
should refer to this superclass for more information regarding those methods.  __call__  < source > ( raw_speech: typing.Union[numpy.ndarray, typing.List[float], typing.List[numpy.ndarray], typing.List[typing.List[float]]] return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None return_attention_mask: typing.Optional[bool] = True sampling_rate: typin


TvltFeatureExtractor

g.Optional[int] = None resample: bool = False mask_audio: bool = False **kwargs  ) → BatchFeature  Parameters   raw_speech (np.ndarray, List[float], List[np.ndarray], List[List[float]]) —
The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float
values, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not
stereo, i.e. single float per timestep.   return_tensors (str or TensorType, optional) —
If set, will return tensors instead of list of python integers. Acceptable values are:
'pt': Return PyTorch torch.Tensor objects.
'np': Return Numpy np.ndarray objects.
   return_attention_mask (bool, optional, default to True) —
Whether to return the attention mask. If left to the default, will return the attention mask according
to the specific feature_extractor’s default. What are attention masks?
For TvltTransformer models, atten


TvltFeatureExtractor

tion_mask should alwys be passed for batched inference, to avoid
subtle bugs.
   sampling_rate (int, optional) —
The sampling rate at which the raw_speech input was sampled. It is strongly recommended to pass
sampling_rate at the forward call to prevent silent errors and allow automatic speech recognition
pipeline. Current model supports sampling rate 16000 and 44100.   resample (bool, optional, defaults to False) —
If the sampling rate is not matched, resample the input audio to match.   mask_audio (bool, optional, defaults to False) —
Whether or not to mask input audio for MAE task.   Returns
BatchFeature

A BatchFeature with the following fields:
audio_values — Audio values to be fed to a model, of shape (batch_size, num_channels, height,
width).
audio_mask — Audio masks to be fed to a model, of shape (batch_size, num_audio_patches).
  Main method to prepare one or several audio(s) for the mode


TvltModel

class transformers.TvltModel  < source > ( config  )   Parameters   config (TvltConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare TVLT Model transformer outputting raw hidden-states without any specific head on top.
This model is a PyTorch torch.nn.Module subclass. Use it
as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: FloatTensor audio_values: FloatTensor pixel_mask: typing.Optional[torch.FloatTensor] = None audio_mask: typing.Optional[torch.FloatTensor] = None mask_pixel: bool = False mask_audio: bool = False output_attentions: t


TvltModel

yping.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.tvlt.modeling_tvlt.TvltModelOutput or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_frames, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using TvltProcessor. See TvltProcessor.call() for
details.   audio_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Audio values. Audio values can be obtained using TvltProcessor. See TvltProcessor.call() for
details.   pixel_mask (torch.FloatTensor of shape (batch_size, num_pixel_patches)) —
Pixel masks. Pixel masks can be obtained using TvltProcessor. See TvltProcessor.call() for
details.   audio_mask (torch.FloatTensor of shape (batch_size, num_audio_patch


TvltModel

es)) —
Audio masks. Audio masks can be obtained using TvltProcessor. See TvltProcessor.call() for
details.   pixel_values_mixed (torch.FloatTensor of shape (batch_size, num_frames, num_channels, height, width)) —
Pixel values that mix positive and negative samples in Tvlt vision-audio matching. Pixel values mixed can
be obtained using TvltProcessor. See TvltProcessor.call() for details.   pixel_mask_mixed (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel masks of pixel_values_mixed. Pixel masks mixed can be obtained using TvltProcessor. See
TvltProcessor.call() for details.   mask_pixel (bool, optional) —
Whether to mask pixel for MAE tasks. Only set to True in TvltForPreTraining.   mask_audio (bool, optional) —
Whether to mask audio for MAE tasks. Only set to True in TvltForPreTraining.   output_attentions (bool, opt


TvltModel

ional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.models.tvlt.modeling_tvlt.TvltModelOutput or tuple(torch.FloatTensor)

A transformers.models.tvlt.modeling_tvlt.TvltModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (TvltConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the outp


TvltModel

ut of the last layer of the model.
last_pixel_hidden_state (torch.FloatTensor of shape (batch_size, pixel_sequence_length, hidden_size)) — Pixel sequence of hidden-states at the output of the last layer of the model.
last_audio_hidden_state (torch.FloatTensor of shape (batch_size, audio_sequence_length, hidden_size)) — Audio sequence of hidden-states at the output of the last layer of the model.
pixel_label_masks (torch.FloatTensor of shape (batch_size, pixel_patch_length)) — Tensor indicating which pixel patches are masked (1) and which are not (0).
audio_label_masks (torch.FloatTensor of shape (batch_size, audio_patch_length)) — Tensor indicating which audio patches are masked (1) and which are not (0).
pixel_ids_restore (torch.LongTensor of shape (batch_size, pixel_patch_length)) — Tensor containing the ids permutation of pixel masking.
audio_


TvltModel

ids_restore (torch.LongTensor of shape (batch_size, audio_patch_length)) — Tensor containing the ids permutation of audio masking.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings and one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size). Hidden-states of the model at the output of each layer
plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Attentions weights after the attention softmax, used to compute the weighted averag


TvltModel

e in
the self-attention heads.
  The TvltModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import TvltProcessor, TvltModel
>>> import numpy as np
>>> import torch
>>> num_frames = 8
>>> images = list(np.random.randn(num_frames, 3, 224, 224))
>>> audio = list(np.random.randn(10000))
>>> processor = TvltProcessor.from_pretrained("ZinengTang/tvlt-base")
>>> model = TvltModel.from_pretrained("ZinengTang/tvlt-base")
>>> input_dict = processor(images, audio, sampling_rate=44100, return_tensors="pt")
>>> outputs = model(**input_dict)
>>> loss = output


TvltForPreTraining

class transformers.TvltForPreTraining  < source > ( config  )   Parameters   config (TvltConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The TVLT Model transformer with the decoder on top for self-supervised pre-training.
This model is a PyTorch torch.nn.Module subclass. Use it
as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: FloatTensor audio_values: FloatTensor pixel_mask: typing.Optional[torch.FloatTensor] = None audio_mask: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.LongTensor] = None pixel_values_mixed: typing.Optional[torch.FloatTensor] = None pixel_


TvltForPreTraining

mask_mixed: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.tvlt.modeling_tvlt.TvltForPreTrainingOutput or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_frames, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using TvltProcessor. See TvltProcessor.call() for
details.   audio_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Audio values. Audio values can be obtained using TvltProcessor. See TvltProcessor.call() for
details.   pixel_mask (torch.FloatTensor of shape (batch_size, num_pixel_patches)) —
Pixel masks. Pixel masks can be obtained using TvltProcessor. See TvltProcessor.call() for
details.   audio_mask (torch.FloatTensor


TvltForPreTraining

of shape (batch_size, num_audio_patches)) —
Audio masks. Audio masks can be obtained using TvltProcessor. See TvltProcessor.call() for
details.   pixel_values_mixed (torch.FloatTensor of shape (batch_size, num_frames, num_channels, height, width)) —
Pixel values that mix positive and negative samples in Tvlt vision-audio matching. Pixel values mixed can
be obtained using TvltProcessor. See TvltProcessor.call() for details.   pixel_mask_mixed (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel masks of pixel_values_mixed. Pixel masks mixed can be obtained using TvltProcessor. See
TvltProcessor.call() for details.   mask_pixel (bool, optional) —
Whether to mask pixel for MAE tasks. Only set to True in TvltForPreTraining.   mask_audio (bool, optional) —
Whether to mask audio for MAE tasks. Only set to True in TvltForPreTraining.   output_attentions (bool, optional) —


TvltForPreTraining

Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   pixel_values_mixed (torch.FloatTensor of shape (batch_size, num_frames, num_channels, height, width)) —
Pixel values that mix positive and negative samples in Tvlt vision-audio matching. Audio values can be
obtained using TvltProcessor. See TvltProcessor.call() for details.   pixel_mask_mixed (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel masks of pixel_values_mixed. Pixel values mixed can be obtained using TvltProcessor. See
TvltProcessor.call() for details.   labels (torch.LongTe


TvltForPreTraining

nsor of shape (batch_size, num_labels), optional) —
Labels for computing the vision audio matching loss. Indices should be in [0, 1]. num_labels has to be 1.   Returns
transformers.models.tvlt.modeling_tvlt.TvltForPreTrainingOutput or tuple(torch.FloatTensor)

A transformers.models.tvlt.modeling_tvlt.TvltForPreTrainingOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (TvltConfig) and inputs.
loss (torch.FloatTensor of shape (1,)) — Pixel reconstruction loss.
matching_logits (torch.FloatTensor of shape (batch_size, 1)) — Matching objective logits.
pixel_logits (torch.FloatTensor of shape
(batch_size, pixel_patch_length, image_patch_size ** 3 * pixel_num_channels)): Pixel reconstruction
logits.
audio_logits (torch.FloatTensor of shape
(batch_size, audio_patch_length, image_patch


TvltForPreTraining

_size[0] * image_patch_size[1])): Audio reconstruction
logits.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings and one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size). Hidden-states of the model at the output of each layer
plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Attentions weights after the attention softmax, used to compute the weighted average in
the self-attention heads.
  The TvltForPreTraining forward method, overrides the __call__ special method. Alt


TvltForPreTraining

hough the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import TvltProcessor, TvltForPreTraining
>>> import numpy as np
>>> import torch
>>> num_frames = 8
>>> images = list(np.random.randn(num_frames, 3, 224, 224))
>>> images_mixed = list(np.random.randn(num_frames, 3, 224, 224))
>>> audio = list(np.random.randn(10000))
>>> processor = TvltProcessor.from_pretrained("ZinengTang/tvlt-base")
>>> model = TvltForPreTraining.from_pretrained("ZinengTang/tvlt-base")
>>> input_dict = processor(
...     images, audio, images_mixed, sampling_rate=44100, mask_pixel=True, mask_audio=True, return_tensors="pt"
... )
>>> outputs = model(**input_dict)
>>> loss = outputs.los


TvltForAudioVisualClassification

class transformers.TvltForAudioVisualClassification  < source > ( config  )   Parameters   config (TvltConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Tvlt Model transformer with a classifier head on top (an MLP on top of the final hidden state of the [CLS] token)
for audiovisual classification tasks, e.g. CMU-MOSEI Sentiment Analysis and Audio to Video Retrieval. This model is a PyTorch torch.nn.Module subclass. Use it
as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: FloatTensor audio_values: FloatTensor pixel_mask: typing.Optional[torch.FloatTensor] = None audio_mask: typing.Optional[torch.FloatTensor] = None output_attenti


TvltForAudioVisualClassification

ons: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None labels: typing.Optional[torch.LongTensor] = None  ) → transformers.modeling_outputs.SequenceClassifierOutput or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_frames, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using TvltProcessor. See TvltProcessor.call() for
details.   audio_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Audio values. Audio values can be obtained using TvltProcessor. See TvltProcessor.call() for
details.   pixel_mask (torch.FloatTensor of shape (batch_size, num_pixel_patches)) —
Pixel masks. Pixel masks can be obtained using TvltProcessor. See TvltProcessor.call() for
details.   audio_mask (torch.FloatTensor of shape (batch_size, num_audio_patches)) —
Audio masks. Audio masks c


TvltForAudioVisualClassification

an be obtained using TvltProcessor. See TvltProcessor.call() for
details.   pixel_values_mixed (torch.FloatTensor of shape (batch_size, num_frames, num_channels, height, width)) —
Pixel values that mix positive and negative samples in Tvlt vision-audio matching. Pixel values mixed can
be obtained using TvltProcessor. See TvltProcessor.call() for details.   pixel_mask_mixed (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel masks of pixel_values_mixed. Pixel masks mixed can be obtained using TvltProcessor. See
TvltProcessor.call() for details.   mask_pixel (bool, optional) —
Whether to mask pixel for MAE tasks. Only set to True in TvltForPreTraining.   mask_audio (bool, optional) —
Whether to mask audio for MAE tasks. Only set to True in TvltForPreTraining.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors f


TvltForAudioVisualClassification

or more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size, num_labels), optional) —
Labels for computing the audiovisual loss. Indices should be in [0, ..., num_classes-1] where num_classes
refers to the number of classes in audiovisual tasks.   Returns
transformers.modeling_outputs.SequenceClassifierOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.SequenceClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (TvltConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or


TvltForAudioVisualClassification

regression if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute t


TvltForAudioVisualClassification

he weighted average in the self-attention
heads.
  The TvltForAudioVisualClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import TvltProcessor, TvltForAudioVisualClassification
>>> import numpy as np
>>> import torch
>>> num_frames = 8
>>> images = list(np.random.randn(num_frames, 3, 224, 224))
>>> audio = list(np.random.randn(10000))
>>> processor = TvltProcessor.from_pretrained("ZinengTang/tvlt-base")
>>> model = TvltForAudioVisualClassification.from_pretrained("ZinengTang/tvlt-base")
>>> input_dict = processor(images, audio, sampling_rate=44100, return_tensors="pt")
>>> outputs = model(**input_dict)
>>> loss = outputs


Overview

The BARTpho model was proposed in BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese by Nguyen Luong Tran, Duong Minh Le and Dat Quoc Nguyen.
The abstract from the paper is the following:
We present BARTpho with two versions — BARTpho_word and BARTpho_syllable — the first public large-scale monolingual
sequence-to-sequence models pre-trained for Vietnamese. Our BARTpho uses the “large” architecture and pre-training
scheme of the sequence-to-sequence denoising model BART, thus especially suitable for generative NLP tasks. Experiments
on a downstream task of Vietnamese text summarization show that in both automatic and human evaluations, our BARTpho
outperforms the strong baseline mBART and improves the state-of-the-art. We release BARTpho to facilitate future
research and applications of generative Vietnamese NLP tasks.
Example of use:
  Copied >>> import torch
>>> from transformers import AutoModel, AutoTokenizer
>>> bartpho = AutoModel.fr


Overview

om_pretrained("vinai/bartpho-syllable")
>>> tokenizer = AutoTokenizer.from_pretrained("vinai/bartpho-syllable")
>>> line = "Chúng tôi là những nghiên cứu viên."
>>> input_ids = tokenizer(line, return_tensors="pt")
>>> with torch.no_grad():
...     features = bartpho(**input_ids)  # Models outputs are now tuples
>>> # With TensorFlow 2.0+:
>>> from transformers import TFAutoModel
>>> bartpho = TFAutoModel.from_pretrained("vinai/bartpho-syllable")
>>> input_ids = tokenizer(line, return_tensors="tf")
>>> features = bartpho(**input_ids)
Tips:
Following mBART, BARTpho uses the “large” architecture of BART with an additional layer-normalization layer on top of
both the encoder and decoder. Thus, usage examples in the documentation of BART, when adapting to use
with BARTpho, should be adjusted by replacing the BART-specialized classes with the mBART-specialized counterparts.
For example:
  Copied >>> from transformers import MBartForConditionalGeneration
>>>


Overview

bartpho = MBartForConditionalGeneration.from_pretrained("vinai/bartpho-syllable")
>>> TXT = "Chúng tôi là <mask> nghiên cứu viên."
>>> input_ids = tokenizer([TXT], return_tensors="pt")["input_ids"]
>>> logits = bartpho(input_ids).logits
>>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item()
>>> probs = logits[0, masked_index].softmax(dim=0)
>>> values, predictions = probs.topk(5)
>>> print(tokenizer.decode(predictions).split())
This implementation is only for tokenization: “monolingual_vocab_file” consists of Vietnamese-specialized types
extracted from the pre-trained SentencePiece model “vocab_file” that is available from the multilingual XLM-RoBERTa.
Other languages, if employing this pre-trained multilingual SentencePiece model “vocab_file” for subword
segmentation, can reuse BartphoTokenizer with their own language-specialized “monolingual_vocab_file”.
This model was contributed by dqnguyen. The original code can be found her


BartphoTokenizer

class transformers.BartphoTokenizer  < source > ( vocab_file monolingual_vocab_file bos_token = '<s>' eos_token = '</s>' sep_token = '</s>' cls_token = '<s>' unk_token = '<unk>' pad_token = '<pad>' mask_token = '<mask>' sp_model_kwargs: typing.Union[typing.Dict[str, typing.Any], NoneType] = None **kwargs  )   Parameters   vocab_file (str) —
Path to the vocabulary file. This vocabulary is the pre-trained SentencePiece model available from the
multilingual XLM-RoBERTa, also used in mBART, consisting of 250K types.   monolingual_vocab_file (str) —
Path to the monolingual vocabulary file. This monolingual vocabulary consists of Vietnamese-specialized
types extracted from the multilingual vocabulary vocab_file of 250K types.   bos_token (str, optional, defaults to "<s>") —
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a seq


BartphoTokenizer

uence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the cls_token.
   eos_token (str, optional, defaults to "</s>") —
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the sep_token.
   sep_token (str, optional, defaults to "</s>") —
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.   cls_token (str, optional, defaults to "<s>") —
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with


BartphoTokenizer

special tokens.   unk_token (str, optional, defaults to "<unk>") —
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.   pad_token (str, optional, defaults to "<pad>") —
The token used for padding, for example when batching sequences of different lengths.   mask_token (str, optional, defaults to "<mask>") —
The token used for masking values. This is the token used when training this model with masked language
modeling. This is the token which the model will try to predict.   sp_model_kwargs (dict, optional) —
Will be passed to the SentencePieceProcessor.__init__() method. The Python wrapper for
SentencePiece can be used, among other things,
to set:
enable_sampling: Enable subword regularization.
nbest_size: Sampling parameters for unigram. Invalid for BPE-Dropout.
nbest_size = {0,1}: No sampling is performed.
nbest_size >


BartphoTokenizer

1: samples from the nbest_size results.
nbest_size < 0: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
using forward-filtering-and-backward-sampling algorithm.
alpha: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
BPE-dropout.
   sp_model (SentencePieceProcessor) —
The SentencePiece processor that is used for every conversion (string, tokens and IDs).    Adapted from XLMRobertaTokenizer. Based on SentencePiece. This tokenizer inherits from PreTrainedTokenizer which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.  build_inputs_with_special_tokens  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None  ) → List[int]  Parameters   token_ids_0 (List[int]) —
List of IDs to which the special tokens will be ad


BartphoTokenizer

ded.   token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.   Returns
List[int]

List of input IDs with the appropriate special tokens.
  Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. An BARTPho sequence has the following format: single sequence: <s> X </s> pair of sequences: <s> A </s></s> B </s>  convert_tokens_to_string  < source > ( tokens  )    Converts a sequence of tokens (strings for sub-words) in a single string.  create_token_type_ids_from_sequences  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None  ) → List[int]  Parameters   token_ids_0 (List[int]) —
List of IDs.   token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.   Returns
List[int]

List of zeros.
  Create a mask from the tw


BartphoTokenizer

o sequences passed to be used in a sequence-pair classification task. BARTPho does not
make use of token type ids, therefore a list of zeros is returned.  get_special_tokens_mask  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None already_has_special_tokens: bool = False  ) → List[int]  Parameters   token_ids_0 (List[int]) —
List of IDs.   token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.   already_has_special_tokens (bool, optional, defaults to False) —
Whether or not the token list is already formatted with special tokens for the model.   Returns
List[int]

A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
  Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer prepare_for_model method


GLPN

This is a recently introduced model so the API hasn’t been tested extensively. There may be some bugs or slight
breaking changes to fix it in the future. If you see something strange, file a Github Issue.


Overview

The GLPN model was proposed in Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepth  by Doyeon Kim, Woonghyun Ga, Pyungwhan Ahn, Donggyu Joo, Sehwan Chun, Junmo Kim.
GLPN combines SegFormer’s hierarchical mix-Transformer with a lightweight decoder for monocular depth estimation. The proposed decoder shows better performance than the previously proposed decoders, with considerably
less computational complexity.
The abstract from the paper is the following:
Depth estimation from a single image is an important task that can be applied to various fields in computer vision, and has grown rapidly with the development of convolutional neural networks. In thi


Overview

s paper, we propose a novel structure and training strategy for monocular depth estimation to further improve the prediction accuracy of the network. We deploy a hierarchical transformer encoder to capture and convey the global context, and design a lightweight yet powerful decoder to generate an estimated depth map while considering local connectivity. By constructing connected paths between multi-scale local features and the global decoding stream with our proposed selective feature fusion module, the network can integrate both representations and recover fine details. In addition, the proposed decoder shows better performance than the previously proposed decoders, with consid


Overview

erably less computational complexity. Furthermore, we improve the depth-specific augmentation method by utilizing an important observation in depth estimation to enhance the model. Our network achieves state-of-the-art performance over the challenging depth dataset NYU Depth V2. Extensive experiments have been conducted to validate and show the effectiveness of the proposed approach. Finally, our model shows better generalisation ability and robustness than other comparative models.
Tips:
One can use GLPNImageProcessor to prepare images for the model.
Summary of the approach. Taken from the original paper.
This model was contributed by nielsr. The original code can be found here


Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with GLPN.
Demo notebooks for GLPNForDepthEstimation can be found here. Monocular depth estimation task guide


GLPNConfig

class transformers.GLPNConfig  < source > ( num_channels = 3 num_encoder_blocks = 4 depths = [2, 2, 2, 2] sr_ratios = [8, 4, 2, 1] hidden_sizes = [32, 64, 160, 256] patch_sizes = [7, 3, 3, 3] strides = [4, 2, 2, 2] num_attention_heads = [1, 2, 5, 8] mlp_ratios = [4, 4, 4, 4] hidden_act = 'gelu' hidden_dropout_prob = 0.0 attention_probs_dropout_prob = 0.0 initializer_range = 0.02 drop_path_rate = 0.1 layer_norm_eps = 1e-06 decoder_hidden_size = 64 max_depth = 10 head_in_index = -1 **kwargs  )   Parameters   num_channels (int, optional, defaults to 3) —
The number of input channels.   num_encoder_blocks (int, optional, defaults to 4) —
The number of encoder blocks (i.e. stages in the Mix Transformer encoder).   depths (List[int], optional, defaults to [2, 2, 2, 2]) —
The number of layers in each encoder block.   sr_ratios (List[int], optional, defaults to [8, 4, 2, 1]) —
Sequence reduc


GLPNConfig

tion ratios in each encoder block.   hidden_sizes (List[int], optional, defaults to [32, 64, 160, 256]) —
Dimension of each of the encoder blocks.   patch_sizes (List[int], optional, defaults to [7, 3, 3, 3]) —
Patch size before each encoder block.   strides (List[int], optional, defaults to [4, 2, 2, 2]) —
Stride before each encoder block.   num_attention_heads (List[int], optional, defaults to [1, 2, 5, 8]) —
Number of attention heads for each attention layer in each block of the Transformer encoder.   mlp_ratios (List[int], optional, defaults to [4, 4, 4, 4]) —
Ratio of the size of the hidden layer compared to the size of the input layer of the Mix FFNs in the
encoder blocks.   hidden_act (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "selu" and "gelu_new" are supported


GLPNConfig

.   hidden_dropout_prob (float, optional, defaults to 0.0) —
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.   attention_probs_dropout_prob (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   drop_path_rate (float, optional, defaults to 0.1) —
The dropout probability for stochastic depth, used in the blocks of the Transformer encoder.   layer_norm_eps (float, optional, defaults to 1e-06) —
The epsilon used by the layer normalization layers.   decoder_hidden_size (int, optional, defaults to 64) —
The dimension of the decoder.   max_depth (int, optional, defaults to 10) —
The maximum depth of the decoder.   head_in_index (int, optional, defaults to -


GLPNConfig

1) —
The index of the features to use in the head.    This is the configuration class to store the configuration of a GLPNModel. It is used to instantiate an GLPN
model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
defaults will yield a similar configuration to that of the GLPN
vinvino02/glpn-kitti architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import GLPNModel, GLPNConfig
>>> # Initializing a GLPN vinvino02/glpn-kitti style configuration
>>> configuration = GLPNConfig()
>>> # Initializing a model from the vinvino02/glpn-kitti style configuration
>>> model = GLPNModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.con


GLPNFeatureExtractor

class transformers.GLPNFeatureExtractor  < source > ( *args **kwargs  )     __call__  < source > ( images **kwargs  )    Preprocess an image or a batch of images.


GLPNImageProcessor

class transformers.GLPNImageProcessor  < source > ( do_resize: bool = True size_divisor: int = 32 resample = <Resampling.BILINEAR: 2> do_rescale: bool = True **kwargs  )   Parameters   do_resize (bool, optional, defaults to True) —
Whether to resize the image’s (height, width) dimensions, rounding them down to the closest multiple of
size_divisor. Can be overridden by do_resize in preprocess.   size_divisor (int, optional, defaults to 32) —
When do_resize is True, images are resized so their height and width are rounded down to the closest
multiple of size_divisor. Can be overridden by size_divisor in preprocess.   resample (PIL.Image resampling filter, optional, defaults to Resampling.BILINEAR) —
Resampling filter to use if resizing the image. Can be overridden by resample in preprocess.   do_rescale (bool, optional, defaults to True) —
Whether or not to apply the scaling factor (to make pixel values floats between 0. and 1.). Can be
ove


GLPNImageProcessor

rridden by do_rescale in preprocess.    Constructs a GLPN image processor.  preprocess  < source > ( images: typing.Union[ForwardRef('PIL.Image.Image'), transformers.utils.generic.TensorType, typing.List[ForwardRef('PIL.Image.Image')], typing.List[transformers.utils.generic.TensorType]] do_resize: typing.Optional[bool] = None size_divisor: typing.Optional[int] = None resample = None do_rescale: typing.Optional[bool] = None return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None data_format: ChannelDimension = <ChannelDimension.FIRST: 'channels_first'> input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None **kwargs  )   Parameters   images (PIL.Image.Image or TensorType or List[np.ndarray] or List[TensorType]) —
Images to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If
passing in images with pixel values between 0 and 1,


GLPNImageProcessor

set do_normalize=False.   do_resize (bool, optional, defaults to self.do_resize) —
Whether to resize the input such that the (height, width) dimensions are a multiple of size_divisor.   size_divisor (int, optional, defaults to self.size_divisor) —
When do_resize is True, images are resized so their height and width are rounded down to the
closest multiple of size_divisor.   resample (PIL.Image resampling filter, optional, defaults to self.resample) —
PIL.Image resampling filter to use if resizing the image e.g. PILImageResampling.BILINEAR. Only has
an effect if do_resize is set to True.   do_rescale (bool, optional, defaults to self.do_rescale) —
Whether or not to apply the scaling factor (to make pixel values floats between 0. and 1.).   return_tensors (str or TensorType, optional) —
The type of tensors to return. Can be one of:
None: Return a list of np.ndarray.
TensorType.TENSORFLOW or 'tf': Return a batch of type tf.Tensor.
TensorType


GLPNImageProcessor

.PYTORCH or 'pt': Return a batch of type torch.Tensor.
TensorType.NUMPY or 'np': Return a batch of type np.ndarray.
TensorType.JAX or 'jax': Return a batch of type jax.numpy.ndarray.
   data_format (ChannelDimension or str, optional, defaults to ChannelDimension.FIRST) —
The channel dimension format for the output image. Can be one of:
ChannelDimension.FIRST: image in (num_channels, height, width) format.
ChannelDimension.LAST: image in (height, width, num_channels) format.
   input_data_format (ChannelDimension or str, optional) —
The channel dimension format for the input image. If unset, the channel dimension format is inferred
from the input image. Can be one of:
"channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
"channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
"none" or ChannelDimension.NONE: image in (height, width) format.
    Preprocess the given image


GLPNModel

class transformers.GLPNModel  < source > ( config  )   Parameters   config (GLPNConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare GLPN encoder (Mix-Transformer) outputting raw hidden-states without any specific head on top.
This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: FloatTensor output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.BaseModelOutput or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batc


GLPNModel

h_size, num_channels, height, width)) —
Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using
AutoImageProcessor. See GLPNImageProcessor.call() for details.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.BaseModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (GLPNConfig) and


GLPNModel

inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted avera


GLPNModel

ge in the self-attention
heads.
  The GLPNModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoImageProcessor, GLPNModel
>>> import torch
>>> from datasets import load_dataset
>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]
>>> image_processor = AutoImageProcessor.from_pretrained("vinvino02/glpn-kitti")
>>> model = GLPNModel.from_pretrained("vinvino02/glpn-kitti")
>>> inputs = image_processor(image, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1, 512, 15,


GLPNForDepthEstimation

class transformers.GLPNForDepthEstimation  < source > ( config  )   Parameters   config (GLPNConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    GLPN Model transformer with a lightweight depth estimation head on top e.g. for KITTI, NYUv2.
This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: FloatTensor labels: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.DepthEstimatorOut


GLPNForDepthEstimation

put or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using
AutoImageProcessor. See GLPNImageProcessor.call() for details.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.FloatTensor of shape (batch_size, height, width), optional) —
Ground truth depth estimation maps for computing the loss.   Returns
transformers.modeling_outputs.DepthEs


GLPNForDepthEstimation

timatorOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.DepthEstimatorOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (GLPNConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
predicted_depth (torch.FloatTensor of shape (batch_size, height, width)) — Predicted depth for each pixel.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, num_channels, height, width).
Hidden-states of the model at the output


GLPNForDepthEstimation

of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, patch_size, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The GLPNForDepthEstimation forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoImageProcessor, GLPNForDepthEstimation
>>> import torch
>>> import numpy as np
>>> from PIL import


GLPNForDepthEstimation

 Image
>>> import requests
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> image_processor = AutoImageProcessor.from_pretrained("vinvino02/glpn-kitti")
>>> model = GLPNForDepthEstimation.from_pretrained("vinvino02/glpn-kitti")
>>> # prepare image for the model
>>> inputs = image_processor(images=image, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
...     predicted_depth = outputs.predicted_depth
>>> # interpolate to original size
>>> prediction = torch.nn.functional.interpolate(
...     predicted_depth.unsqueeze(1),
...     size=image.size[::-1],
...     mode="bicubic",
...     align_corners=False,
... )
>>> # visualize the prediction
>>> output = prediction.squeeze().cpu().numpy()
>>> formatted = (output * 255 / np.max(output)).astype("uint8")
>>> depth = Image.fromarray(formatted


Overview

This page provides code and pre-trained weights for Transformer protein language models from Meta AI’s Fundamental
AI Research Team, providing the state-of-the-art ESMFold and ESM-2, and the previously released ESM-1b and ESM-1v.
Transformer protein language models were introduced in the paper Biological structure and function emerge from scaling
unsupervised learning to 250 million protein sequences by
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott,
C. Lawrence Zitnick, Jerry Ma, and Rob Fergus.
The first version of this paper was preprinted in 2019.
ESM-2 outperforms all tested single-sequence protein language models across a range of structure prediction tasks,
and enables atomic resolution structure prediction.
It was released with the paper Language models of protein sequences at the scale of evolution enable accurate
structure prediction by Zeming Lin, Halil Akin, Roshan Rao, Brian Hie,
Zhongkai Zhu, Wenting


Overview

 Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido and Alexander Rives.
Also introduced in this paper was ESMFold. It uses an ESM-2 stem with a head that can predict folded protein
structures with state-of-the-art accuracy. Unlike AlphaFold2,
it relies on the token embeddings from the large pre-trained protein language model stem and does not perform a multiple
sequence alignment (MSA) step at inference time, which means that ESMFold checkpoints are fully “standalone” -
they do not require a database of known protein sequences and structures with associated external query tools
to make predictions, and are much faster as a result.
The abstract from
“Biological structure and function emerge from scaling unsupervised learning to 250
million protein sequences” is
In the field of artificial intelligence, a combination of scale in data and model capacity enabled by unsupervised
learning has led to major advances in representation learning and statis


Overview

tical generation. In the life sciences, the
anticipated growth of sequencing promises unprecedented data on natural sequence diversity. Protein language modeling
at the scale of evolution is a logical step toward predictive and generative artificial intelligence for biology. To
this end, we use unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250
million protein sequences spanning evolutionary diversity. The resulting model contains information about biological
properties in its representations. The representations are learned from sequence data alone. The learned representation
space has a multiscale organization reflecting structure from the level of biochemical properties of amino acids to
remote homology of proteins. Information about secondary and tertiary structure is encoded in the representations and
can be identified by linear projections. Representation learning produces features that generalize across a range


Overview

 of
applications, enabling state-of-the-art supervised prediction of mutational effect and secondary structure and
improving state-of-the-art features for long-range contact prediction.
The abstract from
“Language models of protein sequences at the scale of evolution enable accurate structure prediction” is
Large language models have recently been shown to develop emergent capabilities with scale, going beyond
simple pattern matching to perform higher level reasoning and generate lifelike images and text. While
language models trained on protein sequences have been studied at a smaller scale, little is known about
what they learn about biology as they are scaled up. In this work we train models up to 15 billion parameters,
the largest language models of proteins to be evaluated to date. We find that as models are scaled they learn
information enabling the prediction of the three-dimensional structure of a protein at the resolution of
individual atoms. We present ESMF


Overview

old for high accuracy end-to-end atomic level structure prediction directly
from the individual sequence of a protein. ESMFold has similar accuracy to AlphaFold2 and RoseTTAFold for
sequences with low perplexity that are well understood by the language model. ESMFold inference is an
order of magnitude faster than AlphaFold2, enabling exploration of the structural space of metagenomic
proteins in practical timescales.
Tips:
ESM models are trained with a masked language modeling (MLM) objective.
The original code can be found here and was
was developed by the Fundamental AI Research team at Meta AI.
ESM-1b, ESM-1v and ESM-2 were contributed to huggingface by jasonliu
and Matt.
ESMFold was contributed to huggingface by Matt and
Sylvain, with a big thank you to Nikita Smetanin, Roshan Rao and Tom Sercu for their
help throughout the process!
The HuggingFace port of ESMFold uses portions of the openfold library.
The openfold library is licensed under the Apache License 2.0


EsmConfig

class transformers.EsmConfig  < source > ( vocab_size = None mask_token_id = None pad_token_id = None hidden_size = 768 num_hidden_layers = 12 num_attention_heads = 12 intermediate_size = 3072 hidden_dropout_prob = 0.1 attention_probs_dropout_prob = 0.1 max_position_embeddings = 1026 initializer_range = 0.02 layer_norm_eps = 1e-12 position_embedding_type = 'absolute' use_cache = True emb_layer_norm_before = None token_dropout = False is_folding_model = False esmfold_config = None vocab_list = None **kwargs  )   Parameters   vocab_size (int, optional) —
Vocabulary size of the ESM model. Defines the number of different tokens that can be represented by the
inputs_ids passed when calling ESMModel.   mask_token_id (int, optional) —
The index of the mask token in the vocabulary. This must be included in the config because of the
“mask-dropout” scaling trick, which will scale the inputs depending


EsmConfig

 on the number of masked tokens.   pad_token_id (int, optional) —
The index of the padding token in the vocabulary. This must be included in the config because certain parts
of the ESM code use this instead of the attention mask.   hidden_size (int, optional, defaults to 768) —
Dimensionality of the encoder layers and the pooler layer.   num_hidden_layers (int, optional, defaults to 12) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 12) —
Number of attention heads for each attention layer in the Transformer encoder.   intermediate_size (int, optional, defaults to 3072) —
Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.   hidden_dropout_prob (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.   attention_prob


EsmConfig

s_dropout_prob (float, optional, defaults to 0.1) —
The dropout ratio for the attention probabilities.   max_position_embeddings (int, optional, defaults to 1026) —
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   layer_norm_eps (float, optional, defaults to 1e-12) —
The epsilon used by the layer normalization layers.   position_embedding_type (str, optional, defaults to "absolute") —
Type of position embedding. Choose one of "absolute", "relative_key", "relative_key_query", "rotary".
For positional embeddings use "absolute". For more information on "relative_key", please refer to
Self-Attention with Relative Position Representations (Sha


EsmConfig

w et al.).
For more information on "relative_key_query", please refer to Method 4 in Improve Transformer Models
with Better Relative Position Embeddings (Huang et al.).   is_decoder (bool, optional, defaults to False) —
Whether the model is used as a decoder or not. If False, the model is used as an encoder.   use_cache (bool, optional, defaults to True) —
Whether or not the model should return the last key/values attentions (not used by all models). Only
relevant if config.is_decoder=True.   emb_layer_norm_before (bool, optional) —
Whether to apply layer normalization after embeddings but before the main stem of the network.   token_dropout (bool, defaults to False) —
When this is enabled, masked tokens are treated as if they had been dropped out by input dropout.    This is the configuration class to store the configuration of a ESMModel. It is used to instantiate a ESM model
according to


EsmConfig

 the specified arguments, defining the model architecture. Instantiating a configuration with the
defaults will yield a similar configuration to that of the ESM
facebook/esm-1b architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Examples:   Copied >>> from transformers import EsmModel, EsmConfig
>>> # Initializing a ESM facebook/esm-1b style configuration >>> configuration = EsmConfig()
>>> # Initializing a model from the configuration >>> model = ESMModel(configuration)
>>> # Accessing the model configuration >>> configuration = model.config  to_dict  < source > ( ) → Dict[str, any]  Returns
Dict[str, any]

Dictionary of all the attributes that make up this configuration instance,
  Serializes this instance to a Python dictionary. Override the default to_dict()


EsmTokenizer

class transformers.EsmTokenizer  < source > ( vocab_file unk_token = '<unk>' cls_token = '<cls>' pad_token = '<pad>' mask_token = '<mask>' eos_token = '<eos>' **kwargs  )    Constructs an ESM tokenizer.  build_inputs_with_special_tokens  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None  )     get_special_tokens_mask  < source > ( token_ids_0: typing.List token_ids_1: typing.Optional[typing.List] = None already_has_special_tokens: bool = False  ) → A list of integers in the range [0, 1]  Parameters   token_ids_0 (List[int]) —
List of ids of the first sequence.   token_ids_1 (List[int], optional) —
List of ids of the second sequence.   already_has_special_tokens (bool, optional, defaults to False) —
Whether or not the token list is already formatted with special tokens for the model.   Returns


EsmTokenizer

A list of integers in the range [0, 1]

1 for a special token, 0 for a sequence token.
  Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer prepare_for_model or encode_plus methods.  create_token_type_ids_from_sequences  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None  ) → List[int]  Parameters   token_ids_0 (List[int]) — The first tokenized sequence.   token_ids_1 (List[int], optional) — The second tokenized sequence.   Returns
List[int]

The token type ids.
  Create the token type IDs corresponding to the sequences passed. What are token type
IDs? Should be overridden in a subclass if the model has a special way of building those.  save_vocabulary  < source > ( save_directory filename_prefix  )


EsmModel

class transformers.EsmModel  < source > ( config add_pooling_layer = True  )   Parameters   config (EsmConfig) — Model configuration class with all the parameters of the
model. Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare ESM Model transformer outputting raw hidden-states without any specific head on top. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior. The model can behave as an encoder (with only self-attent


EsmModel

ion) as well as a decoder, in which case a layer of
cross-attention is added between the self-attention layers, following the architecture described in Attention is
all you need by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit,
Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the is_decoder argument of the configuration set
to True. To be used in a Seq2Seq model, the model needs to initialized with both is_decoder argument and
add_cross_attention set to True; an encoder_hidden_states is then expected as an input to the forward pass.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None encoder_


EsmModel

hidden_states: typing.Optional[torch.Tensor] = None encoder_attention_mask: typing.Optional[torch.Tensor] = None past_key_values: typing.Optional[typing.List[torch.FloatTensor]] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape ((batch_size, sequence_length))) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape ((batch_size, sequence_length)), optional) —
Mask to avoid performing attention on padding token indices. Mask valu


EsmModel

es selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (torch.LongTensor of shape ((batch_size, sequence_length)), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape ((batch_size, sequence_length), hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associa


EsmModel

ted vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   encoder_hidden_states  (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if
the model is configured as a decoder.   encoder_attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on the padding token indices of the encoder i


EsmModel

nput. This mask is used in
the cross-attention if the model is configured as a decoder. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
   past_key_values (tuple(tuple(torch.FloatTensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)) —
Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   Returns
transfor


EsmModel

mers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (EsmConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing
through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns
the classification token after processing through a linear layer and a tanh activation function. The linear
lay


EsmModel

er weights are trained from the next sentence prediction (classification) objective during pretraining.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
cross_a


EsmModel

ttentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True and config.add_cross_attention=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and optionally if
config.is_encoder_decoder=True 2 additional tensors of shape (batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hi


EsmModel

dden-states (key and values in the self-attention blocks and optionally if
config.is_encoder_decoder=True in the cross-attention blocks) that can be used (see past_key_values
input) to speed up sequential decoding.
  The EsmModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, EsmModel
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/esm2_t6_8M_UR50D")
>>> model = EsmModel.from_pretrained("facebook/esm2_t6_8M_UR50D")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_


EsmForMaskedLM

class transformers.EsmForMaskedLM  < source > ( config  )   Parameters   config (EsmConfig) — Model configuration class with all the parameters of the
model. Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    ESM Model with a language modeling head on top. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.Tenso


EsmForMaskedLM

r] = None position_ids: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None encoder_hidden_states: typing.Optional[torch.FloatTensor] = None encoder_attention_mask: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.LongTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.MaskedLMOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_l


EsmForMaskedLM

ength), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation.


EsmForMaskedLM

This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Labels for computing the masked language modeling loss. Indices should be in [-100, 0, ..., config.vocab_size] (see input_ids docstring) Tokens with indices set to -100 are ignored (masked), the
loss is only computed for the tokens with labels in [0, ...,


EsmForMaskedLM

 config.vocab_size]   kwargs (Dict[str, any], optional, defaults to {}) —
Used to hide legacy arguments that have been deprecated.   Returns
transformers.modeling_outputs.MaskedLMOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.MaskedLMOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (EsmConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Masked language modeling (MLM) loss.
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of to


EsmForMaskedLM

rch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The EsmForMaskedLM forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pr


EsmForMaskedLM

e and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, EsmForMaskedLM
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/esm2_t6_8M_UR50D")
>>> model = EsmForMaskedLM.from_pretrained("facebook/esm2_t6_8M_UR50D")
>>> inputs = tokenizer("The capital of France is <mask>.", return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> # retrieve index of <mask>
>>> mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]
>>> predicted_token_id = logits[0, mask_token_index].argmax(axis=-1)
>>> labels = tokenizer("The capital of France is Paris.", return_tensors="pt")["input_ids"]
>>> # mask labels of non-<mask> tokens
>>> labels = torch.where(inputs.input_ids == tokenizer.mask_token_id, labels, -100)
>>> outputs = model(**inputs, labels=labe


EsmForSequenceClassification

class transformers.EsmForSequenceClassification  < source > ( config  )   Parameters   config (EsmConfig) — Model configuration class with all the parameters of the
model. Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    ESM Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled
output) e.g. for GLUE tasks. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < sourc


EsmForSequenceClassification

e > ( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.LongTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.SequenceClassifierOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Ma


EsmForSequenceClassification

sk to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want mo


EsmForSequenceClassification

re control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-Entropy).   Returns
transformers.modeling_ou