FlaxRobertaForSequenceClassification

g.return_dict=False) comprising various
elements depending on the configuration (RobertaConfig) and inputs.
logits (jnp.ndarray of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_len


FlaxRobertaForSequenceClassification

gth, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The FlaxRobertaPreTrainedModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, FlaxRobertaForSequenceClassification
>>> tokenizer = AutoTokenizer.from_pretrained("roberta-base")
>>> model = FlaxRobertaForSequenceClassification.from_pretrained("roberta-base")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="jax")
>>> outputs = model(**inputs)
>>> logits = outputs.log


FlaxRobertaForMultipleChoice

class transformers.FlaxRobertaForMultipleChoice  < source > ( config: RobertaConfig input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True gradient_checkpointing: bool = False **kwargs  )   Parameters   config (RobertaConfig) — Model configuration class with all the parameters of the
model. Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Roberta Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a
softmax) e.g. for RocStories/SWAG tasks. This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading, saving and converting weights from PyTorch models) This


FlaxRobertaForMultipleChoice

model is also a Flax Linen flax.linen.Module
subclass. Use it as a regular Flax linen Module and refer to the Flax documentation for all matter related to
general usage and behavior. Finally, this model supports inherent JAX features such as: Just-In-Time (JIT) compilation Automatic Differentiation Vectorization Parallelization  __call__  < source > ( input_ids attention_mask = None token_type_ids = None position_ids = None head_mask = None encoder_hidden_states = None encoder_attention_mask = None params: dict = None dropout_rng: PRNGKey = None train: bool = False output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None past_key_values: dict = None  ) → transformers.modeling_flax_outputs.FlaxMultipleChoiceModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (numpy.ndarray of shape (batch_size, num_cho


FlaxRobertaForMultipleChoice

ices, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (numpy.ndarray of shape (batch_size, num_choices, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (numpy.ndarray of shape (batch_size, num_choices, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (numpy.ndarray of shape (batch_size, num_choices, sequence_length), optional) —
Indices of


FlaxRobertaForMultipleChoice

 positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].   head_mask (numpy.ndarray of shape (batch_size, num_choices, sequence_length), optional) -- Mask to nullify selected heads of the attention modules. Mask values selected in [0, 1]`:
1 indicates the head is not masked,
0 indicates the head is masked.
   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_flax_outputs.FlaxMultipleChoiceModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_flax_outputs.FlaxMultipleChoiceModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (RobertaConfig) and inputs.
logits (jnp.ndarray of shape (batch_size, num_choices)) — num_choic


FlaxRobertaForMultipleChoice

es is the second dimension of the input tensors. (see input_ids above).
Classification scores (before SoftMax).
hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The FlaxRobertaPreTrainedModel forward method, overrides the _


FlaxRobertaForMultipleChoice

_call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, FlaxRobertaForMultipleChoice
>>> tokenizer = AutoTokenizer.from_pretrained("roberta-base")
>>> model = FlaxRobertaForMultipleChoice.from_pretrained("roberta-base")
>>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
>>> choice0 = "It is eaten with a fork and a knife."
>>> choice1 = "It is eaten while held in the hand."
>>> encoding = tokenizer([prompt, prompt], [choice0, choice1], return_tensors="jax", padding=True)
>>> outputs = model(**{k: v[None, :] for k, v in encoding.items()})
>>> logits = outputs.log


FlaxRobertaForTokenClassification

class transformers.FlaxRobertaForTokenClassification  < source > ( config: RobertaConfig input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True gradient_checkpointing: bool = False **kwargs  )   Parameters   config (RobertaConfig) — Model configuration class with all the parameters of the
model. Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Roberta Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for
Named-Entity-Recognition (NER) tasks. This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as


FlaxRobertaForTokenClassification

 downloading, saving and converting weights from PyTorch models) This model is also a Flax Linen flax.linen.Module
subclass. Use it as a regular Flax linen Module and refer to the Flax documentation for all matter related to
general usage and behavior. Finally, this model supports inherent JAX features such as: Just-In-Time (JIT) compilation Automatic Differentiation Vectorization Parallelization  __call__  < source > ( input_ids attention_mask = None token_type_ids = None position_ids = None head_mask = None encoder_hidden_states = None encoder_attention_mask = None params: dict = None dropout_rng: PRNGKey = None train: bool = False output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None past_key_values: dict = None  ) → transformers.modeling_fl


FlaxRobertaForTokenClassification

ax_outputs.FlaxTokenClassifierOutput or tuple(torch.FloatTensor)  Parameters   input_ids (numpy.ndarray of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corr


FlaxRobertaForTokenClassification

esponds to a sentence B token.
What are token type IDs?   position_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].   head_mask (numpy.ndarray of shape (batch_size, sequence_length), optional) -- Mask to nullify selected heads of the attention modules. Mask values selected in [0, 1]`:
1 indicates the head is not masked,
0 indicates the head is masked.
   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_flax_outputs.FlaxTokenClassifierOutput or tuple(torch.FloatTensor)

A transformers.modeling_flax_outputs.FlaxTokenClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is p


FlaxRobertaForTokenClassification

assed or when config.return_dict=False) comprising various
elements depending on the configuration (RobertaConfig) and inputs.
logits (jnp.ndarray of shape (batch_size, sequence_length, config.num_labels)) — Classification scores (before SoftMax).
hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_l


FlaxRobertaForTokenClassification

ength, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The FlaxRobertaPreTrainedModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, FlaxRobertaForTokenClassification
>>> tokenizer = AutoTokenizer.from_pretrained("roberta-base")
>>> model = FlaxRobertaForTokenClassification.from_pretrained("roberta-base")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="jax")
>>> outputs = model(**inputs)
>>> logits = outputs.l


FlaxRobertaForQuestionAnswering

class transformers.FlaxRobertaForQuestionAnswering  < source > ( config: RobertaConfig input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True gradient_checkpointing: bool = False **kwargs  )   Parameters   config (RobertaConfig) — Model configuration class with all the parameters of the
model. Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Roberta Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear
layers on top of the hidden-states output to compute span start logits and span end logits). This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (


FlaxRobertaForQuestionAnswering

such as downloading, saving and converting weights from PyTorch models) This model is also a Flax Linen flax.linen.Module
subclass. Use it as a regular Flax linen Module and refer to the Flax documentation for all matter related to
general usage and behavior. Finally, this model supports inherent JAX features such as: Just-In-Time (JIT) compilation Automatic Differentiation Vectorization Parallelization  __call__  < source > ( input_ids attention_mask = None token_type_ids = None position_ids = None head_mask = None encoder_hidden_states = None encoder_attention_mask = None params: dict = None dropout_rng: PRNGKey = None train: bool = False output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None past_key_values: dict = None  ) → transformers.modeling_flax_outputs.FlaxQuestionAnsweringModelO


FlaxRobertaForQuestionAnswering

utput or tuple(torch.FloatTensor)  Parameters   input_ids (numpy.ndarray of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (nump


FlaxRobertaForQuestionAnswering

y.ndarray of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].   head_mask (numpy.ndarray of shape (batch_size, sequence_length), optional) -- Mask to nullify selected heads of the attention modules. Mask values selected in [0, 1]`:
1 indicates the head is not masked,
0 indicates the head is masked.
   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_flax_outputs.FlaxQuestionAnsweringModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_flax_outputs.FlaxQuestionAnsweringModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (Roberta


FlaxRobertaForQuestionAnswering

Config) and inputs.
start_logits (jnp.ndarray of shape (batch_size, sequence_length)) — Span-start scores (before SoftMax).
end_logits (jnp.ndarray of shape (batch_size, sequence_length)) — Span-end scores (before SoftMax).
hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softm


FlaxRobertaForQuestionAnswering

ax, used to compute the weighted average in the self-attention
heads.
  The FlaxRobertaPreTrainedModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, FlaxRobertaForQuestionAnswering
>>> tokenizer = AutoTokenizer.from_pretrained("roberta-base")
>>> model = FlaxRobertaForQuestionAnswering.from_pretrained("roberta-base")
>>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
>>> inputs = tokenizer(question, text, return_tensors="jax")
>>> outputs = model(**inputs)
>>> start_scores = outputs.start_logits
>>> end_scores = outputs.end_logit


Overview

The MGP-STR model was proposed in Multi-Granularity Prediction for Scene Text Recognition by Peng Wang, Cheng Da, and Cong Yao. MGP-STR is a conceptually simple yet powerful vision Scene Text Recognition (STR) model, which is built upon the Vision Transformer (ViT). To integrate linguistic knowledge, Multi-Granularity Prediction (MGP) strategy is proposed to inject information from the language modality into the model in an implicit way.
The abstract from the paper is the following:
Scene text recognition (STR) has been an active research topic in computer vision for years. To tackle this challenging problem, numerous innovative methods have been successively proposed and incorporating linguistic knowledge into STR m


Overview

odels has recently become a prominent trend. In this work, we first draw inspiration from the recent progress in Vision Transformer (ViT) to construct a conceptually simple yet powerful vision STR model, which is built upon ViT and outperforms previous state-of-the-art models for scene text recognition, including both pure vision models and language-augmented methods. To integrate linguistic knowledge, we further propose a Multi-Granularity Prediction strategy to inject information from the language modality into the model in an implicit way, i.e. , subword representations (BPE and WordPiece) widely-used in NLP are introduced into the output space, in addition to the conventional character level representation, while


Overview

 no independent language model (LM) is adopted. The resultant algorithm (termed MGP-STR) is able to push the performance envelop of STR to an even higher level. Specifically, it achieves an average recognition accuracy of 93.35% on standard benchmarks.
MGP-STR architecture. Taken from the original paper.
Tips:
MGP-STR is trained on two synthetic datasets MJSynth (MJ) and SynthText(http://www.robots.ox.ac.uk/~vgg/data/scenetext/) (ST) without fine-tuning on other datasets. It achieves state-of-the-art results on six standard Latin scene text benchmarks, including 3 regular text datasets (IC13, SVT, IIIT) and 3 irregular ones (IC15, SVTP, CUTE). This model was contributed by yuekun. The original code can be found here.


Inference

MgpstrModel accepts images as input and generates three types of predictions, which represent textual information at different granularities.
The three types of predictions are fused to give the final prediction result.
The ViTImageProcessor class is responsible for preprocessing the input image and
MgpstrTokenizer decodes the generated character tokens to the target string. The
MgpstrProcessor wraps ViTImageProcessor and MgpstrTokenizer
into a single instance to both extract the input features and decode the predicted token ids.
Step-by-step Optical Character Recognition (OCR)
  Copied >>> from transformers im


Inference

port MgpstrProcessor, MgpstrForSceneTextRecognition
>>> import requests
>>> from PIL import Image
>>> processor = MgpstrProcessor.from_pretrained('alibaba-damo/mgp-str-base')
>>> model = MgpstrForSceneTextRecognition.from_pretrained('alibaba-damo/mgp-str-base')
>>> # load image from the IIIT-5k dataset
>>> url = "https://i.postimg.cc/ZKwLg2Gw/367-14.png"
>>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
>>> pixel_values = processor(images=image, return_tensors="pt").pixel_values
>>> outputs = model(pixel_values)
>>> generated_text = processor.batch_decode(outputs.logits)['generated_text'


MgpstrConfig

class transformers.MgpstrConfig  < source > ( image_size = [32, 128] patch_size = 4 num_channels = 3 max_token_length = 27 num_character_labels = 38 num_bpe_labels = 50257 num_wordpiece_labels = 30522 hidden_size = 768 num_hidden_layers = 12 num_attention_heads = 12 mlp_ratio = 4.0 qkv_bias = True distilled = False layer_norm_eps = 1e-05 drop_rate = 0.0 attn_drop_rate = 0.0 drop_path_rate = 0.0 output_a3_attentions = False initializer_range = 0.02 **kwargs  )   Parameters   image_size (List[int], optional, defaults to [32, 128]) —
The size (resolution) of each image.   patch_size (int, optional, defaults to 4) —
The size (resolution) of each patch.   num_channels (int, optional, defaults to 3) —
The number of input channels.   max_token_length (int, optional, defaults to 27) —
The max number of output tokens.   num_character_lab


MgpstrConfig

els (int, optional, defaults to 38) —
The number of classes for character head .   num_bpe_labels (int, optional, defaults to 50257) —
The number of classes for bpe head .   num_wordpiece_labels (int, optional, defaults to 30522) —
The number of classes for wordpiece head .   hidden_size (int, optional, defaults to 768) —
The embedding dimension.   num_hidden_layers (int, optional, defaults to 12) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 12) —
Number of attention heads for each attention layer in the Transformer encoder.   mlp_ratio (float, optional, defaults to 4.0) —
The ratio of mlp hidden dim to embedding dim.   qkv_bias (bool, optional, defaults to True) —
Whether to add a bias to the queries, keys and values.   distilled (bool, optional, defaults to False) —
Mo


MgpstrConfig

del includes a distillation token and head as in DeiT models.   layer_norm_eps (float, optional, defaults to 1e-05) —
The epsilon used by the layer normalization layers.   drop_rate (float, optional, defaults to 0.0) —
The dropout probability for all fully connected layers in the embeddings, encoder.   attn_drop_rate (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.   drop_path_rate (float, optional, defaults to 0.0) —
The stochastic depth rate.   output_a3_attentions (bool, optional, defaults to False) —
Whether or not the model should returns A^3 module attentions.   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.    This is the configuration class to store the configuration of an Mgpst


MgpstrConfig

rModel. It is used to instantiate an
MGP-STR model according to the specified arguments, defining the model architecture. Instantiating a configuration
with the defaults will yield a similar configuration to that of the MGP-STR
alibaba-damo/mgp-str-base architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import MgpstrConfig, MgpstrForSceneTextRecognition
>>> # Initializing a Mgpstr mgp-str-base style configuration
>>> configuration = MgpstrConfig()
>>> # Initializing a model (with random weights) from the mgp-str-base style configuration
>>> model = MgpstrForSceneTextRecognition(configuration)
>>> # Accessing the model configuration
>>> configuration = model.confi


MgpstrTokenizer

class transformers.MgpstrTokenizer  < source > ( vocab_file unk_token = '[GO]' bos_token = '[GO]' eos_token = '[s]' pad_token = '[GO]' **kwargs  )   Parameters   vocab_file (str) —
Path to the vocabulary file.   unk_token (str, optional, defaults to "[GO]") —
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.   bos_token (str, optional, defaults to "[GO]") —
The beginning of sequence token.   eos_token (str, optional, defaults to "[s]") —
The end of sequence tok


MgpstrTokenizer

en.   pad_token (str or tokenizers.AddedToken, optional, defaults to "[GO]") —
A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by
attention mechanisms or loss computation.    Construct a MGP-STR char tokenizer. This tokenizer inherits from PreTrainedTokenizer which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.  save_vocabulary  < source > ( save_directory: str filename_prefix: typing.Optional[str] = None  )


MgpstrProcessor

class transformers.MgpstrProcessor  < source > ( image_processor = None tokenizer = None **kwargs  )   Parameters   image_processor (ViTImageProcessor, optional) —
An instance of ViTImageProcessor. The image processor is a required input.   tokenizer (MgpstrTokenizer, optional) —
The tokenizer is a required input.    Constructs a MGP-STR processor which wraps an image processor and MGP-STR tokenizers into a single MgpstrProcessor offers all the functionalities of ViTImageProcessor] and MgpstrTokenizer. See the
call() and batch_decode() for more information.  __call__  < source > ( text = None images = None return_tensors = None **kwargs  )    When used in normal mode, this method forwards all its arguments to ViTImageProcessor’s
call() and returns its output. This method also forwards the text and kwargs
arguments to MgpstrTokenizer’s call() if text is not


MgpstrProcessor

None to encode the text. Please
refer to the doctsring of the above methods for more information.  batch_decode  < source > ( sequences  ) → Dict[str, any]  Parameters   sequences (torch.Tensor) —
List of tokenized input ids.   Returns
Dict[str, any]

Dictionary of all the outputs of the decoded results.
generated_text (List[str]): The final results after fusion of char, bpe, and wp. scores
(List[float]): The final scores after fusion of char, bpe, and wp. char_preds (List[str]): The list
of character decoded sentences. bpe_preds (List[str]): The list of bpe decoded sentences. wp_preds
(List[str]): The list of wp decoded sentences.
  Convert a list of lists of token ids into a list of strings by calling decode. This method forwards all its arguments to PreTrainedTokenizer’s batch_decode(). Please
refer to the docstring of this method for more information


MgpstrModel

class transformers.MgpstrModel  < source > ( config: MgpstrConfig  )   Parameters   config (MgpstrConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare MGP-STR Model transformer outputting raw hidden-states without any specific head on top.
This model is a PyTorch torch.nn.Module subclass. Use it
as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: FloatTensor output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  )   Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num


MgpstrModel

_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See ViTImageProcessor.call()
for details.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.    The MgpstrModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.


MgpstrForSceneTextRecognition

class transformers.MgpstrForSceneTextRecognition  < source > ( config: MgpstrConfig  )   Parameters   config (MgpstrConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    MGP-STR Model transformer with three classification heads on top (three A^3 modules and three linear layer on top
of the transformer encoder output) for scene text recognition (STR) . This model is a PyTorch torch.nn.Module subclass. Use it
as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: FloatTensor output_attentions: typing.Optional[bool] = None output_a3_attentions: typing


MgpstrForSceneTextRecognition

.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.mgp_str.modeling_mgp_str.MgpstrModelOutput or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See ViTImageProcessor.call()
for details.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   output_


MgpstrForSceneTextRecognition

a3_attentions (bool, optional) —
Whether or not to return the attentions tensors of a3 modules. See a3_attentions under returned tensors
for more detail.   Returns
transformers.models.mgp_str.modeling_mgp_str.MgpstrModelOutput or tuple(torch.FloatTensor)

A transformers.models.mgp_str.modeling_mgp_str.MgpstrModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.mgp_str.configuration_mgp_str.MgpstrConfig'>) and inputs.
logits (tuple(torch.FloatTensor) of shape (batch_size, config.num_character_labels)) — Tuple of torch.FloatTensor (one for the output of character of shape (batch_size, config.max_token_length, config.num_character_labels), + one for the output of bpe of shape (batch_size, config.max_to


MgpstrForSceneTextRecognition

ken_length, config.num_bpe_labels), + one for the output of wordpiece of shape (batch_size, config.max_token_length, config.num_wordpiece_labels)) .
Classification scores (before SoftMax) of character, bpe and wordpiece.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (bat


MgpstrForSceneTextRecognition

ch_size, config.max_token_length, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
a3_attentions (tuple(torch.FloatTensor), optional, returned when output_a3_attentions=True is passed or when config.output_a3_attentions=True) — Tuple of torch.FloatTensor (one for the attention of character, + one for the attention of bpe, + one for the attention of wordpiece) of shape (batch_size, config.max_token_length, sequence_length)`.
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The MgpstrForSceneTextRecognition forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwa


MgpstrForSceneTextRecognition

rds instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import (
...     MgpstrProcessor,
...     MgpstrForSceneTextRecognition,
... )
>>> import requests
>>> from PIL import Image
>>> # load image from the IIIT-5k dataset
>>> url = "https://i.postimg.cc/ZKwLg2Gw/367-14.png"
>>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
>>> processor = MgpstrProcessor.from_pretrained("alibaba-damo/mgp-str-base")
>>> pixel_values = processor(images=image, return_tensors="pt").pixel_values
>>> model = MgpstrForSceneTextRecognition.from_pretrained("alibaba-damo/mgp-str-base")
>>> # inference
>>> outputs = model(pixel_values)
>>> out_strs = processor.batch_decode(outputs.logits)
>>> out_strs["generated_text"]
'["ticket"]


Overview

The AltCLIP model was proposed in AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities by Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Fulong Ye, Qinghong Yang, Ledell Wu. AltCLIP
(Altering the Language Encoder in CLIP) is a neural network trained on a variety of image-text and text-text pairs. By switching CLIP’s
text encoder with a pretrained multilingual text encoder XLM-R, we could obtain very close performances with CLIP on almost all tasks, and extended original CLIP’s capabilities such as multilingual understanding.
The abstract from the paper is the following:
In this work, we present a conceptually simple and effective method to train a strong bilingual multimodal r


Overview

epresentation model.
Starting from the pretrained multimodal representation model CLIP released by OpenAI, we switched its text encoder with a pretrained
multilingual text encoder XLM-R, and aligned both languages and image representations by a two-stage training schema consisting of
teacher learning and contrastive learning. We validate our method through evaluations of a wide range of tasks. We set new state-of-the-art
performances on a bunch of tasks including ImageNet-CN, Flicker30k- CN, and COCO-CN. Further, we obtain very close performances with
CLIP on almost all tasks, suggesting that one can simply alter the text encoder in CLIP for extended capabilities such as multilingual understanding.


Usage

The usage of AltCLIP is very similar to the CLIP. the difference between CLIP is the text encoder. Note that we use bidirectional attention instead of casual attention
and we take the [CLS] token in XLM-R to represent text embedding.
AltCLIP is a multi-modal vision and language model. It can be used for image-text similarity and for zero-shot image
classification. AltCLIP uses a ViT like transformer to get visual features and a bidirectional language model to get the text
features. Both the text and visual features are then projected to a latent space with identical dimension. The dot
product between the projected image and text features is then used as a similar score.
To feed images to the Transformer encoder, each image


Usage

 is split into a sequence of fixed-size non-overlapping patches,
which are then linearly embedded. A [CLS] token is added to serve as representation of an entire image. The authors
also add absolute position embeddings, and feed the resulting sequence of vectors to a standard Transformer encoder.
The CLIPImageProcessor can be used to resize (or rescale) and normalize images for the model.
The AltCLIPProcessor wraps a CLIPImageProcessor and a XLMRobertaTokenizer into a single instance to both
encode the text and prepare the images. The following example shows how to get the image-text similarity scores using
AltCLIPProcessor and AltCLIPModel.
  Copied >>> from PIL import Image
>>> import requests
>>> from transformers impor


Usage

t AltCLIPModel, AltCLIPProcessor
>>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP")
>>> processor = AltCLIPProcessor.from_pretrained("BAAI/AltCLIP")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True)
>>> outputs = model(**inputs)
>>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
>>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities
Tips:
This model is build on CLIPModel, so use it like a original CLIP.
This model was contributed by jongjyh


AltCLIPConfig

class transformers.AltCLIPConfig  < source > ( text_config = None vision_config = None projection_dim = 768 logit_scale_init_value = 2.6592 **kwargs  )   Parameters   text_config (dict, optional) —
Dictionary of configuration options used to initialize AltCLIPTextConfig.   vision_config (dict, optional) —
Dictionary of configuration options used to initialize AltCLIPVisionConfig.   projection_dim (int, optional, defaults to 768) —
Dimentionality of text and vision projection layers.   logit_scale_init_value (float, optional, defaults to 2.6592) —
The inital value of the logit_scale paramter. Default is used as per the original CLIP implementation.   kwargs (optional) —
Dictionary of keyword arguments.    This is the configuration class to


AltCLIPConfig

store the configuration of a AltCLIPModel. It is used to instantiate an
AltCLIP model according to the specified arguments, defining the model architecture. Instantiating a configuration
with the defaults will yield a similar configuration to that of the AltCLIP
BAAI/AltCLIP architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import AltCLIPConfig, AltCLIPModel
>>> # Initializing a AltCLIPConfig with BAAI/AltCLIP style configuration
>>> configuration = AltCLIPConfig()
>>> # Initializing a AltCLIPModel (with random weights) from the BAAI/AltCLIP style configuration
>>> model =


AltCLIPConfig

 AltCLIPModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config
>>> # We can also initialize a AltCLIPConfig from a AltCLIPTextConfig and a AltCLIPVisionConfig
>>> # Initializing a AltCLIPText and AltCLIPVision configuration
>>> config_text = AltCLIPTextConfig()
>>> config_vision = AltCLIPVisionConfig()
>>> config = AltCLIPConfig.from_text_vision_configs(config_text, config_vision)  from_text_vision_configs  < source > ( text_config: AltCLIPTextConfig vision_config: AltCLIPVisionConfig **kwargs  ) → AltCLIPConfig  Returns
AltCLIPConfig

An instance of a configuration object
  Instantiate a AltCLIPConfig (or a derived class) from altclip text model configuration and altclip vision
model configuration


AltCLIPTextConfig

class transformers.AltCLIPTextConfig  < source > ( vocab_size = 250002 hidden_size = 1024 num_hidden_layers = 24 num_attention_heads = 16 intermediate_size = 4096 hidden_act = 'gelu' hidden_dropout_prob = 0.1 attention_probs_dropout_prob = 0.1 max_position_embeddings = 514 type_vocab_size = 1 initializer_range = 0.02 initializer_factor = 0.02 layer_norm_eps = 1e-05 pad_token_id = 1 bos_token_id = 0 eos_token_id = 2 position_embedding_type = 'absolute' use_cache = True project_dim = 768 **kwargs  )   Parameters   vocab_size (int, optional, defaults to 250002) —
Vocabulary size of the AltCLIP model. Defines the number of different tokens that can be represented by the
inputs_ids passed when calling AltCLIPTextModel.   hidden_size (int, optional, defaults to 1024) —
Dimensionality of the encoder


AltCLIPTextConfig

 layers and the pooler layer.   num_hidden_layers (int, optional, defaults to 24) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 16) —
Number of attention heads for each attention layer in the Transformer encoder.   intermediate_size (int, optional, defaults to 4096) —
Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.   hidden_act (str or Callable, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "silu" and "gelu_new" are supported.   hidden_dropout_prob (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.   attention_pr


AltCLIPTextConfig

obs_dropout_prob (float, optional, defaults to 0.1) —
The dropout ratio for the attention probabilities.   max_position_embeddings (int, optional, defaults to 514) —
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).   type_vocab_size (int, optional, defaults to 2) —
The vocabulary size of the token_type_ids passed when calling AltCLIPTextModel   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   layer_norm_eps (float, optional, defaults to 1e-5) —
The epsilon used by the layer normalization layers.   position_embedding_type (str, optional, defaults to "absolute") —
Type of position embedding.


AltCLIPTextConfig

Choose one of "absolute", "relative_key", "relative_key_query". For
positional embeddings use "absolute". For more information on "relative_key", please refer to
Self-Attention with Relative Position Representations (Shaw et al.).
For more information on "relative_key_query", please refer to Method 4 in Improve Transformer Models
with Better Relative Position Embeddings (Huang et al.).   use_cache (bool, optional, defaults to True) —
Whether or not the model should return the last key/values attentions (not used by all models). Only
relevant if config.is_decoder=True.   project_dim (int, optional, defaults to 768) —
The dimentions of the teacher model before the mapping layer.    This is the configuration class to store the configuration of a AltCLIPTextModel. It is used to instantiate a
AltC


AltCLIPTextConfig

LIP text model according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the AltCLIP
BAAI/AltCLIP architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Examples:   Copied >>> from transformers import AltCLIPTextModel, AltCLIPTextConfig
>>> # Initializing a AltCLIPTextConfig with BAAI/AltCLIP style configuration
>>> configuration = AltCLIPTextConfig()
>>> # Initializing a AltCLIPTextModel (with random weights) from the BAAI/AltCLIP style configuration
>>> model = AltCLIPTextModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.confi


AltCLIPVisionConfig

class transformers.AltCLIPVisionConfig  < source > ( hidden_size = 768 intermediate_size = 3072 projection_dim = 512 num_hidden_layers = 12 num_attention_heads = 12 num_channels = 3 image_size = 224 patch_size = 32 hidden_act = 'quick_gelu' layer_norm_eps = 1e-05 attention_dropout = 0.0 initializer_range = 0.02 initializer_factor = 1.0 **kwargs  )   Parameters   hidden_size (int, optional, defaults to 768) —
Dimensionality of the encoder layers and the pooler layer.   intermediate_size (int, optional, defaults to 3072) —
Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.   num_hidden_layers (int, optional, defaults to 12) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 12) —
Number of attention heads for each attention layer in the Transformer encoder.   image_size (int, optional, defaults to 224) —


AltCLIPVisionConfig

The size (resolution) of each image.   patch_size (int, optional, defaults to 32) —
The size (resolution) of each patch.   hidden_act (str or function, optional, defaults to "quick_gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "selu" and "gelu_new" `"quick_gelu" are supported.   layer_norm_eps (float, optional, defaults to 1e-5) —
The epsilon used by the layer normalization layers.   attention_dropout (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   initializer_factor (`float“, optional, defaults to 1) —
A factor for initializing all weight matrices (should be kept to 1, used internally for initialization
testing).    This is the c


AltCLIPVisionConfig

onfiguration class to store the configuration of a AltCLIPModel. It is used to instantiate an
AltCLIP model according to the specified arguments, defining the model architecture. Instantiating a configuration
with the defaults will yield a similar configuration to that of the AltCLIP
BAAI/AltCLIP architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import AltCLIPVisionConfig, AltCLIPVisionModel
>>> # Initializing a AltCLIPVisionConfig with BAAI/AltCLIP style configuration
>>> configuration = AltCLIPVisionConfig()
>>> # Initializing a AltCLIPVisionModel (with random weights) from the BAAI/AltCLIP style configuration
>>> model = AltCLIPVisionModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config


AltCLIPProcessor

class transformers.AltCLIPProcessor  < source > ( image_processor = None tokenizer = None **kwargs  )   Parameters   image_processor (CLIPImageProcessor, optional) —
The image processor is a required input.   tokenizer (XLMRobertaTokenizerFast, optional) —
The tokenizer is a required input.    Constructs a AltCLIP processor which wraps a CLIP image processor and a XLM-Roberta tokenizer into a single
processor. AltCLIPProcessor offers all the functionalities of CLIPImageProcessor and XLMRobertaTokenizerFast. See
the __call__() and decode() for more information.  batch_decode  < source > ( *args **kwargs  )    This method forwards all its arguments to XLMRobertaTokenizerFast’s batch_decode().
Please refer to the docstring of this method for more information.  decode  < source > ( *args **kwargs  )    This method forwards all its arguments to XLMRobertaTokenizerFast’s decode(). Please
refer to the docstring of this method for more information.


AltCLIPModel

class transformers.AltCLIPModel  < source > ( config: AltCLIPConfig  )     forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None pixel_values: typing.Optional[torch.FloatTensor] = None attention_mask: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.LongTensor] = None token_type_ids: typing.Optional[torch.Tensor] = None return_loss: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.altclip.modeling_altclip.AltCLIPOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedToken


AltCLIPModel

izer.call() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using
AutoImageProcessor. See CLIPImageProcessor.call() for details.   return_loss (bool, optional) —
Whether or not to return the contrastive loss.   output_attentions (bool, option


AltCLIPModel

al) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.models.altclip.modeling_altclip.AltCLIPOutput or tuple(torch.FloatTensor)

A transformers.models.altclip.modeling_altclip.AltCLIPOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.altclip.configuration_altclip.AltCLIPConfig'>) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when return_loss is True) — Contrastive loss for image-text similarity.
logit


AltCLIPModel

s_per_image:(torch.FloatTensor of shape (image_batch_size, text_batch_size)) — The scaled dot product scores between image_embeds and text_embeds. This represents the image-text
similarity scores.
logits_per_text:(torch.FloatTensor of shape (text_batch_size, image_batch_size)) — The scaled dot product scores between text_embeds and image_embeds. This represents the text-image
similarity scores.
text_embeds(torch.FloatTensor of shape (batch_size, output_dim) — The text embeddings obtained by applying the projection layer to the pooled output of AltCLIPTextModel.
image_embeds(torch.FloatTensor of shape (batch_size, output_dim) — The image embeddings obtained by applying the projection layer to the pooled output of
AltCLIPVisionModel.
text_model_output(BaseModelOutputWithPooling):
The output of the AltCLIPTextModel.
vision_model_output(BaseModelOutputWithPooling):
The output of the AltCLIPVisionModel.
  The AltCLIPModel f


AltCLIPModel

orward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, AltCLIPModel
>>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP")
>>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(
...     text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True
... )
>>> outputs = model(**inputs)
>>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
>


AltCLIPModel

>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities  get_text_features  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None token_type_ids = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → text_features (torch.FloatTensor of shape (batch_size, output_dim)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), opti


AltCLIPModel

onal) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
text_features (torch.FloatTensor of shape (batch_size, ou


AltCLIPModel

tput_dim)

The text embeddings obtained by
applying the projection layer to the pooled output of AltCLIPTextModel.
  The AltCLIPModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoProcessor, AltCLIPModel
>>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP")
>>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP")
>>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
>>> text_features = model.get_text_features(**inputs)  get_image_features  < source > ( pixel_values: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool]


AltCLIPModel

= None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → image_features (torch.FloatTensor of shape (batch_size, output_dim)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using
AutoImageProcessor. See CLIPImageProcessor.call() for details.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
image_features (torch.FloatTensor of shape (batch_size,


AltCLIPModel

 output_dim)

The image embeddings obtained by
applying the projection layer to the pooled output of AltCLIPVisionModel.
  The AltCLIPModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, AltCLIPModel
>>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP")
>>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt")
>>> image_features = model.get_image_features(**inputs


AltCLIPTextModel

class transformers.AltCLIPTextModel  < source > ( config  )     forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None encoder_hidden_states: typing.Optional[torch.Tensor] = None encoder_attention_mask: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None  ) → transformers.modeling_outputs.BaseModelOutputWithPoolingAndProjection or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length))


AltCLIPTextModel

 —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   output_attentions (bool, optional) —
Whether or not to return th


AltCLIPTextModel

e attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.BaseModelOutputWithPoolingAndProjection or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutputWithPoolingAndProjection or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.altclip.configuration_altclip.AltCLIPTextConfig'>) and inputs.
last_hidden_state (torch.FloatTensor of shape (


AltCLIPTextModel

batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing
through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns
the classification token after processing through a linear layer and a tanh activation function. The linear
layer weights are trained from the next sentence prediction (classification) objective during pretraining.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the mode


AltCLIPTextModel

l has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
projection_state (tuple(torch.FloatTensor), returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor of shape (batch_size,config.project_dim).
Text embeddings before the projection layer, used to mi


AltCLIPTextModel

mic the last hidden state of the teacher encoder.
  The AltCLIPTextModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoProcessor, AltCLIPTextModel
>>> model = AltCLIPTextModel.from_pretrained("BAAI/AltCLIP")
>>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP")
>>> texts = ["it's a cat", "it's a dog"]
>>> inputs = processor(text=texts, padding=True, return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_state = outputs.last_hidden_state
>>> pooled_output = outputs.pooler_output  # pooled CLS sta


AltCLIPVisionModel

class transformers.AltCLIPVisionModel  < source > ( config: AltCLIPVisionConfig  )     forward  < source > ( pixel_values: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.BaseModelOutputWithPooling or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using
AutoImageProcessor. See CLIPImageProcessor.call() for details.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned ten


AltCLIPVisionModel

sors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.BaseModelOutputWithPooling or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutputWithPooling or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.altclip.configuration_altclip.AltCLIPVisionConfig'>) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing
through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this r


AltCLIPVisionModel

eturns
the classification token after processing through a linear layer and a tanh activation function. The linear
layer weights are trained from the next sentence prediction (classification) objective during pretraining.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to co


AltCLIPVisionModel

mpute the weighted average in the self-attention
heads.
  The AltCLIPVisionModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, AltCLIPVisionModel
>>> model = AltCLIPVisionModel.from_pretrained("BAAI/AltCLIP")
>>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_state = outputs.last_hidden_state
>>> pooled_output = outputs.pooler_output  # pooled CLS stat


Overview

The RWKV model was proposed in this repo
It suggests a tweak in the traditional Transformer attention to make it linear. This way, the model can be used as recurrent network: passing inputs for timestamp 0 and timestamp 1 together is the same as passing inputs at timestamp 0, then inputs at timestamp 1 along with the state of timestamp 0 (see example below).
This can be more efficient than a regular Transformer and can deal with sentence of any length (even if the model uses a fixed context length for training).
This model was contributed by sgugger.
The original code can be found here.
Example of use as an RNN:
  Copied import torch
from transformers import AutoToken


Overview

izer, RwkvConfig, RwkvModel
model = RwkvModel.from_pretrained("sgugger/rwkv-430M-pile")
tokenizer = AutoTokenizer.from_pretrained("sgugger/rwkv-430M-pile")
inputs = tokenizer("This is an example.", return_tensors="pt")
# Feed everything to the model
outputs = model(inputs["input_ids"])
output_whole = outputs.last_hidden_state
outputs = model(inputs["input_ids"][:, :2])
output_one = outputs.last_hidden_state
# Using the state computed on the first inputs, we will get the same output
outputs = model(inputs["input_ids"][:, 2:], state=outputs.state)
output_two = outputs.last_hidden_state
torch.allclose(torch.cat([output_one, output_two], dim=1), output_whole, atol=1e-5)
I


Overview

f you want to make sure the model stops generating when '\n\n' is detected, we recommend using the following stopping criteria:
  Copied from transformers import StoppingCriteria
class RwkvStoppingCriteria(StoppingCriteria):
    def __init__(self, eos_sequence = [187,187], eos_token_id = 537):
        self.eos_sequence = eos_sequence
        self.eos_token_id = eos_token_id
    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
        last_2_ids = input_ids[:,-2:].tolist()
        return self.eos_sequence in last_2_ids
output = model.generate(inputs["input_ids"], max_new_tokens=64, stopping_criteria = [RwkvStoppingCriteria()


RwkvConfig

class transformers.RwkvConfig  < source > ( vocab_size = 50277 context_length = 1024 hidden_size = 4096 num_hidden_layers = 32 attention_hidden_size = None intermediate_size = None layer_norm_epsilon = 1e-05 bos_token_id = 0 eos_token_id = 0 rescale_every = 6 tie_word_embeddings = False use_cache = True **kwargs  )   Parameters   vocab_size (int, optional, defaults to 50277) —
Vocabulary size of the RWKV model. Defines the number of different tokens that can be represented by the
inputs_ids passed when calling RwkvModel.   context_length (int, optional, defaults to 1024) —
The maximum sequence length that this model can be be used with in a single forward (using it in RNN mode
lets use any sequence length).   hidden_size (int, optional, defaults to 4096) —
Dimensionality of the embeddings and hidden states.   num_hidden_layers (int, optional, defaults to 32) —
Number of hidden layers in the model.   attention_hidden_size (int, optional) —
Dimensiona


RwkvConfig

lity of the attention hidden states. Will default to hidden_size if unset.   intermediate_size (int, optional) —
Dimensionality of the inner feed-forward layers. Will default to 4 times hidden_size if unset.   layer_norm_eps (float, optional, defaults to 1e-5) —
The epsilon to use in the layer normalization layers.   bos_token_id (int, optional, defaults to 0) —
The id of the beginning of sentence token in the vocabulary. Defaults to 0 as RWKV uses the same tokenizer
as GPTNeoX.   eos_token_id (int, optional, defaults to 0) —
The id of the end of sentence token in the vocabulary. Defaults to 0 as RWKV uses the same tokenizer as
GPTNeoX.   rescale_every (int, optional, default to 6) —
At inference, the hidden states (and weights of the correponding output layers) are divided by 2 every
rescale_every layer. If set to 0 or a negative number, no rescale is done.   tie_word_embeddings (bool, optional, defaults to False) —
Whether or not to tie the word e


RwkvConfig

mbeddings with the input token embeddings.   use_cache (bool, optional, defaults to True) —
Whether or not the model should return the last state.    This is the configuration class to store the configuration of a RwkvModel. It is used to instantiate a RWKV
model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
defaults will yield a similar configuration to that of the RWVK-4
RWKV/rwkv-4-169m-pile architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import RwkvConfig, RwkvModel
>>> # Initializing a Rwkv configuration
>>> configuration = RwkvConfig()
>>> # Initializing a model (with random weights) from the configuration
>>> model = RwkvModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.conf


RwkvModel

class transformers.RwkvModel  < source > ( config  )   Parameters   config (RwkvConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare RWKV Model transformer outputting raw hidden-states without any specific head on top. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[to


RwkvModel

rch.LongTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None state: typing.Optional[typing.List[torch.FloatTensor]] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.rwkv.modeling_rwkv.RwkvOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, input_ids_length)) —
input_ids_length = sequence_length if past_key_values is None else
past_key_values[0][0].shape[-2] (sequence_length of input past key value states). Indices of input
sequence tokens in the vocabulary.
If past_key_values is used, only input_ids that do not have their past calculated should be passed as
input_ids.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input ID


RwkvModel

s?   attention_mask (torch.LongTensor of shape (batch_size, input_ids_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
This is currently not used by RwkvModel, but will be supported in the future.
What are attention masks?   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   state (tuple of five torch.FloatTensor of shape (batch_size, hidden_size, num_hidden_layers), optional) —
If passed along, the model uses the previous state in all the blocks (which will give the output for the
input_ids provided as if


RwkvModel

the model add state_input_ids + input_ids as context).   use_cache (bool, optional) —
If set to True, the last state is returned and can be used to quickly generate the next logits.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.models.rwkv.modeling_rwkv.RwkvOutput or tuple(torch.FloatTensor)

A transformers.models.rwkv.modeling_rwkv.RwkvOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (RwkvConfig) and inputs.
last_hidden_s


RwkvModel

tate (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
state (list of five torch.FloatTensor of shape (batch_size, hidden_size, num_hidden_layers)) — The state of the model at the last time step. Can be used in a forward method with the next input_ids to
avoid providing the old input_ids.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_atte


RwkvModel

ntions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The RwkvModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, RwkvModel
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-4-169m-pile")
>>> model = RwkvModel.from_pretrained("RWKV/rwkv-4-169m-pile")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_state


RwkvLMHeadModel

class transformers.RwkvForCausalLM  < source > ( config  )   Parameters   config (RwkvConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The RWKV Model transformer with a language modeling head on top (linear layer with weights tied to the input
embeddings). This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typi


RwkvLMHeadModel

ng.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.LongTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None state: typing.Optional[typing.List[torch.FloatTensor]] = None labels: typing.Optional[torch.LongTensor] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.rwkv.modeling_rwkv.RwkvCausalLMOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, input_ids_length)) —
input_ids_length = sequence_length if past_key_values is None else
past_key_values[0][0].shape[-2] (sequence_length of input past key value states). Indices of input
sequence tokens in the vocabulary.
If past_key_values is used, only input_ids that do not have their past calculated shou


RwkvLMHeadModel

ld be passed as
input_ids.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.LongTensor of shape (batch_size, input_ids_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
This is currently not used by RwkvModel, but will be supported in the future.
What are attention masks?   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   state (tuple of five torch.Float


RwkvLMHeadModel

Tensor of shape (batch_size, hidden_size, num_hidden_layers), optional) —
If passed along, the model uses the previous state in all the blocks (which will give the output for the
input_ids provided as if the model add state_input_ids + input_ids as context).   use_cache (bool, optional) —
If set to True, the last state is returned and can be used to quickly generate the next logits.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Labels for languag


RwkvLMHeadModel

e modeling. Note that the labels are shifted inside the model, i.e. you can set
labels = input_ids Indices are selected in [-100, 0, ..., config.vocab_size] All labels set to -100
are ignored (masked), the loss is only computed for labels in [0, ..., config.vocab_size]   Returns
transformers.models.rwkv.modeling_rwkv.RwkvCausalLMOutput or tuple(torch.FloatTensor)

A transformers.models.rwkv.modeling_rwkv.RwkvCausalLMOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (RwkvConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Language modeling loss (for next-token prediction).
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocab


RwkvLMHeadModel

ulary token before SoftMax).
state (list of five torch.FloatTensor of shape (batch_size, hidden_size, num_hidden_layers)) — The state of the model at the last time step. Can be used in a forward method with the next input_ids to
avoid providing the old input_ids.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size,


RwkvLMHeadModel

 num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The RwkvForCausalLM forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> import torch
>>> from transformers import AutoTokenizer, RwkvForCausalLM
>>> tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-4-169m-pile")
>>> model = RwkvForCausalLM.from_pretrained("RWKV/rwkv-4-169m-pile")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> outputs = model(**inputs, labels=inputs["input_ids"])
>>> loss = outputs.loss
>>> logits = outputs.logit


Rwkv attention and the recurrent formulas

In a traditional auto-regressive Transformer, attention is written as
O=softmax(QKT/d)VO = \hbox{softmax}(QK^{T} / \sqrt{d}) VO=softmax(QKT/d)V
with QQQ, KKK and VVV are matrices of shape seq_len x hidden_size named query, key and value (they are actually bigger matrices with a batch dimension and an attention head dimension but we’re only interested in the last two, which is where the matrix product is taken, so for the sake of simplicity we only consider those two). The product QKTQK^{T}QKT then has shape seq_len x seq_len and we can take the maxtrix product with VVV to get the output OOO of the same shape as the others.
Replacing the softmax by its value gives:
Oi=∑j=1ieQiKjT/dVj∑j=1ieQiKjT/dO_{i} = \frac{\sum_{j=1}^{i} e^{Q_{i} K_{j}^{T} / \sqrt{d}} V_{j}}{\sum_{j=1}^{i} e^{Q_{i} K_{j}^{T} / \sqrt{d}}}Oi=∑j=1ieQiKjT/d∑j=1ieQiKjT/dVj
Note that the entries in QKTQK^{T}QKT corresponding to j>ij > ij>i are ma


sked (the sum stops at j) because the attention is not allowed to look at future tokens (only past ones).
In comparison, the RWKV attention is given by
Oi=σ(Ri)∑j=1ieWi−j+KjVj∑j=1ieWi−j+KjO_{i} = \sigma(R_{i}) \frac{\sum_{j=1}^{i} e^{W_{i-j} + K_{j}} V_{j}}{\sum_{j=1}^{i} e^{W_{i-j} + K_{j}}}Oi=σ(Ri)∑j=1ieWi−j+Kj∑j=1ieWi−j+KjVj
where RRR is a new matrix called receptance by the author, KKK and VVV are still the key and value (σ\sigmaσ here is the sigmoid function). WWW is a new vector that represents the position of the token and is given by
W0=u and Wk=(k−1)w for k≥1W_{0} = u \hbox{  and  } W_{k} = (k-1)w \hbox{ for } k \geq 1W0=u and Wk=(k−1)w for k≥1
with uuu and www learnable parameters called in the code time_first and time_decay respectively. The numerator and denominator can both be expressed recursively. Naming them NiN_{i}Ni and DiD_{i}Di we have:
Ni=eu+KiVi+N^i where N^i=eKi−1Vi−1+ew+Ki−2Vi−2⋯+e(i


−2)w+K1V1N_{i} = e^{u + K_{i}} V_{i} + \hat{N}_{i} \hbox{  where  } \hat{N}_{i} = e^{K_{i-1}} V_{i-1} + e^{w + K_{i-2}} V_{i-2} \cdots + e^{(i-2)w + K_{1}} V_{1}Ni=eu+KiVi+N^i where N^i=eKi−1Vi−1+ew+Ki−2Vi−2⋯+e(i−2)w+K1V1
so N^i\hat{N}_{i}N^i (called numerator_state in the code) satistfies
N^0=0 and N^j+1=eKjVj+ewN^j\hat{N}_{0} = 0 \hbox{  and  } \hat{N}_{j+1} = e^{K_{j}} V_{j} + e^{w} \hat{N}_{j}N^0=0 and N^j+1=eKjVj+ewN^j
and
Di=eu+Ki+D^i where D^i=eKi−1+ew+Ki−2⋯+e(i−2)w+K1D_{i} = e^{u + K_{i}} + \hat{D}_{i} \hbox{  where  } \hat{D}_{i} = e^{K_{i-1}} + e^{w + K_{i-2}} \cdots + e^{(i-2)w + K_{1}}Di=eu+Ki+D^i where D^i=eKi−1+ew+Ki−2⋯+e(i−2)w+K1
so D^i\hat{D}_{i}D^i (called denominator_state in the code) satistfies
D^0=0 and D^j+1=eKj+ewD^j\hat{D}_{0} = 0 \hbox{  and  } \hat{D}_{j+1} = e^{K_{j}} + e^{w} \hat{D}_{j}D^0=0 and D^j+1=eKj+ewD^j
The actual recurrent formula used are a tiny bit more


complex, as for numerical stability we don’t want to compute exponentials of big numbers. Usually the softmax is not computed as is, but the exponential of the maximum term is divided of the numerator and denominator:
exi∑j=1nexj=exi−M∑j=1nexj−M\frac{e^{x_{i}}}{\sum_{j=1}^{n} e^{x_{j}}} = \frac{e^{x_{i} - M}}{\sum_{j=1}^{n} e^{x_{j} - M}}∑j=1nexjexi=∑j=1nexj−Mexi−M
with MMM the maximum of all xjx_{j}xj. So here on top of saving the numerator state (N^\hat{N}N^) and the denominator state (D^\hat{D}D^) we also keep track of the maximum of all terms encountered in the exponentials. So we actually use
N~i=e−MiN^i and D~i=e−MiD^i\tilde{N}_{i} = e^{-M_{i}} \hat{N}_{i} \hbox{  and  } \tilde{D}_{i} = e^{-M_{i}} \hat{D}_{i}N~i=e−MiN^i and D~i=e−MiD^i
defined by the following recurrent formulas:
N~0=0 and N~j+1=eKj−qVj+ew+Mj−qN~j where q=max(Kj,w+Mj)\tilde{N}_{0} = 0 \hbox{  and  } \tilde{N}_{j+1} = e^{K_{j} - q} V


_{j} + e^{w + M_{j} - q} \tilde{N}_{j} \hbox{  where  } q = \max(K_{j}, w + M_{j})N~0=0 and N~j+1=eKj−qVj+ew+Mj−qN~j where q=max(Kj,w+Mj)
and
D~0=0 and D~j+1=eKj−q+ew+Mj−qD~j where q=max(Kj,w+Mj)\tilde{D}_{0} = 0 \hbox{  and  } \tilde{D}_{j+1} = e^{K_{j} - q} + e^{w + M_{j} - q} \tilde{D}_{j} \hbox{  where  } q = \max(K_{j}, w + M_{j})D~0=0 and D~j+1=eKj−q+ew+Mj−qD~j where q=max(Kj,w+Mj)
and Mj+1=qM_{j+1} = qMj+1=q. With those, we can then compute
Ni=eu+Ki−qVi+eMiN~i where q=max(u+Ki,Mi)N_{i} = e^{u + K_{i} - q} V_{i} + e^{M_{i}} \tilde{N}_{i} \hbox{  where  } q = \max(u + K_{i}, M_{i})Ni=eu+Ki−qVi+eMiN~i where q=max(u+Ki,Mi)
and
Di=eu+Ki−q+eMiD~i where q=max(u+Ki,Mi)D_{i} = e^{u + K_{i} - q} + e^{M_{i}} \tilde{D}_{i} \hbox{  where  } q = \max(u + K_{i}, M_{i})Di=eu+Ki−q+eMiD~i where q=max(u+Ki,Mi)
which finally gives us
Oi=σ(Ri)NiDiO_{i} = \sigma(R_{i}) \frac{N_{i}}{D_{i}}Oi=σ(Ri)DiNi


Overview

The MVP model was proposed in MVP: Multi-task Supervised Pre-training for Natural Language Generation by Tianyi Tang, Junyi Li, Wayne Xin Zhao and Ji-Rong Wen.
According to the abstract,
MVP follows a standard Transformer encoder-decoder architecture. MVP is supervised pre-trained using labeled datasets. MVP also has task-specific soft prompts to stimulate the model’s capacity in performing a certain task. MVP is specially designed for natural language generation and can be adapted to a wide range of generation tasks, including but not limited to summarization, data-to-text generation, open-ended dialogue system, story generation, question answering, question generation, task-oriented dialogue system, commonsense generation, paraphrase generation, text style transfer, and text simplification. Our


Overview

 model can also be adapted to natural language understanding tasks such as sequence classification and (extractive) question answering.
Tips:
We have released a series of models here, including MVP, MVP with task-specific prompts, and multi-task pre-trained variants. If you want to use a model without prompts (standard Transformer), you can load it through MvpForConditionalGeneration.from_pretrained('RUCAIBox/mvp'). If you want to use a model with task-specific prompts, such as summarization, you can load it through MvpForConditionalGeneration.from_pretrained('RUCAIBox/mvp-summarization'). Our model supports lightweight prompt tuning following Prefix-tuning with method set_lightweight_tuning().
This model was contributed by Tianyi Tang. The detailed information and instructions can be found here.


Examples

For summarization, it is an example to use MVP and MVP with summarization-specific prompts.
  Copied >>> from transformers import MvpTokenizer, MvpForConditionalGeneration
>>> tokenizer = MvpTokenizer.from_pretrained("RUCAIBox/mvp")
>>> model = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mvp")
>>> model_with_prompt = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mvp-summarization")
>>> inputs = tokenizer(
...     "Summarize: You may want to stick it to your boss and leave your job, but don't do it if these are your reasons.",
...     return_tensors="pt",
... )
>>> generated_ids = model.generate(**inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
["Why You Shouldn't Quit Your Job"]
>>> generated_ids


Examples

 = model_with_prompt.generate(**inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
["Don't do it if these are your reasons"]
For data-to-text generation, it is an example to use MVP and multi-task pre-trained variants.
  Copied >>> from transformers import MvpTokenizerFast, MvpForConditionalGeneration
>>> tokenizer = MvpTokenizerFast.from_pretrained("RUCAIBox/mvp")
>>> model = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mvp")
>>> model_with_mtl = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mtl-data-to-text")
>>> inputs = tokenizer(
...     "Describe the following data: Iron Man | instance of | Superhero [SEP] Stan Lee | creator | Iron Man",
...     return_tensors="pt",
... )
>>> generated_ids = mo


Examples

del.generate(**inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
['Stan Lee created the character of Iron Man, a fictional superhero appearing in American comic']
>>> generated_ids = model_with_mtl.generate(**inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
['Iron Man is a fictional superhero appearing in American comic books published by Marvel Comics.']
For lightweight tuning, i.e., fixing the model and only tuning prompts, you can load MVP with randomly initialized prompts or with task-specific prompts. Our code also supports Prefix-tuning with BART following the original paper.
  Copied >>> from transformers import MvpForConditionalGeneration
>>> model = MvpForConditionalGeneration.fro


Examples

m_pretrained("RUCAIBox/mvp", use_prompt=True)
>>> # the number of trainable parameters (full tuning)
>>> sum(p.numel() for p in model.parameters() if p.requires_grad)
468116832
>>> # lightweight tuning with randomly initialized prompts
>>> model.set_lightweight_tuning()
>>> # the number of trainable parameters (lightweight tuning)
>>> sum(p.numel() for p in model.parameters() if p.requires_grad)
61823328
>>> # lightweight tuning with task-specific prompts
>>> model = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mtl-data-to-text")
>>> model.set_lightweight_tuning()
>>> # original lightweight Prefix-tuning
>>> model = MvpForConditionalGeneration.from_pretrained("facebook/bart-large", use_prompt=True)
>>> model.set_lightweight_tuning()


Documentation resources

Text classification task guide Question answering task guide Causal language modeling task guide Masked language modeling task guide Translation task guide Summarization task guide


MvpConfig

class transformers.MvpConfig  < source > ( vocab_size = 50267 max_position_embeddings = 1024 encoder_layers = 12 encoder_ffn_dim = 4096 encoder_attention_heads = 16 decoder_layers = 12 decoder_ffn_dim = 4096 decoder_attention_heads = 16 encoder_layerdrop = 0.0 decoder_layerdrop = 0.0 activation_function = 'gelu' d_model = 1024 dropout = 0.1 attention_dropout = 0.0 activation_dropout = 0.0 init_std = 0.02 classifier_dropout = 0.0 scale_embedding = False use_cache = True pad_token_id = 1 bos_token_id = 0 eos_token_id = 2 is_encoder_decoder = True decoder_start_token_id = 2 forced_eos_token_id = 2 use_prompt = False prompt_length = 100 prompt_mid_dim = 800 **kwargs  )   Parameters   vocab_size (int, optional, defaults to 50267) —
Vocabulary size of the MVP model. Defines the number of different tokens that can be represented by the
inputs_ids passed when calling MvpModel.   d_model (int, optional, defaults to


MvpConfig

1024) —
Dimensionality of the layers and the pooler layer.   encoder_layers (int, optional, defaults to 12) —
Number of encoder layers.   decoder_layers (int, optional, defaults to 12) —
Number of decoder layers.   encoder_attention_heads (int, optional, defaults to 16) —
Number of attention heads for each attention layer in the Transformer encoder.   decoder_attention_heads (int, optional, defaults to 16) —
Number of attention heads for each attention layer in the Transformer decoder.   decoder_ffn_dim (int, optional, defaults to 4096) —
Dimensionality of the “intermediate” (often named feed-forward) layer in decoder.   encoder_ffn_dim (int, optional, defaults to 4096) —
Dimensionality of the “intermediate” (often named feed-forward) layer in decoder.   activation_function (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If


MvpConfig

 string, "gelu",
"relu", "silu" and "gelu_new" are supported.   dropout (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.   attention_dropout (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.   activation_dropout (float, optional, defaults to 0.0) —
The dropout ratio for activations inside the fully connected layer.   classifier_dropout (float, optional, defaults to 0.0) —
The dropout ratio for classifier.   max_position_embeddings (int, optional, defaults to 1024) —
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).   init_std (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   encoder_layerdrop (float, optio


MvpConfig

nal, defaults to 0.0) —
The LayerDrop probability for the encoder. See the [LayerDrop paper](see https://arxiv.org/abs/1909.11556)
for more details.   decoder_layerdrop (float, optional, defaults to 0.0) —
The LayerDrop probability for the decoder. See the [LayerDrop paper](see https://arxiv.org/abs/1909.11556)
for more details.   scale_embedding (bool, optional, defaults to False) —
Scale embeddings by diving by sqrt(d_model).   use_cache (bool, optional, defaults to True) —
Whether or not the model should return the last key/values attentions (not used by all models).   forced_eos_token_id (int, optional, defaults to 2) —
The id of the token to force as the last generated token when max_length is reached. Usually set to
eos_token_id.   use_prompt (bool, optional, defaults to False) —
Whether or not to use prompt.   prompt_length (int, optional, defaults to 100) —
The length of prompt.   prompt_mid_dim (in


MvpConfig

t, optional, defaults to 800) —
Dimensionality of the “intermediate” layer in prompt.    This is the configuration class to store the configuration of a MvpModel. It is used to instantiate a MVP model
according to the specified arguments, defining the model architecture. Instantiating a configuration with the
defaults will yield a similar configuration to that of the MVP RUCAIBox/mvp
architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import MvpConfig, MvpModel
>>> # Initializing a MVP RUCAIBox/mvp style configuration
>>> configuration = MvpConfig()
>>> # Initializing a model (with random weights) from the RUCAIBox/mvp style configuration
>>> model = MvpModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.con


MvpTokenizer

class transformers.MvpTokenizer  < source > ( vocab_file merges_file errors = 'replace' bos_token = '<s>' eos_token = '</s>' sep_token = '</s>' cls_token = '<s>' unk_token = '<unk>' pad_token = '<pad>' mask_token = '<mask>' add_prefix_space = False **kwargs  )   Parameters   vocab_file (str) —
Path to the vocabulary file.   merges_file (str) —
Path to the merges file.   errors (str, optional, defaults to "replace") —
Paradigm to follow when decoding bytes to UTF-8. See
bytes.decode for more information.   bos_token (str, optional, defaults to "<s>") —
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the cls_token.
   eos_token (str, optional, defaults to "</s>") —
The end of sequence token.
When building a sequen