FlaxMBartForSequenceClassification

size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
encoder_last_hidden_state (jnp.ndarray of shape (batch_size, sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the initial embedding outputs.
encoder_attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shap


FlaxMBartForSequenceClassification

e (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
  The FlaxMBartPreTrainedModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, FlaxMBartForSequenceClassification
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25")
>>> model = FlaxMBartForSequenceClassification.from_pretrained("facebook/mbart-large-cc25")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="jax")
>>> outputs = model(**inputs)
>>> logits = outputs.logits  encode  < source > ( input_ids: Array attention


FlaxMBartForSequenceClassification

_mask: typing.Optional[jax.Array] = None position_ids: typing.Optional[jax.Array] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None train: bool = False params: dict = None dropout_rng: PRNGKey = None  ) → transformers.modeling_flax_outputs.FlaxBaseModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (jnp.ndarray of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (jnp.ndarray of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attentio


FlaxMBartForSequenceClassification

n masks?   position_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_flax_outputs.FlaxBaseModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_flax_outputs.FlaxBaseModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<clas


FlaxMBartForSequenceClassification

s 'transformers.models.mbart.configuration_mbart.MBartConfig'>) and inputs.
last_hidden_state (jnp.ndarray of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
   Exam


FlaxMBartForSequenceClassification

ple:   Copied >>> from transformers import AutoTokenizer, FlaxMBartForConditionalGeneration
>>> model = FlaxMBartForConditionalGeneration.from_pretrained("facebook/mbart-large-cc25")
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25")
>>> text = "My friends are cool but they eat too many carbs."
>>> inputs = tokenizer(text, max_length=1024, return_tensors="jax")
>>> encoder_outputs = model.encode(**inputs)  decode  < source > ( decoder_input_ids encoder_outputs encoder_attention_mask: typing.Optional[jax.Array] = None decoder_attention_mask: typing.Optional[jax.Array] = None decoder_position_ids: typing.Optional[jax.Array] = None past_key_values: dict = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None train: bool = False params: dict = None dropout_rng: PRNGKey = None  ) → transformers.modeling_flax_outputs.FlaxBaseModelOutputWithPastAndCros


FlaxMBartForSequenceClassification

sAttentions or tuple(torch.FloatTensor)  Parameters   decoder_input_ids (jnp.ndarray of shape (batch_size, target_sequence_length)) —
Indices of decoder input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are decoder input IDs?
For translation and summarization training, decoder_input_ids should be provided. If no
decoder_input_ids is provided, the model will create this tensor by shifting the input_ids to the right
for denoising pre-training following the paper.   encoder_outputs (tuple(tuple(jnp.ndarray)) —
Tuple consists of (last_hidden_state, optional: hidden_states, optional: attentions)
last_hidden_state of shape (batch_size, sequence_length, hidden_size), optional) is a sequence of
hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.   encoder_attention_mask (jnp.ndarray of shape (batch


FlaxMBartForSequenceClassification

_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   decoder_attention_mask (jnp.ndarray of shape (batch_size, target_sequence_length), optional) —
Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also
be used by default.
If you want to change padding behavior, you should modify to your needs. See diagram 1 in the
paper for more information on the default strategy.   decoder_position_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Indices of positions of each decoder input sequence tokens in the position embeddings. Selected in the
range [0, config.max_position_embeddings - 1].   past_key_values (Dict[str, np.ndarray], optional, returned by init_cache or when passing previous past_key_values) —
Dictionary of pre-computed


FlaxMBartForSequenceClassification

hidden-states (key and values in the attention blocks) that can be used for fast
auto-regressive decoding. Pre-computed key and value hidden-states are of shape [batch_size, max_length].   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_flax_outputs.FlaxBaseModelOutputWithPastAndCrossAttentions or tuple(torch.FloatTensor)

A transformers.modeling_flax_outputs.FlaxBaseModelOutputWithPastAndCrossAttentions or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration


FlaxMBartForSequenceClassification

(<class 'transformers.models.mbart.configuration_mbart.MBartConfig'>) and inputs.
last_hidden_state (jnp.ndarray of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
If past_key_values is used only the last hidden-state of the sequences of shape (batch_size, 1, hidden_size) is output.
past_key_values (tuple(tuple(jnp.ndarray)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(jnp.ndarray) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and optionally if
config.is_encoder_decoder=True 2 additional tensors of shape (batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if
config.is_encoder_decoder=True in the cross-attention blocks) that can be used


FlaxMBartForSequenceClassification

 (see past_key_values
input) to speed up sequential decoding.
hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
cross_attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True and config.add_cross_attention=True is passed or when config.output_attentions=True) — Tuple o


FlaxMBartForSequenceClassification

f jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
   Example:   Copied >>> from transformers import AutoTokenizer, FlaxMBartForConditionalGeneration
>>> model = FlaxMBartForConditionalGeneration.from_pretrained("facebook/mbart-large-cc25")
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25")
>>> text = "My friends are cool but they eat too many carbs."
>>> inputs = tokenizer(text, max_length=1024, return_tensors="jax")
>>> encoder_outputs = model.encode(**inputs)
>>> decoder_start_token_id = model.config.decoder_start_token_id
>>> decoder_input_ids = jnp.ones((inputs.input_ids.shape[0], 1), dtype="i4") * decoder_start_token_id
>>> outputs = model.decode(decoder_input_ids, encoder_outputs)
>>> last_decoder_hidden_states = outputs.last


FlaxMBartForQuestionAnswering

class transformers.FlaxMBartForQuestionAnswering  < source > ( config: MBartConfig input_shape: typing.Tuple[int] = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True **kwargs  )   Parameters   config (MBartConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.   dtype (jax.numpy.dtype, optional, defaults to jax.numpy.float32) —
The data type of the computation. Can be one of jax.numpy.float32, jax.numpy.float16 (on GPUs) and
jax.numpy.bfloat16 (on TPUs).
This can be used to enable mixed-precision training or half-precision inference on GPUs or TPUs. If
specified all the computation will be performed with the given dtype.
Note that this only specifies the dtype of the computation and does not influence the dtype of model
parameters.
If you wish to


FlaxMBartForQuestionAnswering

change the dtype of the model parameters, see to_fp16() and
to_bf16().    MBart Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear
layer on top of the hidden-states output to compute span start logits and span end logits). This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a Flax Linen
flax.nn.Module subclass. Use it as a
regular Flax Module and refer to the Flax documentation for all matter related to general usage and behavior. Finally, this model supports inherent JAX features such as: Just-In-Time (JIT) compilation Automatic Differentiation Vectorization Parallelization  __call__  < source > ( input_ids: Array attention_mask: typing.Optional[jax.Array] = None decoder_input_ids: typing.Optional[jax.Array] = None decoder_at


FlaxMBartForQuestionAnswering

tention_mask: typing.Optional[jax.Array] = None position_ids: typing.Optional[jax.Array] = None decoder_position_ids: typing.Optional[jax.Array] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None train: bool = False params: dict = None dropout_rng: PRNGKey = None  ) → transformers.modeling_flax_outputs.FlaxSeq2SeqQuestionAnsweringModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (jnp.ndarray of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (jnp.ndarray of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens


FlaxMBartForQuestionAnswering

 that are not masked,
0 for tokens that are masked.
What are attention masks?   decoder_input_ids (jnp.ndarray of shape (batch_size, target_sequence_length), optional) —
Indices of decoder input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are decoder input IDs?
For translation and summarization training, decoder_input_ids should be provided. If no
decoder_input_ids is provided, the model will create this tensor by shifting the input_ids to the right
for denoising pre-training following the paper.   decoder_attention_mask (jnp.ndarray of shape (batch_size, target_sequence_length), optional) —
Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also
be used by default.
If you want to change padding behavior, you should modify to your needs. See diagram 1 in the
paper for more information on the default strategy.


FlaxMBartForQuestionAnswering

  position_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].   decoder_position_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Indices of positions of each decoder input sequence tokens in the position embeddings. Selected in the
range [0, config.max_position_embeddings - 1].   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_flax_outputs.FlaxSeq2SeqQuestionAnsweringModelOutput or tuple(torch


FlaxMBartForQuestionAnswering

.FloatTensor)

A transformers.modeling_flax_outputs.FlaxSeq2SeqQuestionAnsweringModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (MBartConfig) and inputs.
start_logits (jnp.ndarray of shape (batch_size, sequence_length)) — Span-start scores (before SoftMax).
end_logits (jnp.ndarray of shape (batch_size, sequence_length)) — Span-end scores (before SoftMax).
past_key_values (tuple(tuple(jnp.ndarray)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(jnp.ndarray) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks)


FlaxMBartForQuestionAnswering

 that can be used (see past_key_values input) to speed up sequential decoding.
decoder_hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the decoder at the output of each layer plus the initial embedding outputs.
decoder_attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
cross_attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple


FlaxMBartForQuestionAnswering

of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
encoder_last_hidden_state (jnp.ndarray of shape (batch_size, sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the initial embedding outputs.
encoder_attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jn


FlaxMBartForQuestionAnswering

p.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
  The FlaxMBartPreTrainedModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, FlaxMBartForQuestionAnswering
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25")
>>> model = FlaxMBartForQuestionAnswering.from_pretrained("facebook/mbart-large-cc25")
>>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
>>> inputs = tokenizer(question, text, return_tensors="jax")
>>> outputs = model(**inpu


FlaxMBartForQuestionAnswering

ts)
>>> start_scores = outputs.start_logits
>>> end_scores = outputs.end_logits  encode  < source > ( input_ids: Array attention_mask: typing.Optional[jax.Array] = None position_ids: typing.Optional[jax.Array] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None train: bool = False params: dict = None dropout_rng: PRNGKey = None  ) → transformers.modeling_flax_outputs.FlaxBaseModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (jnp.ndarray of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (jnp.ndarray of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices.


FlaxMBartForQuestionAnswering

 Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_flax_outputs.FlaxBaseModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_flax_outputs.FlaxBaseModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed


FlaxMBartForQuestionAnswering

 or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.mbart.configuration_mbart.MBartConfig'>) and inputs.
last_hidden_state (jnp.ndarray of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attent


FlaxMBartForQuestionAnswering

ion softmax, used to compute the weighted average in the self-attention
heads.
   Example:   Copied >>> from transformers import AutoTokenizer, FlaxMBartForConditionalGeneration
>>> model = FlaxMBartForConditionalGeneration.from_pretrained("facebook/mbart-large-cc25")
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25")
>>> text = "My friends are cool but they eat too many carbs."
>>> inputs = tokenizer(text, max_length=1024, return_tensors="jax")
>>> encoder_outputs = model.encode(**inputs)  decode  < source > ( decoder_input_ids encoder_outputs encoder_attention_mask: typing.Optional[jax.Array] = None decoder_attention_mask: typing.Optional[jax.Array] = None decoder_position_ids: typing.Optional[jax.Array] = None past_key_values: dict = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None train: bool = False params: dict = None dropout_rng: PRNGKey = None  )


FlaxMBartForQuestionAnswering

 → transformers.modeling_flax_outputs.FlaxBaseModelOutputWithPastAndCrossAttentions or tuple(torch.FloatTensor)  Parameters   decoder_input_ids (jnp.ndarray of shape (batch_size, target_sequence_length)) —
Indices of decoder input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are decoder input IDs?
For translation and summarization training, decoder_input_ids should be provided. If no
decoder_input_ids is provided, the model will create this tensor by shifting the input_ids to the right
for denoising pre-training following the paper.   encoder_outputs (tuple(tuple(jnp.ndarray)) —
Tuple consists of (last_hidden_state, optional: hidden_states, optional: attentions)
last_hidden_state of shape (batch_size, sequence_length, hidden_size), optional) is a sequence of
hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the deco


FlaxMBartForQuestionAnswering

der.   encoder_attention_mask (jnp.ndarray of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   decoder_attention_mask (jnp.ndarray of shape (batch_size, target_sequence_length), optional) —
Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also
be used by default.
If you want to change padding behavior, you should modify to your needs. See diagram 1 in the
paper for more information on the default strategy.   decoder_position_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Indices of positions of each decoder input sequence tokens in the position embeddings. Selected in the
range [0, config.max_position_embeddings - 1].   past_key_values (Dict[str, np.ndarray], optional, returned by init_cache or when passing previous pa


FlaxMBartForQuestionAnswering

st_key_values) —
Dictionary of pre-computed hidden-states (key and values in the attention blocks) that can be used for fast
auto-regressive decoding. Pre-computed key and value hidden-states are of shape [batch_size, max_length].   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_flax_outputs.FlaxBaseModelOutputWithPastAndCrossAttentions or tuple(torch.FloatTensor)

A transformers.modeling_flax_outputs.FlaxBaseModelOutputWithPastAndCrossAttentions or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements d


FlaxMBartForQuestionAnswering

epending on the configuration (<class 'transformers.models.mbart.configuration_mbart.MBartConfig'>) and inputs.
last_hidden_state (jnp.ndarray of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
If past_key_values is used only the last hidden-state of the sequences of shape (batch_size, 1, hidden_size) is output.
past_key_values (tuple(tuple(jnp.ndarray)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(jnp.ndarray) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and optionally if
config.is_encoder_decoder=True 2 additional tensors of shape (batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if
config.is_encoder_decoder=True in the cross-attention blocks)


FlaxMBartForQuestionAnswering

that can be used (see past_key_values
input) to speed up sequential decoding.
hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
cross_attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True and config.add_cross_attention=True is passed or when config.output_attentions=True) — Tuple


FlaxMBartForQuestionAnswering

 of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
   Example:   Copied >>> from transformers import AutoTokenizer, FlaxMBartForConditionalGeneration
>>> model = FlaxMBartForConditionalGeneration.from_pretrained("facebook/mbart-large-cc25")
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25")
>>> text = "My friends are cool but they eat too many carbs."
>>> inputs = tokenizer(text, max_length=1024, return_tensors="jax")
>>> encoder_outputs = model.encode(**inputs)
>>> decoder_start_token_id = model.config.decoder_start_token_id
>>> decoder_input_ids = jnp.ones((inputs.input_ids.shape[0], 1), dtype="i4") * decoder_start_token_id
>>> outputs = model.decode(decoder_input_ids, encoder_outputs)
>>> last_decoder_hidden_states = outputs.last_hidden_stat


Overview

The ImageGPT model was proposed in Generative Pretraining from Pixels by Mark
Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, Ilya Sutskever. ImageGPT (iGPT) is a GPT-2-like
model trained to predict the next pixel value, allowing for both unconditional and conditional image generation.
The abstract from the paper is the following:
Inspired by progress in unsupervised representation learning for natural language, we examine whether similar models
can learn useful representations for images. We train a sequence Transformer to auto-regressively predict pixels,
without incorporating knowledge of the 2D input structure. Despite training on low-resolution ImageNet without labels,
we find that a GPT-2 scale model learns strong image representations as measured by linear probing, fine-tuning, and
low-data classification. On CIFAR-10, we achieve 96.3% accuracy with a lin


Overview

ear probe, outperforming a supervised Wide
ResNet, and 99.0% accuracy with full fine-tuning, matching the top supervised pre-trained models. We are also
competitive with self-supervised benchmarks on ImageNet when substituting pixels for a VQVAE encoding, achieving 69.0%
top-1 accuracy on a linear probe of our features.
Summary of the approach. Taken from the [original paper](https://cdn.openai.com/papers/Generative_Pretraining_from_Pixels_V2.pdf).
This model was contributed by nielsr, based on this issue. The original code can be found
here.
Tips:
ImageGPT is almost exactly the same as GPT-2, with the exception that a different activation
function is used (namely “quick gelu”), and the layer normalization layers don’t mean center the inputs. ImageGPT
also doesn’t have tied input- and output embeddings. As the time- and memory requirements of the attention mechanism of Transformers


Overview

scales quadratically in the sequence
length, the authors pre-trained ImageGPT on smaller input resolutions, such as 32x32 and 64x64. However, feeding a
sequence of 32x32x3=3072 tokens from 0..255 into a Transformer is still prohibitively large. Therefore, the authors
applied k-means clustering to the (R,G,B) pixel values with k=512. This way, we only have a 32*32 = 1024-long
sequence, but now of integers in the range 0..511. So we are shrinking the sequence length at the cost of a bigger
embedding matrix. In other words, the vocabulary size of ImageGPT is 512, + 1 for a special “start of sentence” (SOS)
token, used at the beginning of every sequence. One can use ImageGPTImageProcessor to prepare
images for the model. Despite being pre-trained entirely unsupervised (i.e. without the use of any labels), ImageGPT produces fairly
performant image features useful for downstream tasks, su


Overview

ch as image classification. The authors showed that the
features in the middle of the network are the most performant, and can be used as-is to train a linear model (such as
a sklearn logistic regression model for example). This is also referred to as “linear probing”. Features can be
easily obtained by first forwarding the image through the model, then specifying output_hidden_states=True, and
then average-pool the hidden states at whatever layer you like. Alternatively, one can further fine-tune the entire model on a downstream dataset, similar to BERT. For this, you can
use ImageGPTForImageClassification. ImageGPT comes in different sizes: there’s ImageGPT-small, ImageGPT-medium and ImageGPT-large. The authors did also
train an XL variant, which they didn’t release. The differences in size are summarized in the following table:
| Model variant   | Depths        | Hidden sizes


Overview

    |   Decoder hidden size |   Params (M) |   ImageNet-1k Top 1 |
|-----------------|---------------|---------------------|-----------------------|--------------|---------------------|
| MiT-b0          | [2, 2, 2, 2]  | [32, 64, 160, 256]  |                   256 |          3.7 |                70.5 |
| MiT-b1          | [2, 2, 2, 2]  | [64, 128, 320, 512] |                   256 |         14   |                78.7 |
| MiT-b2          | [3, 4, 6, 3]  | [64, 128, 320, 512] |                   768 |         25.4 |                81.6 |
| MiT-b3          | [3, 4, 18, 3] | [64, 128, 320, 512] |                   768 |         45.2 |                83.1 |
| MiT-b4          | [3, 8, 27, 3] | [64, 128, 320, 512] |                   768 |         62.6 |                83.6 |
| MiT-b5          | [3, 6, 40, 3] | [64, 128, 320, 512] |                   768 |         82   |                83


Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ImageGPT.
 Image Classification
Demo notebooks for ImageGPT can be found here. ImageGPTForImageClassification is supported by this example script and notebook. See also: Image classification task guide
If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we’ll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


ImageGPTConfig

class transformers.ImageGPTConfig  < source > ( vocab_size = 513 n_positions = 1024 n_embd = 512 n_layer = 24 n_head = 8 n_inner = None activation_function = 'quick_gelu' resid_pdrop = 0.1 embd_pdrop = 0.1 attn_pdrop = 0.1 layer_norm_epsilon = 1e-05 initializer_range = 0.02 scale_attn_weights = True use_cache = True tie_word_embeddings = False scale_attn_by_inverse_layer_idx = False reorder_and_upcast_attn = False **kwargs  )   Parameters   vocab_size (int, optional, defaults to 512) —
Vocabulary size of the GPT-2 model. Defines the number of different tokens that can be represented by the
inputs_ids passed when calling ImageGPTModel or TFImageGPTModel.   n_positions (int, optional, defaults to 32*32) —
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).   n_embd (int, optional, defaults


ImageGPTConfig

 to 512) —
Dimensionality of the embeddings and hidden states.   n_layer (int, optional, defaults to 24) —
Number of hidden layers in the Transformer encoder.   n_head (int, optional, defaults to 8) —
Number of attention heads for each attention layer in the Transformer encoder.   n_inner (int, optional, defaults to None) —
Dimensionality of the inner feed-forward layers. None will set it to 4 times n_embd   activation_function (str, optional, defaults to "quick_gelu") —
Activation function (can be one of the activation functions defined in src/transformers/activations.py).
Defaults to “quick_gelu”.   resid_pdrop (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.   embd_pdrop (int, optional, defaults to 0.1) —
The dropout ratio for the embeddings.   attn_pdrop (float, optional, defaults to 0.1) —
The dr


ImageGPTConfig

opout ratio for the attention.   layer_norm_epsilon (float, optional, defaults to 1e-5) —
The epsilon to use in the layer normalization layers.   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   scale_attn_weights (bool, optional, defaults to True) —
Scale attention weights by dividing by sqrt(hidden_size)..   use_cache (bool, optional, defaults to True) —
Whether or not the model should return the last key/values attentions (not used by all models).   scale_attn_by_inverse_layer_idx (bool, optional, defaults to False) —
Whether to additionally scale attention weights by 1 / layer_idx + 1.   reorder_and_upcast_attn (bool, optional, defaults to False) —
Whether to scale keys (K) prior to computing attention (dot-product) and upcast attention
dot-product/softmax to float() when tr


ImageGPTConfig

aining with mixed precision.    This is the configuration class to store the configuration of a ImageGPTModel or a TFImageGPTModel. It is
used to instantiate a GPT-2 model according to the specified arguments, defining the model architecture.
Instantiating a configuration with the defaults will yield a similar configuration to that of the ImageGPT
openai/imagegpt-small architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import ImageGPTConfig, ImageGPTModel
>>> # Initializing a ImageGPT configuration
>>> configuration = ImageGPTConfig()
>>> # Initializing a model (with random weights) from the configuration
>>> model = ImageGPTModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config


ImageGPTFeatureExtractor

class transformers.ImageGPTFeatureExtractor  < source > ( *args **kwargs  )     __call__  < source > ( images **kwargs  )    Preprocess an image or a batch of images.


ImageGPTImageProcessor

class transformers.ImageGPTImageProcessor  < source > ( clusters: typing.Union[typing.List[typing.List[int]], numpy.ndarray, NoneType] = None do_resize: bool = True size: typing.Dict[str, int] = None resample: Resampling = <Resampling.BILINEAR: 2> do_normalize: bool = True do_color_quantize: bool = True **kwargs  )   Parameters   clusters (np.ndarray or List[List[int]], optional) —
The color clusters to use, of shape (n_clusters, 3) when color quantizing. Can be overriden by clusters
in preprocess.   do_resize (bool, optional, defaults to True) —
Whether to resize the image’s dimensions to (size["height"], size["width"]). Can be overridden by
do_resize in preprocess.   size (Dict[str, int] optional, defaults to {"height" -- 256, "width": 256}):
Size of the image after resizing. Can be overridden by size in preprocess.   resample (PILImageResampling, optional, defaults to Resampling.BILINEAR) —
Resampling f


ImageGPTImageProcessor

ilter to use if resizing the image. Can be overridden by resample in preprocess.   do_normalize (bool, optional, defaults to True) —
Whether to normalize the image pixel value to between [-1, 1]. Can be overridden by do_normalize in
preprocess.   do_color_quantize (bool, optional, defaults to True) —
Whether to color quantize the image. Can be overridden by do_color_quantize in preprocess.    Constructs a ImageGPT image processor. This image processor can be used to resize images to a smaller resolution
(such as 32x32 or 64x64), normalize them and finally color quantize them to obtain sequences of “pixel values”
(color clusters).  preprocess  < source > ( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]] do_resize: bool = None size: typing.Dict[str, i


ImageGPTImageProcessor

nt] = None resample: Resampling = None do_normalize: bool = None do_color_quantize: typing.Optional[bool] = None clusters: typing.Union[typing.List[typing.List[int]], numpy.ndarray, NoneType] = None return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = <ChannelDimension.FIRST: 'channels_first'> input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None **kwargs  )   Parameters   images (ImageInput) —
Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If
passing in images with pixel values between 0 and 1, set do_normalize=False.   do_resize (bool, optional, defaults to self.do_resize) —
Whether to resize the image.   size (Dict[str, int], optional, defaults to self.size) —
Size of the image after resizi


ImageGPTImageProcessor

ng.   resample (int, optional, defaults to self.resample) —
Resampling filter to use if resizing the image. This can be one of the enum PILImageResampling, Only
has an effect if do_resize is set to True.   do_normalize (bool, optional, defaults to self.do_normalize) —
Whether to normalize the image   do_color_quantize (bool, optional, defaults to self.do_color_quantize) —
Whether to color quantize the image.   clusters (np.ndarray or List[List[int]], optional, defaults to self.clusters) —
Clusters used to quantize the image of shape (n_clusters, 3). Only has an effect if
do_color_quantize is set to True.   return_tensors (str or TensorType, optional) —
The type of tensors to return. Can be one of:
Unset: Return a list of np.ndarray.
TensorType.TENSORFLOW or 'tf': Return a batch of type tf.Tensor.
TensorType.PYTORCH or 'pt': Return a batch of type torch.Tensor.
TensorType.NUMPY or 'np': Return a batch of ty


ImageGPTImageProcessor

pe np.ndarray.
TensorType.JAX or 'jax': Return a batch of type jax.numpy.ndarray.
   data_format (ChannelDimension or str, optional, defaults to ChannelDimension.FIRST) —
The channel dimension format for the output image. Can be one of:
ChannelDimension.FIRST: image in (num_channels, height, width) format.
ChannelDimension.LAST: image in (height, width, num_channels) format.
Only has an effect if do_color_quantize is set to False.
   input_data_format (ChannelDimension or str, optional) —
The channel dimension format for the input image. If unset, the channel dimension format is inferred
from the input image. Can be one of:
"channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
"channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
"none" or ChannelDimension.NONE: image in (height, width) format.
    Preprocess an image or batch of image


ImageGPTModel

class transformers.ImageGPTModel  < source > ( config: ImageGPTConfig  )   Parameters   config (ImageGPTConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare ImageGPT Model transformer outputting raw hidden-states without any specific head on top. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None past_key_valu


ImageGPTModel

es: typing.Optional[typing.Tuple[typing.Tuple[torch.Tensor]]] = None attention_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None encoder_hidden_states: typing.Optional[torch.Tensor] = None encoder_attention_mask: typing.Optional[torch.Tensor] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None **kwargs: typing.Any  ) → transformers.modeling_outputs.BaseModelOutputWithPastAndCrossAttentions or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
input_ids_length = sequence_length if past_key_values is None else
past_key_values[0][0].shape[-2] (sequence_


ImageGPTModel

length of input past key value states). Indices of input
sequence tokens in the vocabulary.
If past_key_values is used, only input_ids that do not have their past calculated should be passed as
input_ids.
Indices can be obtained using AutoImageProcessor. See ImageGPTImageProcessor.call() for details.   past_key_values (Tuple[Tuple[torch.Tensor]] of length config.n_layers) —
Contains precomputed hidden-states (key and values in the attention blocks) as computed by the model (see
past_key_values output below). Can be used to speed up sequential decoding. The input_ids which have
their past given to this model should not be passed as input_ids as they have already been computed.   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attentio


ImageGPTModel

n masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_id


ImageGPTModel

s you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.
If past_key_values is used, optionally only the last inputs_embeds have to be input (see
past_key_values).   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,


ImageGPTModel

sequence_length), optional) —
Labels for language modeling. Note that the labels are shifted inside the model, i.e. you can set
labels = input_ids Indices are selected in [-100, 0, ..., config.vocab_size] All labels set to -100
are ignored (masked), the loss is only computed for labels in [0, ..., config.vocab_size]   Returns
transformers.modeling_outputs.BaseModelOutputWithPastAndCrossAttentions or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutputWithPastAndCrossAttentions or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (ImageGPTConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
If past_key_values is used only the last hidden-state of the sequences of shape (batch_size


ImageGPTModel

, 1, hidden_size) is output.
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and optionally if
config.is_encoder_decoder=True 2 additional tensors of shape (batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if
config.is_encoder_decoder=True in the cross-attention blocks) that can be used (see past_key_values
input) to speed up sequential decoding.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an


ImageGPTModel

embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True and config.add_cross_attention=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention so


ImageGPTModel

ftmax, used to compute the
weighted average in the cross-attention heads.
  The ImageGPTModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoImageProcessor, ImageGPTModel
>>> from PIL import Image
>>> import requests
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small")
>>> model = ImageGPTModel.from_pretrained("openai/imagegpt-small")
>>> inputs = image_processor(images=image, return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_


ImageGPTForCausalImageModeling

class transformers.ImageGPTForCausalImageModeling  < source > ( config: ImageGPTConfig  )   Parameters   config (ImageGPTConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The ImageGPT Model transformer with a language modeling head on top (linear layer with weights tied to the input
embeddings). This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward


ImageGPTForCausalImageModeling

< source > ( input_ids: typing.Optional[torch.Tensor] = None past_key_values: typing.Optional[typing.Tuple[typing.Tuple[torch.Tensor]]] = None attention_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None encoder_hidden_states: typing.Optional[torch.Tensor] = None encoder_attention_mask: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None **kwargs: typing.Any  ) → transformers.modeling_outputs.CausalLMOutputWithCrossAttentions or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape


ImageGPTForCausalImageModeling

 (batch_size, sequence_length)) —
input_ids_length = sequence_length if past_key_values is None else
past_key_values[0][0].shape[-2] (sequence_length of input past key value states). Indices of input
sequence tokens in the vocabulary.
If past_key_values is used, only input_ids that do not have their past calculated should be passed as
input_ids.
Indices can be obtained using AutoImageProcessor. See ImageGPTImageProcessor.call() for details.   past_key_values (Tuple[Tuple[torch.Tensor]] of length config.n_layers) —
Contains precomputed hidden-states (key and values in the attention blocks) as computed by the model (see
past_key_values output below). Can be used to speed up sequential decoding. The input_ids which have
their past given to this model should not be passed as input_ids as they have already been computed.   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —


ImageGPTForCausalImageModeling

Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 i


ImageGPTForCausalImageModeling

ndicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.
If past_key_values is used, optionally only the last inputs_embeds have to be input (see
past_key_values).   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return th


ImageGPTForCausalImageModeling

e hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Labels for language modeling. Note that the labels are shifted inside the model, i.e. you can set
labels = input_ids Indices are selected in [-100, 0, ..., config.vocab_size] All labels set to -100
are ignored (masked), the loss is only computed for labels in [0, ..., config.vocab_size]   Returns
transformers.modeling_outputs.CausalLMOutputWithCrossAttentions or tuple(torch.FloatTensor)

A transformers.modeling_outputs.CausalLMOutputWithCrossAttentions or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (ImageGPTConfig) and inputs.
loss


ImageGPTForCausalImageModeling

 (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Language modeling loss (for next-token prediction).
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTens


ImageGPTForCausalImageModeling

or (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Cross attentions weights after the attention softmax, used to compute the weighted average in the
cross-attention heads.
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of torch.FloatTensor tuples of length config.n_layers, with each tuple containing the cached key,
value states of the self-attention and the cross-attention layers if model is used in encoder-d


ImageGPTForCausalImageModeling

ecoder
setting. Only relevant if config.is_decoder = True.
Contains pre-computed hidden-states (key and values in the attention blocks) that can be used (see
past_key_values input) to speed up sequential decoding.
  The ImageGPTForCausalImageModeling forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoImageProcessor, ImageGPTForCausalImageModeling
>>> import torch
>>> import matplotlib.pyplot as plt
>>> import numpy as np
>>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small")
>>> model = ImageGPTForCausalImageModeling.from_pretrained("openai/imagegpt-small")
>>> device =


ImageGPTForCausalImageModeling

torch.device("cuda" if torch.cuda.is_available() else "cpu")
>>> model.to(device)
>>> # unconditional generation of 8 images
>>> batch_size = 4
>>> context = torch.full((batch_size, 1), model.config.vocab_size - 1)  # initialize with SOS token
>>> context = context.to(device)
>>> output = model.generate(
...     input_ids=context, max_length=model.config.n_positions + 1, temperature=1.0, do_sample=True, top_k=40
... )
>>> clusters = image_processor.clusters
>>> height = image_processor.size["height"]
>>> width = image_processor.size["width"]
>>> samples = output[:, 1:].cpu().detach().numpy()
>>> samples_img = [
...     np.reshape(np.rint(127.5 * (clusters[s] + 1.0)), [height, width, 3]).astype(np.uint8) for s in samples
... ]  # convert color cluster tokens back to pixels
>>> f, axes = plt.subplots(1, batch_size, dpi=300)
>>> for img, ax in zip(samples_img, axes):
...     ax.axis("off")
...     ax.imsho


ImageGPTForImageClassification

class transformers.ImageGPTForImageClassification  < source > ( config: ImageGPTConfig  )   Parameters   config (ImageGPTConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The ImageGPT Model transformer with an image classification head on top (linear layer).
ImageGPTForImageClassification average-pools the hidden states in order to do the classification. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < sou


ImageGPTForImageClassification

rce > ( input_ids: typing.Optional[torch.Tensor] = None past_key_values: typing.Optional[typing.Tuple[typing.Tuple[torch.Tensor]]] = None attention_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None **kwargs: typing.Any  ) → transformers.modeling_outputs.SequenceClassifierOutputWithPast or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
input_ids_length = sequence_length if past_key_values is None else
past_key_values[0][0].shape[-2] (sequence_length of input past key value states). Indices of


ImageGPTForImageClassification

input
sequence tokens in the vocabulary.
If past_key_values is used, only input_ids that do not have their past calculated should be passed as
input_ids.
Indices can be obtained using AutoImageProcessor. See ImageGPTImageProcessor.call() for details.   past_key_values (Tuple[Tuple[torch.Tensor]] of length config.n_layers) —
Contains precomputed hidden-states (key and values in the attention blocks) as computed by the model (see
past_key_values output below). Can be used to speed up sequential decoding. The input_ids which have
their past given to this model should not be passed as input_ids as they have already been computed.   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), op


ImageGPTForImageClassification

tional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert


ImageGPTForImageClassification

 input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.
If past_key_values is used, optionally only the last inputs_embeds have to be input (see
past_key_values).   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels =


ImageGPTForImageClassification

= 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-Entropy).   Returns
transformers.modeling_outputs.SequenceClassifierOutputWithPast or tuple(torch.FloatTensor)

A transformers.modeling_outputs.SequenceClassifierOutputWithPast or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (ImageGPTConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with ea


ImageGPTForImageClassification

ch tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head))
Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see
past_key_values input) to speed up sequential decoding.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attent


ImageGPTForImageClassification

ion softmax, used to compute the weighted average in the self-attention
heads.
  The ImageGPTForImageClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoImageProcessor, ImageGPTForImageClassification
>>> from PIL import Image
>>> import requests
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small")
>>> model = ImageGPTForImageClassification.from_pretrained("openai/imagegpt-small")
>>> inputs = image_processor(images=image, return_tensors="pt")
>>> outputs = model(**inputs)
>>> logits = outputs.log


Overview

The FlauBERT model was proposed in the paper FlauBERT: Unsupervised Language Model Pre-training for French by Hang Le et al. It’s a transformer model pretrained using a masked language
modeling (MLM) objective (like BERT).
The abstract from the paper is the following:
Language models have become a key step to achieve state-of-the art results in many different Natural Language
Processing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way
to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their
contextualization at the sentence level. This has been widely demonstrated for English using contextualized
representations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al.,
2019; Yang et al., 2019b). In this paper, we


Overview

introduce and share FlauBERT, a model learned on a very large and
heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for
Scientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text
classification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most of the
time they outperform other pretraining approaches. Different versions of FlauBERT as well as a unified evaluation
protocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared to the research
community for further reproducible experiments in French NLP.
This model was contributed by formiel. The original code can be found here.
Tips:
Like RoBERTa, without the sentence ordering prediction (so just trained on the MLM objective)


Documentation resources

Text classification task guide Token classification task guide Question answering task guide Masked language modeling task guide Multiple choice task guide


FlaubertConfig

class transformers.FlaubertConfig  < source > ( pre_norm = False layerdrop = 0.0 vocab_size = 30145 emb_dim = 2048 n_layers = 12 n_heads = 16 dropout = 0.1 attention_dropout = 0.1 gelu_activation = True sinusoidal_embeddings = False causal = False asm = False n_langs = 1 use_lang_emb = True max_position_embeddings = 512 embed_init_std = 0.02209708691207961 layer_norm_eps = 1e-12 init_std = 0.02 bos_index = 0 eos_index = 1 pad_index = 2 unk_index = 3 mask_index = 5 is_encoder = True summary_type = 'first' summary_use_proj = True summary_activation = None summary_proj_to_labels = True summary_first_dropout = 0.1 start_n_top = 5 end_n_top = 5 mask_token_id = 0 lang_id = 0 pad_token_id = 2 bos_token_id = 0 **kwargs  )   Parameters   pre_norm (bool, optional, defaults to False) —
Whether to apply the layer normalization before or after the feed forward layer following the attention in
each layer (Vaswani et al., Tensor2Tensor for Neural


FlaubertConfig

 Machine Translation. 2018)   layerdrop (float, optional, defaults to 0.0) —
Probability to drop layers during training (Fan et al., Reducing Transformer Depth on Demand with
Structured Dropout. ICLR 2020)   vocab_size (int, optional, defaults to 30145) —
Vocabulary size of the FlauBERT model. Defines the number of different tokens that can be represented by
the inputs_ids passed when calling FlaubertModel or TFFlaubertModel.   emb_dim (int, optional, defaults to 2048) —
Dimensionality of the encoder layers and the pooler layer.   n_layer (int, optional, defaults to 12) —
Number of hidden layers in the Transformer encoder.   n_head (int, optional, defaults to 16) —
Number of attention heads for each attention layer in the Transformer encoder.   dropout (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.   attention_dropout (float, optional, defaults to


FlaubertConfig

0.1) —
The dropout probability for the attention mechanism   gelu_activation (bool, optional, defaults to True) —
Whether or not to use a gelu activation instead of relu.   sinusoidal_embeddings (bool, optional, defaults to False) —
Whether or not to use sinusoidal positional embeddings instead of absolute positional embeddings.   causal (bool, optional, defaults to False) —
Whether or not the model should behave in a causal manner. Causal models use a triangular attention mask in
order to only attend to the left-side context instead if a bidirectional context.   asm (bool, optional, defaults to False) —
Whether or not to use an adaptive log softmax projection layer instead of a linear layer for the prediction
layer.   n_langs (int, optional, defaults to 1) —
The number of languages the model handles. Set to 1 for monolingual models.   use_lang_emb (bool, optional, defaults to True) —
Whether to use language embeddings. Some models


FlaubertConfig

 use additional language embeddings, see the multilingual
models page for information
on how to use them.   max_position_embeddings (int, optional, defaults to 512) —
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).   embed_init_std (float, optional, defaults to 2048^-0.5) —
The standard deviation of the truncated_normal_initializer for initializing the embedding matrices.   init_std (int, optional, defaults to 50257) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices except the
embedding matrices.   layer_norm_eps (float, optional, defaults to 1e-12) —
The epsilon used by the layer normalization layers.   bos_index (int, optional, defaults to 0) —
The index of the beginning of sentence token in the vocabulary.   eos_index (int, optional, defaults to 1) —
The index of the end of sentence


FlaubertConfig

 token in the vocabulary.   pad_index (int, optional, defaults to 2) —
The index of the padding token in the vocabulary.   unk_index (int, optional, defaults to 3) —
The index of the unknown token in the vocabulary.   mask_index (int, optional, defaults to 5) —
The index of the masking token in the vocabulary.   is_encoder(bool, optional, defaults to True) —
Whether or not the initialized model should be a transformer encoder or decoder as seen in Vaswani et al.   summary_type (string, optional, defaults to “first”) —
Argument used when doing sequence summary. Used in the sequence classification and multiple choice models.
Has to be one of the following options:
"last": Take the last token hidden state (like XLNet).
"first": Take the first token hidden state (like BERT).
"mean": Take the mean of all tokens hidden states.
"cls_index": Supply a Tensor of classification token position (like GPT/GPT-2).
"attn": Not implemented now, use


FlaubertConfig

 multi-head attention.
   summary_use_proj (bool, optional, defaults to True) —
Argument used when doing sequence summary. Used in the sequence classification and multiple choice models.
Whether or not to add a projection after the vector extraction.   summary_activation (str, optional) —
Argument used when doing sequence summary. Used in the sequence classification and multiple choice models.
Pass "tanh" for a tanh activation to the output, any other value will result in no activation.   summary_proj_to_labels (bool, optional, defaults to True) —
Used in the sequence classification and multiple choice models.
Whether the projection outputs should have config.num_labels or config.hidden_size classes.   summary_first_dropout (float, optional, defaults to 0.1) —
Used in the sequence classification and multiple choice models.
The dropout ratio to be used after the projection and activation.   start_n_top (int, optional, defaults to 5)


FlaubertConfig

 —
Used in the SQuAD evaluation script.   end_n_top (int, optional, defaults to 5) —
Used in the SQuAD evaluation script.   mask_token_id (int, optional, defaults to 0) —
Model agnostic parameter to identify masked tokens when generating text in an MLM context.   lang_id (int, optional, defaults to 1) —
The ID of the language used by the model. This parameter is used when generating text in a given language.    This is the configuration class to store the configuration of a FlaubertModel or a TFFlaubertModel. It is
used to instantiate a FlauBERT model according to the specified arguments, defining the model architecture.
Instantiating a configuration with the defaults will yield a similar configuration to that of the FlauBERT
flaubert/flaubert_base_uncased architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.


FlaubertTokenizer

class transformers.FlaubertTokenizer  < source > ( vocab_file merges_file do_lowercase = False unk_token = '<unk>' bos_token = '<s>' sep_token = '</s>' pad_token = '<pad>' cls_token = '</s>' mask_token = '<special1>' additional_special_tokens = ['<special0>', '<special1>', '<special2>', '<special3>', '<special4>', '<special5>', '<special6>', '<special7>', '<special8>', '<special9>'] lang2id = None id2lang = None **kwargs  )   Parameters   vocab_file (str) —
Vocabulary file.   merges_file (str) —
Merges file.   do_lowercase (bool, optional, defaults to False) —
Controls lower casing.   unk_token (str, optional, defaults to "<unk>") —
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.   bos_token (str, optional, defaults to "<s>") —
The beginning of sequence token that was used during pretrain


FlaubertTokenizer

ing. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the cls_token.
   sep_token (str, optional, defaults to "</s>") —
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.   pad_token (str, optional, defaults to "<pad>") —
The token used for padding, for example when batching sequences of different lengths.   cls_token (str, optional, defaults to "</s>") —
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence


FlaubertTokenizer

when built with special tokens.   mask_token (str, optional, defaults to "<special1>") —
The token used for masking values. This is the token used when training this model with masked language
modeling. This is the token which the model will try to predict.   additional_special_tokens (List[str], optional, defaults to ['<special0>', '<special1>', '<special2>', '<special3>', '<special4>', '<special5>', '<special6>', '<special7>', '<special8>', '<special9>']) —
List of additional special tokens.   lang2id (Dict[str, int], optional) —
Dictionary mapping languages string identifiers to their IDs.   id2lang (Dict[int, str], optional) —
Dictionary mapping language IDs to their string identifiers.    Construct a Flaubert tokenizer. Based on Byte-Pair Encoding. The tokenization process is the following: Moses preprocessing and tokenization. Normalizing all inputs text.


FlaubertTokenizer

The arguments special_tokens and the function set_special_tokens, can be used to add additional symbols (like
”classify”) to a vocabulary. The argument do_lowercase controls lower casing (automatically set for pretrained vocabularies). This tokenizer inherits from PreTrainedTokenizer which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.  build_inputs_with_special_tokens  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None  ) → List[int]  Parameters   token_ids_0 (List[int]) —
List of IDs to which the special tokens will be added.   token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.   Returns
List[int]

List of input IDs with the appropriate special tokens.
  Build model inputs from a sequence or a pair of sequenc


FlaubertTokenizer

e for sequence classification tasks by concatenating and
adding special tokens. An XLM sequence has the following format: single sequence: <s> X </s> pair of sequences: <s> A </s> B </s>  convert_tokens_to_string  < source > ( tokens  )    Converts a sequence of tokens (string) in a single string.  create_token_type_ids_from_sequences  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None  ) → List[int]  Parameters   token_ids_0 (List[int]) —
List of IDs.   token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.   Returns
List[int]

List of token type IDs according to the given sequence(s).
  Create a mask from the two sequences passed to be used in a sequence-pair classification task. An XLM sequence  pair mask has the following format:   Copied 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
| f


FlaubertTokenizer

irst sequence    | second sequence | If token_ids_1 is None, this method only returns the first portion of the mask (0s).  get_special_tokens_mask  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None already_has_special_tokens: bool = False  ) → List[int]  Parameters   token_ids_0 (List[int]) —
List of IDs.   token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.   already_has_special_tokens (bool, optional, defaults to False) —
Whether or not the token list is already formatted with special tokens for the model.   Returns
List[int]

A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
  Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer prepare_for_model method.


FlaubertModel

class transformers.FlaubertModel  < source > ( config  )     forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None langs: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.LongTensor] = None lengths: typing.Optional[torch.LongTensor] = None cache: typing.Union[typing.Dict[str, torch.FloatTensor], NoneType] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.BaseModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary


FlaubertModel

.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
Wh


FlaubertModel

at are position IDs?   lengths (torch.LongTensor of shape (batch_size,), optional) —
Length of each sentence that can be used to avoid performing attention on padding token indices. You can
also use attention_mask for the same result (see above), kept here for compatibility. Indices selected in
[0, ..., input_ids.size(-1)]:   cache (Dict[str, torch.FloatTensor], optional) —
Dictionary strings to torch.FloatTensor that contains precomputed hidden-states (key and values in the
attention blocks) as computed by the model (see cache output below). Can be used to speed up sequential
decoding. The dictionary object will be modified in-place during the forward pass to add newly computed
hidden-states.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates


FlaubertModel

 the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.BaseModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutput or


FlaubertModel

 a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (FlaubertConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.F


FlaubertModel

loatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The FlaubertModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, FlaubertModel
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("flaubert/flaubert_base_cased")
>>> model = FlaubertModel.from_pretrained("flaubert/flaubert_base_cased")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_sta


FlaubertWithLMHeadModel

class transformers.FlaubertWithLMHeadModel  < source > ( config  )   Parameters   config (FlaubertConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The Flaubert Model transformer with a language modeling head on top (linear layer with weights tied to the input
embeddings). This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optiona


FlaubertWithLMHeadModel

l[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None langs: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None lengths: typing.Optional[torch.Tensor] = None cache: typing.Union[typing.Dict[str, torch.Tensor], NoneType] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.MaskedLMOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedT


FlaubertWithLMHeadModel

okenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   lengths (torch.LongTensor of shape (batch_size,), optional) —
Lengt


FlaubertWithLMHeadModel

h of each sentence that can be used to avoid performing attention on padding token indices. You can
also use attention_mask for the same result (see above), kept here for compatibility. Indices selected in
[0, ..., input_ids.size(-1)]:   cache (Dict[str, torch.FloatTensor], optional) —
Dictionary strings to torch.FloatTensor that contains precomputed hidden-states (key and values in the
attention blocks) as computed by the model (see cache output below). Can be used to speed up sequential
decoding. The dictionary object will be modified in-place during the forward pass to add newly computed
hidden-states.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequen


FlaubertWithLMHeadModel

ce_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Labels for language modeling. Note that the labels are shifted inside the model, i.e. you can set
labels = input_ids Indices are selected i


FlaubertWithLMHeadModel

n [-100, 0, ..., config.vocab_size] All labels set to -100
are ignored (masked), the loss is only computed for labels in [0, ..., config.vocab_size]   Returns
transformers.modeling_outputs.MaskedLMOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.MaskedLMOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (FlaubertConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Masked language modeling (MLM) loss.
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of t


FlaubertWithLMHeadModel

orch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The FlaubertWithLMHeadModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post p


FlaubertWithLMHeadModel

rocessing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, FlaubertWithLMHeadModel
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("flaubert/flaubert_base_cased")
>>> model = FlaubertWithLMHeadModel.from_pretrained("flaubert/flaubert_base_cased")
>>> inputs = tokenizer("The capital of France is <special1>.", return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> # retrieve index of <special1>
>>> mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]
>>> predicted_token_id = logits[0, mask_token_index].argmax(axis=-1)
>>> labels = tokenizer("The capital of France is Paris.", return_tensors="pt")["input_ids"]
>>> # mask labels of non-<special1> tokens
>>> labels = torch.where(inputs.input_ids == tokenizer.mask_token_id, labels, -100)
>>> outputs = model(**inputs, labels=la


FlaubertForSequenceClassification

class transformers.FlaubertForSequenceClassification  < source > ( config  )   Parameters   config (FlaubertConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Flaubert Model with a sequence classification/regression head on top (a linear layer on top of the pooled output)
e.g. for GLUE tasks. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional