TFElectraForMaskedLM

ist of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_mask]) or model([input_ids, attention_mask, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None


TFElectraForMaskedLM

 return_dict: Optional[bool] = None labels: np.ndarray | tf.Tensor | None = None training: Optional[bool] = False  ) → transformers.modeling_tf_outputs.TFMaskedLMOutput or tuple(tf.Tensor)  Parameters   input_ids (Numpy array or tf.Tensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and
PreTrainedTokenizer.encode() for details.
What are input IDs?   attention_mask (Numpy array or tf.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (Numpy array or tf.Tensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selecte


TFElectraForMaskedLM

d in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (Numpy array or tf.Tensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (tf.Tensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the


TFElectraForMaskedLM

config will be used instead.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail. This argument can be used only in eager mode, in graph mode the value in the config will be
used instead.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in
eager mode, in graph mode the value will always be set to True.   training (bool, optional, defaults to False) —
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).   labels (tf.Tensor of shape (batch_size, sequence_length), optional) —
Labels for computing the masked language modeling loss. Indices should be in [-100, 0, ..., config.vocab_size] (see input_ids docstring) Tokens with indices set to -100 are ignored (ma


TFElectraForMaskedLM

sked), the
loss is only computed for the tokens with labels in [0, ..., config.vocab_size]   Returns
transformers.modeling_tf_outputs.TFMaskedLMOutput or tuple(tf.Tensor)

A transformers.modeling_tf_outputs.TFMaskedLMOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (ElectraConfig) and inputs.
loss (tf.Tensor of shape (n,), optional, where n is the number of non-masked labels, returned when labels is provided) — Masked language modeling (MLM) loss.
logits (tf.Tensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embedding


TFElectraForMaskedLM

s + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The TFElectraForMaskedLM forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer,


TFElectraForMaskedLM

TFElectraForMaskedLM
>>> import tensorflow as tf
>>> tokenizer = AutoTokenizer.from_pretrained("google/electra-small-generator")
>>> model = TFElectraForMaskedLM.from_pretrained("google/electra-small-generator")
>>> inputs = tokenizer("The capital of France is [MASK].", return_tensors="tf")
>>> logits = model(**inputs).logits
>>> # retrieve index of [MASK]
>>> mask_token_index = tf.where((inputs.input_ids == tokenizer.mask_token_id)[0])
>>> selected_logits = tf.gather_nd(logits[0], indices=mask_token_index)
>>> predicted_token_id = tf.math.argmax(selected_logits, axis=-1)
>>> tokenizer.decode(predicted_token_id)
'paris'    Copied >>> labels = tokenizer("The capital of France is Paris.", return_tensors="tf")["input_ids"]
>>> # mask labels of non-[MASK] tokens
>>> labels = tf.where(inputs.input_ids == tokenizer.mask_token_id, labels, -100)
>>> outputs = model(**inputs, labels=labels)
>>> round(float(outputs.loss), 2)
1.2


TFElectraForSequenceClassification

class transformers.TFElectraForSequenceClassification  < source > ( *args **kwargs  )   Parameters   config (ElectraConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    ELECTRA Model transformer with a sequence classification/regression head on top (a linear layer on top of the
pooled output) e.g. for GLUE tasks. This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavi


TFElectraForSequenceClassification

or. TensorFlow models and layers in transformers accept two formats as input: having all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a list of varying length with one or sever


TFElectraForSequenceClassification

al input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_mask]) or model([input_ids, attention_mask, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optiona


TFElectraForSequenceClassification

l[bool] = None labels: np.ndarray | tf.Tensor | None = None training: Optional[bool] = False  ) → transformers.modeling_tf_outputs.TFSequenceClassifierOutput or tuple(tf.Tensor)  Parameters   input_ids (Numpy array or tf.Tensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and
PreTrainedTokenizer.encode() for details.
What are input IDs?   attention_mask (Numpy array or tf.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (Numpy array or tf.Tensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings.


TFElectraForSequenceClassification

Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (Numpy array or tf.Tensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (tf.Tensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail. This argument can be used only in eager mode, in gra


TFElectraForSequenceClassification

ph mode the value in the
config will be used instead.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail. This argument can be used only in eager mode, in graph mode the value in the config will be
used instead.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in
eager mode, in graph mode the value will always be set to True.   training (bool, optional, defaults to False) —
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).   labels (tf.Tensor of shape (batch_size,), optional) —
Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regressio


TFElectraForSequenceClassification

n loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-Entropy).   Returns
transformers.modeling_tf_outputs.TFSequenceClassifierOutput or tuple(tf.Tensor)

A transformers.modeling_tf_outputs.TFSequenceClassifierOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (ElectraConfig) and inputs.
loss (tf.Tensor of shape (batch_size, ), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (tf.Tensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output


TFElectraForSequenceClassification

of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The TFElectraForSequenceClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Cop


TFElectraForSequenceClassification

ied >>> from transformers import AutoTokenizer, TFElectraForSequenceClassification
>>> import tensorflow as tf
>>> tokenizer = AutoTokenizer.from_pretrained("bhadresh-savani/electra-base-emotion")
>>> model = TFElectraForSequenceClassification.from_pretrained("bhadresh-savani/electra-base-emotion")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="tf")
>>> logits = model(**inputs).logits
>>> predicted_class_id = int(tf.math.argmax(logits, axis=-1)[0])
>>> model.config.id2label[predicted_class_id]
'joy'    Copied >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`
>>> num_labels = len(model.config.id2label)
>>> model = TFElectraForSequenceClassification.from_pretrained("bhadresh-savani/electra-base-emotion", num_labels=num_labels)
>>> labels = tf.constant(1)
>>> loss = model(**inputs, labels=labels).loss
>>> round(float(loss), 2)


TFElectraForMultipleChoice

class transformers.TFElectraForMultipleChoice  < source > ( *args **kwargs  )   Parameters   config (ElectraConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    ELECTRA Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a
softmax) e.g. for RocStories/SWAG tasks. This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to gener


TFElectraForMultipleChoice

al usage and
behavior. TensorFlow models and layers in transformers accept two formats as input: having all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a lis


TFElectraForMultipleChoice

t of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_mask]) or model([input_ids, attention_mask, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hi


TFElectraForMultipleChoice

dden_states: Optional[bool] = None return_dict: Optional[bool] = None labels: np.ndarray | tf.Tensor | None = None training: Optional[bool] = False  ) → transformers.modeling_tf_outputs.TFMultipleChoiceModelOutput or tuple(tf.Tensor)  Parameters   input_ids (Numpy array or tf.Tensor of shape (batch_size, num_choices, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and
PreTrainedTokenizer.encode() for details.
What are input IDs?   attention_mask (Numpy array or tf.Tensor of shape (batch_size, num_choices, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (Numpy array or tf.Tensor of shape (batch_size, num_choic


TFElectraForMultipleChoice

es, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (Numpy array or tf.Tensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (tf.Tensor of shape (batch_size, num_choices, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attenti


TFElectraForMultipleChoice

ons tensors of all attention layers. See attentions under returned
tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the
config will be used instead.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail. This argument can be used only in eager mode, in graph mode the value in the config will be
used instead.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in
eager mode, in graph mode the value will always be set to True.   training (bool, optional, defaults to False) —
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).   labels (tf.Tensor of shape (batch_size,), optiona


TFElectraForMultipleChoice

l) —
Labels for computing the multiple choice classification loss. Indices should be in [0, ..., num_choices]
where num_choices is the size of the second dimension of the input tensors. (See input_ids above)   Returns
transformers.modeling_tf_outputs.TFMultipleChoiceModelOutput or tuple(tf.Tensor)

A transformers.modeling_tf_outputs.TFMultipleChoiceModelOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (ElectraConfig) and inputs.
loss (tf.Tensor of shape (batch_size, ), optional, returned when labels is provided) — Classification loss.
logits (tf.Tensor of shape (batch_size, num_choices)) — num_choices is the second dimension of the input tensors. (see input_ids above).
Classification scores (before SoftMax).
hidden_states (tuple(tf.Tensor), optional, returned when output_hidd


TFElectraForMultipleChoice

en_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The TFElectraForMultipleChoice forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the


TFElectraForMultipleChoice

former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, TFElectraForMultipleChoice
>>> import tensorflow as tf
>>> tokenizer = AutoTokenizer.from_pretrained("google/electra-small-discriminator")
>>> model = TFElectraForMultipleChoice.from_pretrained("google/electra-small-discriminator")
>>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
>>> choice0 = "It is eaten with a fork and a knife."
>>> choice1 = "It is eaten while held in the hand."
>>> encoding = tokenizer([prompt, prompt], [choice0, choice1], return_tensors="tf", padding=True)
>>> inputs = {k: tf.expand_dims(v, 0) for k, v in encoding.items()}
>>> outputs = model(inputs)  # batch size is 1
>>> # the linear classifier still needs to be trained
>>> logits = outputs


TFElectraForTokenClassification

class transformers.TFElectraForTokenClassification  < source > ( *args **kwargs  )   Parameters   config (ElectraConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Electra model with a token classification head on top. Both the discriminator and generator may be loaded into this model. This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior. TensorFlow models and lay


TFElectraForTokenClassification

ers in transformers accept two formats as input: having all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a list of varying length with one or several input Tensors IN THE ORDER gi


TFElectraForTokenClassification

ven in the docstring:
model([input_ids, attention_mask]) or model([input_ids, attention_mask, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None labels: np.ndarray |


TFElectraForTokenClassification

 tf.Tensor | None = None training: Optional[bool] = False  ) → transformers.modeling_tf_outputs.TFTokenClassifierOutput or tuple(tf.Tensor)  Parameters   input_ids (Numpy array or tf.Tensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and
PreTrainedTokenizer.encode() for details.
What are input IDs?   attention_mask (Numpy array or tf.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (Numpy array or tf.Tensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_posi


TFElectraForTokenClassification

tion_embeddings - 1].
What are position IDs?   head_mask (Numpy array or tf.Tensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (tf.Tensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the
config will be used


TFElectraForTokenClassification

 instead.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail. This argument can be used only in eager mode, in graph mode the value in the config will be
used instead.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in
eager mode, in graph mode the value will always be set to True.   training (bool, optional, defaults to False) —
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).   labels (tf.Tensor of shape (batch_size, sequence_length), optional) —
Labels for computing the token classification loss. Indices should be in [0, ..., config.num_labels - 1].   Returns
transformers.modeling_tf_outputs.TFTokenClassifierOutput or tuple(tf.Te


TFElectraForTokenClassification

nsor)

A transformers.modeling_tf_outputs.TFTokenClassifierOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (ElectraConfig) and inputs.
loss (tf.Tensor of shape (n,), optional, where n is the number of unmasked labels, returned when labels is provided)  — Classification loss.
logits (tf.Tensor of shape (batch_size, sequence_length, config.num_labels)) — Classification scores (before SoftMax).
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional


TFElectraForTokenClassification

, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The TFElectraForTokenClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, TFElectraForTokenClassification
>>> import tensorflow as tf
>>> tokenizer = AutoTokenizer.from_pretrained("bhadresh-savani/electra-base-discriminator-finetuned-conll03-english")
>>> model = TFEl


TFElectraForTokenClassification

ectraForTokenClassification.from_pretrained("bhadresh-savani/electra-base-discriminator-finetuned-conll03-english")
>>> inputs = tokenizer(
...     "HuggingFace is a company based in Paris and New York", add_special_tokens=False, return_tensors="tf"
... )
>>> logits = model(**inputs).logits
>>> predicted_token_class_ids = tf.math.argmax(logits, axis=-1)
>>> # Note that tokens are classified rather then input words which means that
>>> # there might be more predicted token classes than words.
>>> # Multiple token classes might account for the same word
>>> predicted_tokens_classes = [model.config.id2label[t] for t in predicted_token_class_ids[0].numpy().tolist()]
>>> predicted_tokens_classes
['B-LOC', 'B-ORG', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'O', 'B-LOC', 'I-LOC']    Copied >>> labels = predicted_token_class_ids
>>> loss = tf.math.reduce_mean(model(**inputs, labels=labels).loss)
>>> round(float(loss), 2


TFElectraForQuestionAnswering

class transformers.TFElectraForQuestionAnswering  < source > ( *args **kwargs  )   Parameters   config (ElectraConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Electra Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear
layers on top of the hidden-states output to compute span start logits and span end logits). This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior. TensorFlow models


TFElectraForQuestionAnswering

and layers in transformers accept two formats as input: having all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_mask]) or model([i


TFElectraForQuestionAnswering

nput_ids, attention_mask, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None start_positions: np.ndarray | tf.Tensor | None = None end_positions: np.ndarray | tf.Tensor | None = None training: Optional[bool] = False  ) → transformers.modeli


TFElectraForQuestionAnswering

ng_tf_outputs.TFQuestionAnsweringModelOutput or tuple(tf.Tensor)  Parameters   input_ids (Numpy array or tf.Tensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and
PreTrainedTokenizer.encode() for details.
What are input IDs?   attention_mask (Numpy array or tf.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (Numpy array or tf.Tensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (Numpy array or tf.Tensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask


TFElectraForQuestionAnswering

to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (tf.Tensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the
config will be used instead.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail. This argument can be used only in eager mode, in graph mo


TFElectraForQuestionAnswering

de the value in the config will be
used instead.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in
eager mode, in graph mode the value will always be set to True.   training (bool, optional, defaults to False) —
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).   start_positions (tf.Tensor of shape (batch_size,), optional) —
Labels for position (index) of the start of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
are not taken into account for computing the loss.   end_positions (tf.Tensor of shape (batch_size,), optional) —
Labels for position (index) of the end of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_len


TFElectraForQuestionAnswering

gth). Position outside of the sequence
are not taken into account for computing the loss.   Returns
transformers.modeling_tf_outputs.TFQuestionAnsweringModelOutput or tuple(tf.Tensor)

A transformers.modeling_tf_outputs.TFQuestionAnsweringModelOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (ElectraConfig) and inputs.
loss (tf.Tensor of shape (batch_size, ), optional, returned when start_positions and end_positions are provided) — Total span extraction loss is the sum of a Cross-Entropy for the start and end positions.
start_logits (tf.Tensor of shape (batch_size, sequence_length)) — Span-start scores (before SoftMax).
end_logits (tf.Tensor of shape (batch_size, sequence_length)) — Span-end scores (before SoftMax).
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor


TFElectraForQuestionAnswering

(one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The TFElectraForQuestionAnswering forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, TFElectraForQue


TFElectraForQuestionAnswering

stionAnswering
>>> import tensorflow as tf
>>> tokenizer = AutoTokenizer.from_pretrained("bhadresh-savani/electra-base-squad2")
>>> model = TFElectraForQuestionAnswering.from_pretrained("bhadresh-savani/electra-base-squad2")
>>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
>>> inputs = tokenizer(question, text, return_tensors="tf")
>>> outputs = model(**inputs)
>>> answer_start_index = int(tf.math.argmax(outputs.start_logits, axis=-1)[0])
>>> answer_end_index = int(tf.math.argmax(outputs.end_logits, axis=-1)[0])
>>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
>>> tokenizer.decode(predict_answer_tokens)
'a nice puppet'    Copied >>> # target is "nice puppet"
>>> target_start_index = tf.constant([11])
>>> target_end_index = tf.constant([12])
>>> outputs = model(**inputs, start_positions=target_start_index, end_positions=target_end_index)
>>> loss = tf.math.reduce_mean(outputs.loss)
>>> round(float(loss), 2)
2.64


FlaxElectraModel

class transformers.FlaxElectraModel  < source > ( config: ElectraConfig input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True gradient_checkpointing: bool = False **kwargs  )   Parameters   config (ElectraConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare Electra Model transformer outputting raw hidden-states without any specific head on top. This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading, saving and converting weights from PyTorch models) Thi


FlaxElectraModel

s model is also a Flax Linen
flax.nn.Module subclass. Use it as a
regular Flax Module and refer to the Flax documentation for all matter related to general usage and behavior. Finally, this model supports inherent JAX features such as: Just-In-Time (JIT) compilation Automatic Differentiation Vectorization Parallelization  __call__  < source > ( input_ids attention_mask = None token_type_ids = None position_ids = None head_mask = None encoder_hidden_states = None encoder_attention_mask = None params: dict = None dropout_rng: PRNGKey = None train: bool = False output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None past_key_values: dict = None  ) → transformers.modeling_flax_outputs.FlaxBaseModelOutput or tuple(torch.FloatTensor)  Parameters


FlaxElectraModel

input_ids (numpy.ndarray of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (num


FlaxElectraModel

py.ndarray of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].   head_mask (numpy.ndarray of shape (batch_size, sequence_length), optional) -- Mask to nullify selected heads of the attention modules. Mask values selected in [0, 1]`:
1 indicates the head is not masked,
0 indicates the head is masked.
   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_flax_outputs.FlaxBaseModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_flax_outputs.FlaxBaseModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on th


FlaxElectraModel

e configuration (ElectraConfig) and inputs.
last_hidden_state (jnp.ndarray of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weight


FlaxElectraModel

s after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The FlaxElectraPreTrainedModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, FlaxElectraModel
>>> tokenizer = AutoTokenizer.from_pretrained("google/electra-small-discriminator")
>>> model = FlaxElectraModel.from_pretrained("google/electra-small-discriminator")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="jax")
>>> outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_state


FlaxElectraForPreTraining

class transformers.FlaxElectraForPreTraining  < source > ( config: ElectraConfig input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True gradient_checkpointing: bool = False **kwargs  )   Parameters   config (ElectraConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Electra model with a binary classification head on top as used during pretraining for identifying generated tokens. It is recommended to load the discriminator checkpoint into that model. This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as do


FlaxElectraForPreTraining

wnloading, saving and converting weights from PyTorch models) This model is also a Flax Linen
flax.nn.Module subclass. Use it as a
regular Flax Module and refer to the Flax documentation for all matter related to general usage and behavior. Finally, this model supports inherent JAX features such as: Just-In-Time (JIT) compilation Automatic Differentiation Vectorization Parallelization  __call__  < source > ( input_ids attention_mask = None token_type_ids = None position_ids = None head_mask = None encoder_hidden_states = None encoder_attention_mask = None params: dict = None dropout_rng: PRNGKey = None train: bool = False output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None past_key_values: dict = None  ) → transformers.models.electra.modeling_flax_electra.FlaxElectraForPr


FlaxElectraForPreTraining

eTrainingOutput or tuple(torch.FloatTensor)  Parameters   input_ids (numpy.ndarray of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type


FlaxElectraForPreTraining

IDs?   position_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].   head_mask (numpy.ndarray of shape (batch_size, sequence_length), optional) -- Mask to nullify selected heads of the attention modules. Mask values selected in [0, 1]`:
1 indicates the head is not masked,
0 indicates the head is masked.
   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.models.electra.modeling_flax_electra.FlaxElectraForPreTrainingOutput or tuple(torch.FloatTensor)

A transformers.models.electra.modeling_flax_electra.FlaxElectraForPreTrainingOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False)


FlaxElectraForPreTraining

 comprising various
elements depending on the configuration (ElectraConfig) and inputs.
logits (jnp.ndarray of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, seq


FlaxElectraForPreTraining

uence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The FlaxElectraPreTrainedModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, FlaxElectraForPreTraining
>>> tokenizer = AutoTokenizer.from_pretrained("google/electra-small-discriminator")
>>> model = FlaxElectraForPreTraining.from_pretrained("google/electra-small-discriminator")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="np")
>>> outputs = model(**inputs)
>>> prediction_logits = outputs.log


FlaxElectraForCausalLM

class transformers.FlaxElectraForCausalLM  < source > ( config: ElectraConfig input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True gradient_checkpointing: bool = False **kwargs  )   Parameters   config (ElectraConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Electra Model with a language modeling head on top (a linear layer on top of the hidden-states output) e.g for
autoregressive tasks. This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading, saving and converting weights from PyTorch m


FlaxElectraForCausalLM

odels) This model is also a Flax Linen
flax.nn.Module subclass. Use it as a
regular Flax Module and refer to the Flax documentation for all matter related to general usage and behavior. Finally, this model supports inherent JAX features such as: Just-In-Time (JIT) compilation Automatic Differentiation Vectorization Parallelization  __call__  < source > ( input_ids attention_mask = None token_type_ids = None position_ids = None head_mask = None encoder_hidden_states = None encoder_attention_mask = None params: dict = None dropout_rng: PRNGKey = None train: bool = False output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None past_key_values: dict = None  ) → transformers.modeling_flax_outputs.FlaxCausalLMOutputWithCrossAttentions or tuple(torch.FloatTensor)  Parameters   inp


FlaxElectraForCausalLM

ut_ids (numpy.ndarray of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (numpy.ndarray of shape (batch_size, s


FlaxElectraForCausalLM

equence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].   head_mask (numpy.ndarray of shape (batch_size, sequence_length), optional) -- Mask to nullify selected heads of the attention modules. Mask values selected in [0, 1]`:
1 indicates the head is not masked,
0 indicates the head is masked.
   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_flax_outputs.FlaxCausalLMOutputWithCrossAttentions or tuple(torch.FloatTensor)

A transformers.modeling_flax_outputs.FlaxCausalLMOutputWithCrossAttentions or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (ElectraConfi


FlaxElectraForCausalLM

g) and inputs.
logits (jnp.ndarray of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to


FlaxElectraForCausalLM

 compute the weighted average in the self-attention
heads.
cross_attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Cross attentions weights after the attention softmax, used to compute the weighted average in the
cross-attention heads.
past_key_values (tuple(tuple(jnp.ndarray)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of jnp.ndarray tuples of length config.n_layers, with each tuple containing the cached key, value
states of the self-attention and the cross-attention layers if model is used in encoder-decoder setting.
Only relevant if config.is_decoder = True.
Contains pre-computed hidden-states (key and values in the atten


FlaxElectraForCausalLM

tion blocks) that can be used (see
past_key_values input) to speed up sequential decoding.
  The FlaxElectraPreTrainedModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, FlaxElectraForCausalLM
>>> tokenizer = AutoTokenizer.from_pretrained("google/electra-small-discriminator")
>>> model = FlaxElectraForCausalLM.from_pretrained("google/electra-small-discriminator")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="np")
>>> outputs = model(**inputs)
>>> # retrieve logts for next token
>>> next_token_logits = outputs.logits[:


FlaxElectraForMaskedLM

class transformers.FlaxElectraForMaskedLM  < source > ( config: ElectraConfig input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True gradient_checkpointing: bool = False **kwargs  )   Parameters   config (ElectraConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Electra Model with a language modeling head on top. This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading, saving and converting weights from PyTorch models) This model is also a Flax Linen
flax.nn


FlaxElectraForMaskedLM

.Module subclass. Use it as a
regular Flax Module and refer to the Flax documentation for all matter related to general usage and behavior. Finally, this model supports inherent JAX features such as: Just-In-Time (JIT) compilation Automatic Differentiation Vectorization Parallelization  __call__  < source > ( input_ids attention_mask = None token_type_ids = None position_ids = None head_mask = None encoder_hidden_states = None encoder_attention_mask = None params: dict = None dropout_rng: PRNGKey = None train: bool = False output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None past_key_values: dict = None  ) → transformers.modeling_flax_outputs.FlaxMaskedLMOutput or tuple(torch.FloatTensor)  Parameters   input_ids (numpy.ndarray of shape


FlaxElectraForMaskedLM

 (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (numpy.ndarray of shape (batch_si


FlaxElectraForMaskedLM

ze, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].   head_mask (numpy.ndarray of shape (batch_size, sequence_length), optional) -- Mask to nullify selected heads of the attention modules. Mask values selected in [0, 1]`:
1 indicates the head is not masked,
0 indicates the head is masked.
   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_flax_outputs.FlaxMaskedLMOutput or tuple(torch.FloatTensor)

A transformers.modeling_flax_outputs.FlaxMaskedLMOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (ElectraCon


FlaxElectraForMaskedLM

fig) and inputs.
logits (jnp.ndarray of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights


FlaxElectraForMaskedLM

 after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The FlaxElectraPreTrainedModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, FlaxElectraForMaskedLM
>>> tokenizer = AutoTokenizer.from_pretrained("google/electra-small-discriminator")
>>> model = FlaxElectraForMaskedLM.from_pretrained("google/electra-small-discriminator")
>>> inputs = tokenizer("The capital of France is [MASK].", return_tensors="jax")
>>> outputs = model(**inputs)
>>> logits = outputs.log


FlaxElectraForSequenceClassification

class transformers.FlaxElectraForSequenceClassification  < source > ( config: ElectraConfig input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True gradient_checkpointing: bool = False **kwargs  )   Parameters   config (ElectraConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Electra Model transformer with a sequence classification/regression head on top (a linear layer on top of the
pooled output) e.g. for GLUE tasks. This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading, sa


FlaxElectraForSequenceClassification

ving and converting weights from PyTorch models) This model is also a Flax Linen
flax.nn.Module subclass. Use it as a
regular Flax Module and refer to the Flax documentation for all matter related to general usage and behavior. Finally, this model supports inherent JAX features such as: Just-In-Time (JIT) compilation Automatic Differentiation Vectorization Parallelization  __call__  < source > ( input_ids attention_mask = None token_type_ids = None position_ids = None head_mask = None encoder_hidden_states = None encoder_attention_mask = None params: dict = None dropout_rng: PRNGKey = None train: bool = False output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None past_key_values: dict = None  ) → transformers.modeling_flax_outputs.FlaxSequenceClassifierOutp


FlaxElectraForSequenceClassification

ut or tuple(torch.FloatTensor)  Parameters   input_ids (numpy.ndarray of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token


FlaxElectraForSequenceClassification

type IDs?   position_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].   head_mask (numpy.ndarray of shape (batch_size, sequence_length), optional) -- Mask to nullify selected heads of the attention modules. Mask values selected in [0, 1]`:
1 indicates the head is not masked,
0 indicates the head is masked.
   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_flax_outputs.FlaxSequenceClassifierOutput or tuple(torch.FloatTensor)

A transformers.modeling_flax_outputs.FlaxSequenceClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising v


FlaxElectraForSequenceClassification

arious
elements depending on the configuration (ElectraConfig) and inputs.
logits (jnp.ndarray of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weigh


FlaxElectraForSequenceClassification

ts after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The FlaxElectraPreTrainedModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, FlaxElectraForSequenceClassification
>>> tokenizer = AutoTokenizer.from_pretrained("google/electra-small-discriminator")
>>> model = FlaxElectraForSequenceClassification.from_pretrained("google/electra-small-discriminator")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="jax")
>>> outputs = model(**inputs)
>>> logits = outputs.logi


FlaxElectraForMultipleChoice

class transformers.FlaxElectraForMultipleChoice  < source > ( config: ElectraConfig input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True gradient_checkpointing: bool = False **kwargs  )   Parameters   config (ElectraConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    ELECTRA Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a
softmax) e.g. for RocStories/SWAG tasks. This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading, saving and converting weights from PyTorch models) This model


FlaxElectraForMultipleChoice

is also a Flax Linen
flax.nn.Module subclass. Use it as a
regular Flax Module and refer to the Flax documentation for all matter related to general usage and behavior. Finally, this model supports inherent JAX features such as: Just-In-Time (JIT) compilation Automatic Differentiation Vectorization Parallelization  __call__  < source > ( input_ids attention_mask = None token_type_ids = None position_ids = None head_mask = None encoder_hidden_states = None encoder_attention_mask = None params: dict = None dropout_rng: PRNGKey = None train: bool = False output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None past_key_values: dict = None  ) → transformers.modeling_flax_outputs.FlaxMultipleChoiceModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (numpy.ndarray of shape (batch_size, num_choices, sequence_length


FlaxElectraForMultipleChoice

)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (numpy.ndarray of shape (batch_size, num_choices, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (numpy.ndarray of shape (batch_size, num_choices, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (numpy.ndarray of shape (batch_size, num_choices, sequence_length), optional) —
Indices of positions of each input se


FlaxElectraForMultipleChoice

quence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].   head_mask (numpy.ndarray of shape (batch_size, num_choices, sequence_length), optional) -- Mask to nullify selected heads of the attention modules. Mask values selected in [0, 1]`:
1 indicates the head is not masked,
0 indicates the head is masked.
   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_flax_outputs.FlaxMultipleChoiceModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_flax_outputs.FlaxMultipleChoiceModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (ElectraConfig) and inputs.
logits (jnp.ndarray of shape (batch_size, num_choices)) — num_choices is the second dimension of the


FlaxElectraForMultipleChoice

 input tensors. (see input_ids above).
Classification scores (before SoftMax).
hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The FlaxElectraPreTrainedModel forward method, overrides the __call__ special method. Although the re


FlaxElectraForMultipleChoice

cipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, FlaxElectraForMultipleChoice
>>> tokenizer = AutoTokenizer.from_pretrained("google/electra-small-discriminator")
>>> model = FlaxElectraForMultipleChoice.from_pretrained("google/electra-small-discriminator")
>>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
>>> choice0 = "It is eaten with a fork and a knife."
>>> choice1 = "It is eaten while held in the hand."
>>> encoding = tokenizer([prompt, prompt], [choice0, choice1], return_tensors="jax", padding=True)
>>> outputs = model(**{k: v[None, :] for k, v in encoding.items()})
>>> logits = outputs.logi


FlaxElectraForTokenClassification

class transformers.FlaxElectraForTokenClassification  < source > ( config: ElectraConfig input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True gradient_checkpointing: bool = False **kwargs  )   Parameters   config (ElectraConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Electra model with a token classification head on top. Both the discriminator and generator may be loaded into this model. This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading, saving and conver


FlaxElectraForTokenClassification

ting weights from PyTorch models) This model is also a Flax Linen
flax.nn.Module subclass. Use it as a
regular Flax Module and refer to the Flax documentation for all matter related to general usage and behavior. Finally, this model supports inherent JAX features such as: Just-In-Time (JIT) compilation Automatic Differentiation Vectorization Parallelization  __call__  < source > ( input_ids attention_mask = None token_type_ids = None position_ids = None head_mask = None encoder_hidden_states = None encoder_attention_mask = None params: dict = None dropout_rng: PRNGKey = None train: bool = False output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None past_key_values: dict = None  ) → transformers.modeling_flax_outputs.FlaxTokenClassifierOutput or t


FlaxElectraForTokenClassification

uple(torch.FloatTensor)  Parameters   input_ids (numpy.ndarray of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are to


FlaxElectraForTokenClassification

ken type IDs?   position_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].   head_mask (numpy.ndarray of shape (batch_size, sequence_length), optional) -- Mask to nullify selected heads of the attention modules. Mask values selected in [0, 1]`:
1 indicates the head is not masked,
0 indicates the head is masked.
   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_flax_outputs.FlaxTokenClassifierOutput or tuple(torch.FloatTensor)

A transformers.modeling_flax_outputs.FlaxTokenClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) com


FlaxElectraForTokenClassification

prising various
elements depending on the configuration (ElectraConfig) and inputs.
logits (jnp.ndarray of shape (batch_size, sequence_length, config.num_labels)) — Classification scores (before SoftMax).
hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights


FlaxElectraForTokenClassification

after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The FlaxElectraPreTrainedModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, FlaxElectraForTokenClassification
>>> tokenizer = AutoTokenizer.from_pretrained("google/electra-small-discriminator")
>>> model = FlaxElectraForTokenClassification.from_pretrained("google/electra-small-discriminator")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="jax")
>>> outputs = model(**inputs)
>>> logits = outputs.lo


FlaxElectraForQuestionAnswering

class transformers.FlaxElectraForQuestionAnswering  < source > ( config: ElectraConfig input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True gradient_checkpointing: bool = False **kwargs  )   Parameters   config (ElectraConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    ELECTRA Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear
layers on top of the hidden-states output to compute span start logits and span end logits). This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such a


FlaxElectraForQuestionAnswering

s downloading, saving and converting weights from PyTorch models) This model is also a Flax Linen
flax.nn.Module subclass. Use it as a
regular Flax Module and refer to the Flax documentation for all matter related to general usage and behavior. Finally, this model supports inherent JAX features such as: Just-In-Time (JIT) compilation Automatic Differentiation Vectorization Parallelization  __call__  < source > ( input_ids attention_mask = None token_type_ids = None position_ids = None head_mask = None encoder_hidden_states = None encoder_attention_mask = None params: dict = None dropout_rng: PRNGKey = None train: bool = False output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None past_key_values: dict = None  ) → transformers.modeling_flax_outputs.FlaxQuestionAnsweringModelOutput or tuple(torch.


FlaxElectraForQuestionAnswering

FloatTensor)  Parameters   input_ids (numpy.ndarray of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (numpy.ndarray of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (numpy.ndarray of shape (batch_s


FlaxElectraForQuestionAnswering

ize, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].   head_mask (numpy.ndarray of shape (batch_size, sequence_length), optional) -- Mask to nullify selected heads of the attention modules. Mask values selected in [0, 1]`:
1 indicates the head is not masked,
0 indicates the head is masked.
   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_flax_outputs.FlaxQuestionAnsweringModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_flax_outputs.FlaxQuestionAnsweringModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (ElectraConfig) and inputs.
start_logits


FlaxElectraForQuestionAnswering

(jnp.ndarray of shape (batch_size, sequence_length)) — Span-start scores (before SoftMax).
end_logits (jnp.ndarray of shape (batch_size, sequence_length)) — Span-end scores (before SoftMax).
hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted averag


FlaxElectraForQuestionAnswering

e in the self-attention
heads.
  The FlaxElectraPreTrainedModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, FlaxElectraForQuestionAnswering
>>> tokenizer = AutoTokenizer.from_pretrained("google/electra-small-discriminator")
>>> model = FlaxElectraForQuestionAnswering.from_pretrained("google/electra-small-discriminator")
>>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
>>> inputs = tokenizer(question, text, return_tensors="jax")
>>> outputs = model(**inputs)
>>> start_scores = outputs.start_logits
>>> end_scores = outputs.end_logits


Overview

The X-CLIP model was proposed in Expanding Language-Image Pretrained Models for General Video Recognition by Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, Haibin Ling.
X-CLIP is a minimal extension of CLIP for video. The model consists of a text encoder, a cross-frame vision encoder, a multi-frame integration Transformer, and a video-specific prompt generator.
The abstract from the paper is the following:
Contrastive language-image pretraining has shown great success in learning visual-textual joint representation from web-scale data, demonstrating remarkable “zero-shot” generalization ability for various image tasks. However, how to effectively expand such new language-imag


Overview

e pretraining methods to video domains is still an open problem. In this work, we present a simple yet effective approach that adapts the pretrained language-image models to video recognition directly, instead of pretraining a new model from scratch. More concretely, to capture the long-range dependencies of frames along the temporal dimension, we propose a cross-frame attention mechanism that explicitly exchanges information across frames. Such module is lightweight and can be plugged into pretrained language-image models seamlessly. Moreover, we propose a video-specific prompting scheme, which leverages video content information for generating discriminative textual prompts. Extensive experiments demonstrate that our app


Overview

roach is effective and can be generalized to different video recognition scenarios. In particular, under fully-supervised settings, our approach achieves a top-1 accuracy of 87.1% on Kinectics-400, while using 12 times fewer FLOPs compared with Swin-L and ViViT-H. In zero-shot experiments, our approach surpasses the current state-of-the-art methods by +7.6% and +14.9% in terms of top-1 accuracy under two popular protocols. In few-shot scenarios, our approach outperforms previous best methods by +32.1% and +23.1% when the labeled data is extremely limited.
Tips:
Usage of X-CLIP is identical to CLIP.
X-CLIP architecture. Taken from the original paper.
This model was contributed by nielsr.
The original code can be found here.


Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with X-CLIP.
Demo notebooks for X-CLIP can be found here.
If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we’ll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


XCLIPProcessor

class transformers.XCLIPProcessor  < source > ( image_processor = None tokenizer = None **kwargs  )   Parameters   image_processor (VideoMAEImageProcessor, optional) —
The image processor is a required input.   tokenizer (CLIPTokenizerFast, optional) —
The tokenizer is a required input.    Constructs an X-CLIP processor which wraps a VideoMAE image processor and a CLIP tokenizer into a single processor. XCLIPProcessor offers all the functionalities of VideoMAEImageProcessor and CLIPTokenizerFast. See the
__call__() and decode() for more information.  batch_decode  < source > ( *args **kwargs  )    This method forwards all its arguments to CLIPTokenizerFast’s batch_decode(). Please
refer to the docstring of this method for more information.  decode  < source > ( *args **kwargs  )    This method forwards all its arguments to CLIPTokenizerFast’s decode(). Please refer to
the docstring of this method for more information.


XCLIPConfig

class transformers.XCLIPConfig  < source > ( text_config = None vision_config = None projection_dim = 512 prompt_layers = 2 prompt_alpha = 0.1 prompt_hidden_act = 'quick_gelu' prompt_num_attention_heads = 8 prompt_attention_dropout = 0.0 prompt_projection_dropout = 0.0 logit_scale_init_value = 2.6592 **kwargs  )   Parameters   text_config (dict, optional) —
Dictionary of configuration options used to initialize XCLIPTextConfig.   vision_config (dict, optional) —
Dictionary of configuration options used to initialize XCLIPVisionConfig.   projection_dim (int, optional, defaults to 512) —
Dimentionality of text and vision projection layers.   prompt_layers (int, optional, defaults to 2) —
Number of layers in the video specific prompt generator.   prompt_alpha (float, optional, defaults to 0.1) —
Alpha value to use in the video specific prompt generator.   prompt_hidden


XCLIPConfig

_act (str or function, optional, defaults to "quick_gelu") —
The non-linear activation function (function or string) in the video specific prompt generator. If string,
"gelu", "relu", "selu" and "gelu_new" `"quick_gelu" are supported.   prompt_num_attention_heads (int, optional, defaults to 8) —
Number of attention heads in the cross-attention of the video specific prompt generator.   prompt_attention_dropout (float, optional, defaults to 0.0) —
The dropout probability for the attention layers in the video specific prompt generator.   prompt_projection_dropout (float, optional, defaults to 0.0) —
The dropout probability for the projection layers in the video specific prompt generator.   logit_scale_init_value (float, optional, defaults to 2.6592) —
The inital value of the logit_scale parameter. Default is used as per the original XCLIP implementation.   kwargs (opti


XCLIPConfig

onal) —
Dictionary of keyword arguments.    XCLIPConfig is the configuration class to store the configuration of a XCLIPModel. It is used to
instantiate X-CLIP model according to the specified arguments, defining the text model and vision model configs.
Instantiating a configuration with the defaults will yield a similar configuration to that of the X-CLIP
microsoft/xclip-base-patch32 architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  from_text_vision_configs  < source > ( text_config: XCLIPTextConfig vision_config: XCLIPVisionConfig **kwargs  ) → XCLIPConfig  Returns
XCLIPConfig

An instance of a configuration object
  Instantiate a XCLIPConfig (or a derived class) from xclip text model configuration and xclip vision model
configuratio


XCLIPTextConfig

class transformers.XCLIPTextConfig  < source > ( vocab_size = 49408 hidden_size = 512 intermediate_size = 2048 num_hidden_layers = 12 num_attention_heads = 8 max_position_embeddings = 77 hidden_act = 'quick_gelu' layer_norm_eps = 1e-05 attention_dropout = 0.0 initializer_range = 0.02 initializer_factor = 1.0 pad_token_id = 1 bos_token_id = 0 eos_token_id = 2 **kwargs  )   Parameters   vocab_size (int, optional, defaults to 49408) —
Vocabulary size of the X-CLIP text model. Defines the number of different tokens that can be represented by
the inputs_ids passed when calling XCLIPModel.   hidden_size (int, optional, defaults to 512) —
Dimensionality of the encoder layers and the pooler layer.   intermediate_size (int, optional, defaults to 2048) —
Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.   num_hidden_layers (int, optional, defaults to 12) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defau


XCLIPTextConfig

lts to 8) —
Number of attention heads for each attention layer in the Transformer encoder.   max_position_embeddings (int, optional, defaults to 77) —
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).   hidden_act (str or function, optional, defaults to "quick_gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "selu" and "gelu_new" `"quick_gelu" are supported.   layer_norm_eps (float, optional, defaults to 1e-5) —
The epsilon used by the layer normalization layers.   attention_dropout (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   initializer_factor (`float“, optional, defaults to 1) —
A factor for initializing all weight m


XCLIPTextConfig

atrices (should be kept to 1, used internally for initialization
testing).    This is the configuration class to store the configuration of a XCLIPModel. It is used to instantiate an X-CLIP
model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
defaults will yield a similar configuration to that of the X-CLIP
microsoft/xclip-base-patch32 architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import XCLIPTextModel, XCLIPTextConfig
>>> # Initializing a XCLIPTextModel with microsoft/xclip-base-patch32 style configuration
>>> configuration = XCLIPTextConfig()
>>> # Initializing a XCLIPTextConfig from the microsoft/xclip-base-patch32 style configuration
>>> model = XCLIPTextModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.confi


XCLIPVisionConfig

class transformers.XCLIPVisionConfig  < source > ( hidden_size = 768 intermediate_size = 3072 num_hidden_layers = 12 num_attention_heads = 12 mit_hidden_size = 512 mit_intermediate_size = 2048 mit_num_hidden_layers = 1 mit_num_attention_heads = 8 num_channels = 3 image_size = 224 patch_size = 32 num_frames = 8 hidden_act = 'quick_gelu' layer_norm_eps = 1e-05 attention_dropout = 0.0 initializer_range = 0.02 initializer_factor = 1.0 drop_path_rate = 0.0 **kwargs  )   Parameters   hidden_size (int, optional, defaults to 768) —
Dimensionality of the encoder layers and the pooler layer.   intermediate_size (int, optional, defaults to 3072) —
Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.   num_hidden_layers (int, optional, defaults to 12) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, def


XCLIPVisionConfig

aults to 12) —
Number of attention heads for each attention layer in the Transformer encoder.   mit_hidden_size (int, optional, defaults to 512) —
Dimensionality of the encoder layers of the Multiframe Integration Transformer (MIT).   mit_intermediate_size (int, optional, defaults to 2048) —
Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Multiframe Integration Transformer
(MIT).   mit_num_hidden_layers (int, optional, defaults to 1) —
Number of hidden layers in the Multiframe Integration Transformer (MIT).   mit_num_attention_heads (int, optional, defaults to 8) —
Number of attention heads for each attention layer in the Multiframe Integration Transformer (MIT).   image_size (int, optional, defaults to 224) —
The size (resolution) of each image.   patch_size (int, optional, defaults to 32) —
The size (resolution) of each patch.   hidden_act (str or


XCLIPVisionConfig

 function, optional, defaults to "quick_gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "selu", "gelu_new" and `"quick_gelu" are supported.   layer_norm_eps (float, optional, defaults to 1e-5) —
The epsilon used by the layer normalization layers.   attention_dropout (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   initializer_factor (`float“, optional, defaults to 1) —
A factor for initializing all weight matrices (should be kept to 1, used internally for initialization
testing).   drop_path_rate (float, optional, defaults to 0.0) —
Stochastic depth rate.    This is the configuration class to store the configu


XCLIPVisionConfig

ration of a XCLIPModel. It is used to instantiate an X-CLIP
model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
defaults will yield a similar configuration to that of the X-CLIP
microsoft/xclip-base-patch32 architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import XCLIPVisionModel, XCLIPVisionConfig
>>> # Initializing a XCLIPVisionModel with microsoft/xclip-base-patch32 style configuration
>>> configuration = XCLIPVisionConfig()
>>> # Initializing a XCLIPVisionModel model from the microsoft/xclip-base-patch32 style configuration
>>> model = XCLIPVisionModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config


XCLIPModel

class transformers.XCLIPModel  < source > ( config: XCLIPConfig  )   Parameters   config (XCLIPConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    This model is a PyTorch torch.nn.Module subclass. Use it
as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None pixel_values: typing.Optional[torch.FloatTensor] = None attention_mask: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.LongTensor] = None return_loss: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.x_clip.modeling_x_c