XLNet specific outputs

ned when labels is provided)  —
Classification loss.   logits (tf.Tensor of shape (batch_size, sequence_length, config.num_labels)) —
Classification scores (before SoftMax).   mems (List[tf.Tensor] of length config.n_layers) —
Contains pre-computed hidden-states. Can be used (see mems input) to speed up sequential decoding. The
token ids which have their past given to this model should not be passed as input_ids as they have
already been computed.   hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.   attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) —
Tuple of tf.Tensor (one for each layer


XLNet specific outputs

) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.    Output type of TFXLNetForTokenClassificationOutput.
 class transformers.models.xlnet.modeling_tf_xlnet.TFXLNetForQuestionAnsweringSimpleOutput  < source > ( loss: tf.Tensor | None = None start_logits: tf.Tensor = None end_logits: tf.Tensor = None mems: List[tf.Tensor] | None = None hidden_states: Tuple[tf.Tensor] | None = None attentions: Tuple[tf.Tensor] | None = None  )   Parameters   loss (tf.Tensor of shape (1,), optional, returned when labels is provided) —
Total span extraction loss is the sum of a Cross-Entropy for the start and end positions.   start_logits (tf.Tensor of shape (batch_size, sequence_length,)) —
Span-start scores (before SoftMax).   end_logits (tf.Tensor of shape (batch_size, sequence_length,)) —
Span-end scores (before SoftMax).   mems (List[tf.Tensor] of length config.n_lay


XLNet specific outputs

ers) —
Contains pre-computed hidden-states. Can be used (see mems input) to speed up sequential decoding. The
token ids which have their past given to this model should not be passed as input_ids as they have
already been computed.   hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.   attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) —
Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.    Output type of TFXLNetForQuestion


XLNetModel

class transformers.XLNetModel  < source > ( config  )   Parameters   config (XLNetConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare XLNet Model transformer outputting raw hidden-states without any specific head on top. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask:


XLNetModel

 typing.Optional[torch.Tensor] = None mems: typing.Optional[torch.Tensor] = None perm_mask: typing.Optional[torch.Tensor] = None target_mapping: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None input_mask: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None use_mems: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None **kwargs  ) → transformers.models.xlnet.modeling_xlnet.XLNetModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
Wh


XLNetModel

at are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   mems (List[torch.FloatTensor] of length config.n_layers) —
Contains pre-computed hidden-states (see mems output below) . Can be used to speed up sequential
decoding. The token ids which have their past given to this model should not be passed as input_ids as
they have already been computed.
use_mems has to be set to True to make use of mems.   perm_mask (torch.FloatTensor of shape (batch_size, sequence_length, sequence_length), optional) —
Mask to indicate the attention pattern for each input token with values selected in [0, 1]:
if perm_mask[k, i, j] = 0, i attend to j in batch k;
if perm_mask[k, i, j] = 1, i does not attend


XLNetModel

 to j in batch k.
If not set, each token attends to all the others (full bidirectional attention). Only used during
pretraining (to define factorization order) or for sequential decoding (generation).   target_mapping (torch.FloatTensor of shape (batch_size, num_predict, sequence_length), optional) —
Mask to indicate the output tokens to use. If target_mapping[k, i, j] = 1, the i-th predict in batch k is
on the j-th token. Only used during pretraining for partial prediction or for sequential decoding
(generation).   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   input_mask (torch.FloatTensor of shape batch_size, sequence_length, optional) —
Mask to avoid performing atte


XLNetModel

ntion on padding token indices. Negative of attention_mask, i.e. with 0 for
real tokens and 1 for padding which is kept for compatibility with the original code base.
Mask values selected in [0, 1]:
1 for tokens that are masked,
0 for tokens that are not masked.
You can only uses one of input_mask and attention_mask.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup mat


XLNetModel

rix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.models.xlnet.modeling_xlnet.XLNetModelOutput or tuple(torch.FloatTensor)

A transformers.models.xlnet.modeling_xlnet.XLNetModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (XLNetConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, num_predict, hidden_size)) — Sequence of hidden-states at the last layer of the model.
num_pre


XLNetModel

dict corresponds to target_mapping.shape[1]. If target_mapping is None, then num_predict
corresponds to sequence_length.
mems (List[torch.FloatTensor] of length config.n_layers) — Contains pre-computed hidden-states. Can be used (see mems input) to speed up sequential decoding. The
token ids which have their past given to this model should not be passed as input_ids as they have
already been computed.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=T


XLNetModel

rue) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The XLNetModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, XLNetModel
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("xlnet-base-cased")
>>> model = XLNetModel.from_pretrained("xlnet-base-cased")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_


XLNetLMHeadModel

class transformers.XLNetLMHeadModel  < source > ( config  )   Parameters   config (XLNetConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    XLNet Model with a language modeling head on top (linear layer with weights tied to the input embeddings). This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: ty


XLNetLMHeadModel

ping.Optional[torch.Tensor] = None mems: typing.Optional[torch.Tensor] = None perm_mask: typing.Optional[torch.Tensor] = None target_mapping: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None input_mask: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None use_mems: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None **kwargs  ) → transformers.models.xlnet.modeling_xlnet.XLNetLMHeadModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedToke


XLNetLMHeadModel

nizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   mems (List[torch.FloatTensor] of length config.n_layers) —
Contains pre-computed hidden-states (see mems output below) . Can be used to speed up sequential
decoding. The token ids which have their past given to this model should not be passed as input_ids as
they have already been computed.
use_mems has to be set to True to make use of mems.   perm_mask (torch.FloatTensor of shape (batch_size, sequence_length, sequence_length), optional) —
Mask to indicate the attention pattern for each input token with values selected in [0, 1]:
if perm_mask[k, i, j] = 0, i attend to j in batch k;
if perm_mask[k, i, j] = 1, i does no


XLNetLMHeadModel

t attend to j in batch k.
If not set, each token attends to all the others (full bidirectional attention). Only used during
pretraining (to define factorization order) or for sequential decoding (generation).   target_mapping (torch.FloatTensor of shape (batch_size, num_predict, sequence_length), optional) —
Mask to indicate the output tokens to use. If target_mapping[k, i, j] = 1, the i-th predict in batch k is
on the j-th token. Only used during pretraining for partial prediction or for sequential decoding
(generation).   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   input_mask (torch.FloatTensor of shape batch_size, sequence_length, optional) —
Mask to avoid performing attention on pad


XLNetLMHeadModel

ding token indices. Negative of attention_mask, i.e. with 0 for
real tokens and 1 for padding which is kept for compatibility with the original code base.
Mask values selected in [0, 1]:
1 for tokens that are masked,
0 for tokens that are not masked.
You can only uses one of input_mask and attention_mask.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool,


XLNetLMHeadModel

optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size, num_predict), optional) —
Labels for masked language modeling. num_predict corresponds to target_mapping.shape[1]. If
target_mapping is None, then num_predict corresponds to sequence_length.
The labels should correspond to the masked input words that should be predicted and depends on
target_mapping. Note in order to perform standard auto-regressive language modeling a  token has
to be added to the input_ids (see the prepare_inputs_for_generation function and examples below)
Indices are selecte


XLNetLMHeadModel

d in [-100, 0, ..., config.vocab_size] All labels set to -100 are ignored, the loss
is only computed for labels in [0, ..., config.vocab_size]   Returns
transformers.models.xlnet.modeling_xlnet.XLNetLMHeadModelOutput or tuple(torch.FloatTensor)

A transformers.models.xlnet.modeling_xlnet.XLNetLMHeadModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (XLNetConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided)
Language modeling loss (for next-token prediction).
logits (torch.FloatTensor of shape (batch_size, num_predict, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
num_predict corresponds to target_mapping.shape[1]. If target_mapping is None, then num_predict
corresponds to sequence_length.
m


XLNetLMHeadModel

ems (List[torch.FloatTensor] of length config.n_layers) — Contains pre-computed hidden-states. Can be used (see mems input) to speed up sequential decoding. The
token ids which have their past given to this model should not be passed as input_ids as they have
already been computed.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights


XLNetLMHeadModel

after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The XLNetLMHeadModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoTokenizer, XLNetLMHeadModel
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("xlnet-large-cased")
>>> model = XLNetLMHeadModel.from_pretrained("xlnet-large-cased")
>>> # We show how to setup inputs to predict a next token using a bi-directional context.
>>> input_ids = torch.tensor(
...     tokenizer.encode("Hello, my dog is very <mask>", add_special_tokens=False)
... ).unsqueeze(
...     0
... )  # We will predict the masked token
>>> perm_mask


XLNetLMHeadModel

 = torch.zeros((1, input_ids.shape[1], input_ids.shape[1]), dtype=torch.float)
>>> perm_mask[:, :, -1] = 1.0  # Previous tokens don't see last token
>>> target_mapping = torch.zeros(
...     (1, 1, input_ids.shape[1]), dtype=torch.float
... )  # Shape [1, 1, seq_length] => let's predict one token
>>> target_mapping[
...     0, 0, -1
... ] = 1.0  # Our first (and only) prediction will be the last token of the sequence (the masked token)
>>> outputs = model(input_ids, perm_mask=perm_mask, target_mapping=target_mapping)
>>> next_token_logits = outputs[
...     0
... ]  # Output has shape [target_mapping.size(0), target_mapping.size(1), config.vocab_size]
>>> # The same way can the XLNetLMHeadModel be used to be trained by standard auto-regressive language modeling.
>>> input_ids = torch.tensor(
...     tokenizer.encode("Hello, my dog is very <mask>", add_special_tokens=False)
... ).unsqueeze(
...     0
... )  # We will predict the ma


XLNetLMHeadModel

sked token
>>> labels = torch.tensor(tokenizer.encode("cute", add_special_tokens=False)).unsqueeze(0)
>>> assert labels.shape[0] == 1, "only one word will be predicted"
>>> perm_mask = torch.zeros((1, input_ids.shape[1], input_ids.shape[1]), dtype=torch.float)
>>> perm_mask[
...     :, :, -1
... ] = 1.0  # Previous tokens don't see last token as is done in standard auto-regressive lm training
>>> target_mapping = torch.zeros(
...     (1, 1, input_ids.shape[1]), dtype=torch.float
... )  # Shape [1, 1, seq_length] => let's predict one token
>>> target_mapping[
...     0, 0, -1
... ] = 1.0  # Our first (and only) prediction will be the last token of the sequence (the masked token)
>>> outputs = model(input_ids, perm_mask=perm_mask, target_mapping=target_mapping, labels=labels)
>>> loss = outputs.loss
>>> next_token_logits = (
...     outputs.logits
... )  # Logits have shape [target_mapping.size(0), target_mapping.size(1), config.voc


XLNetForSequenceClassification

class transformers.XLNetForSequenceClassification  < source > ( config  )   Parameters   config (XLNetConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    XLNet Model with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g.
for GLUE tasks. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[t


XLNetForSequenceClassification

orch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None mems: typing.Optional[torch.Tensor] = None perm_mask: typing.Optional[torch.Tensor] = None target_mapping: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None input_mask: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None use_mems: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None **kwargs  ) → transformers.models.xlnet.modeling_xlnet.XLNetForSequenceClassificationOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer.


XLNetForSequenceClassification

See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   mems (List[torch.FloatTensor] of length config.n_layers) —
Contains pre-computed hidden-states (see mems output below) . Can be used to speed up sequential
decoding. The token ids which have their past given to this model should not be passed as input_ids as
they have already been computed.
use_mems has to be set to True to make use of mems.   perm_mask (torch.FloatTensor of shape (batch_size, sequence_length, sequence_length), optional) —
Mask to indicate the attention pattern for each input token with values selected in [0, 1]:
if perm_mask[k, i, j] = 0, i attend to


XLNetForSequenceClassification

j in batch k;
if perm_mask[k, i, j] = 1, i does not attend to j in batch k.
If not set, each token attends to all the others (full bidirectional attention). Only used during
pretraining (to define factorization order) or for sequential decoding (generation).   target_mapping (torch.FloatTensor of shape (batch_size, num_predict, sequence_length), optional) —
Mask to indicate the output tokens to use. If target_mapping[k, i, j] = 1, the i-th predict in batch k is
on the j-th token. Only used during pretraining for partial prediction or for sequential decoding
(generation).   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   input_mask (torch.FloatTensor of shape batch_size, sequence_length, opti


XLNetForSequenceClassification

onal) —
Mask to avoid performing attention on padding token indices. Negative of attention_mask, i.e. with 0 for
real tokens and 1 for padding which is kept for compatibility with the original code base.
Mask values selected in [0, 1]:
1 for tokens that are masked,
0 for tokens that are not masked.
You can only uses one of input_mask and attention_mask.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embe


XLNetForSequenceClassification

dding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-Entropy).   Returns
transformers.models.xlnet.modeling_xlnet.XLNetForSequenceClassificationOutput or tuple(torch.FloatTensor)

A transformers.models.xlnet.modeling_xlnet.XLN


XLNetForSequenceClassification

etForSequenceClassificationOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (XLNetConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when label is provided) — Classification (or regression if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
mems (List[torch.FloatTensor] of length config.n_layers) — Contains pre-computed hidden-states. Can be used (see mems input) to speed up sequential decoding. The
token ids which have their past given to this model should not be passed as input_ids as they have
already been computed.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tupl


XLNetForSequenceClassification

e of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The XLNetForSequenceClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignor


XLNetForSequenceClassification

es them.  Example of single-label classification:   Copied >>> import torch
>>> from transformers import AutoTokenizer, XLNetForSequenceClassification
>>> tokenizer = AutoTokenizer.from_pretrained("xlnet-base-cased")
>>> model = XLNetForSequenceClassification.from_pretrained("xlnet-base-cased")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_class_id = logits.argmax().item()
>>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`
>>> num_labels = len(model.config.id2label)
>>> model = XLNetForSequenceClassification.from_pretrained("xlnet-base-cased", num_labels=num_labels)
>>> labels = torch.tensor([1])
>>> loss = model(**inputs, labels=labels).loss  Example of multi-label classification:   Copied >>> import torch
>>> from transformers import AutoTokenizer, XLNetForSequenceClas


XLNetForSequenceClassification

sification
>>> tokenizer = AutoTokenizer.from_pretrained("xlnet-base-cased")
>>> model = XLNetForSequenceClassification.from_pretrained("xlnet-base-cased", problem_type="multi_label_classification")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_class_ids = torch.arange(0, logits.shape[-1])[torch.sigmoid(logits).squeeze(dim=0) > 0.5]
>>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`
>>> num_labels = len(model.config.id2label)
>>> model = XLNetForSequenceClassification.from_pretrained(
...     "xlnet-base-cased", num_labels=num_labels, problem_type="multi_label_classification"
... )
>>> labels = torch.sum(
...     torch.nn.functional.one_hot(predicted_class_ids[None, :].clone(), num_classes=num_labels), dim=1
... ).to(torch.float)
>>> loss = model(**inputs, labels=labels)


XLNetForMultipleChoice

class transformers.XLNetForMultipleChoice  < source > ( config  )   Parameters   config (XLNetConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    XLNet Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a
softmax) e.g. for RACE/SWAG tasks. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids


XLNetForMultipleChoice

: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None input_mask: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None mems: typing.Optional[torch.Tensor] = None perm_mask: typing.Optional[torch.Tensor] = None target_mapping: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None use_mems: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None **kwargs  ) → transformers.models.xlnet.modeling_xlnet.XLNetForMultipleChoiceOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, num_choices, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices c


XLNetForMultipleChoice

an be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, num_choices, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   mems (List[torch.FloatTensor] of length config.n_layers) —
Contains pre-computed hidden-states (see mems output below) . Can be used to speed up sequential
decoding. The token ids which have their past given to this model should not be passed as input_ids as
they have already been computed.
use_mems has to be set to True to make use of mems.   perm_mask (torch.FloatTensor of shape (batch_size, sequence_length, sequence_length), optional) —
Mask to indicate the attention pattern for each input token with value


XLNetForMultipleChoice

s selected in [0, 1]:
if perm_mask[k, i, j] = 0, i attend to j in batch k;
if perm_mask[k, i, j] = 1, i does not attend to j in batch k.
If not set, each token attends to all the others (full bidirectional attention). Only used during
pretraining (to define factorization order) or for sequential decoding (generation).   target_mapping (torch.FloatTensor of shape (batch_size, num_predict, sequence_length), optional) —
Mask to indicate the output tokens to use. If target_mapping[k, i, j] = 1, the i-th predict in batch k is
on the j-th token. Only used during pretraining for partial prediction or for sequential decoding
(generation).   token_type_ids (torch.LongTensor of shape (batch_size, num_choices, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are toke


XLNetForMultipleChoice

n type IDs?   input_mask (torch.FloatTensor of shape batch_size, num_choices, sequence_length, optional) —
Mask to avoid performing attention on padding token indices. Negative of attention_mask, i.e. with 0 for
real tokens and 1 for padding which is kept for compatibility with the original code base.
Mask values selected in [0, 1]:
1 for tokens that are masked,
0 for tokens that are not masked.
You can only uses one of input_mask and attention_mask.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, num_choices, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is


XLNetForMultipleChoice

useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the multiple choice classification loss. Indices should be in [0, ..., num_choices-1] where num_choices is the size of the second dimension of the input tensors. (See
input_ids above)   Returns
transformers.models.xlnet.modeling_xlnet.XLNetForMultipleChoiceOutput or tuple(torch.F


XLNetForMultipleChoice

loatTensor)

A transformers.models.xlnet.modeling_xlnet.XLNetForMultipleChoiceOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (XLNetConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification loss.
logits (torch.FloatTensor of shape (batch_size, num_choices)) — num_choices is the second dimension of the input tensors. (see input_ids above).
Classification scores (before SoftMax).
mems (List[torch.FloatTensor] of length config.n_layers) — Contains pre-computed hidden-states. Can be used (see mems input) to speed up sequential decoding. The
token ids which have their past given to this model should not be passed as input_ids as they have
already been computed.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=Tr


XLNetForMultipleChoice

ue is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The XLNetForMultipleChoice forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running


XLNetForMultipleChoice

the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, XLNetForMultipleChoice
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("xlnet-base-cased")
>>> model = XLNetForMultipleChoice.from_pretrained("xlnet-base-cased")
>>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
>>> choice0 = "It is eaten with a fork and a knife."
>>> choice1 = "It is eaten while held in the hand."
>>> labels = torch.tensor(0).unsqueeze(0)  # choice0 is correct (according to Wikipedia ;)), batch size 1
>>> encoding = tokenizer([prompt, prompt], [choice0, choice1], return_tensors="pt", padding=True)
>>> outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()}, labels=labels)  # batch size is 1
>>> # the linear classifier still needs to be trained
>>> loss = outputs.loss
>>> logits = outputs


XLNetForTokenClassification

class transformers.XLNetForTokenClassification  < source > ( config  )   Parameters   config (XLNetConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    XLNet Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for
Named-Entity-Recognition (NER) tasks. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < sour


XLNetForTokenClassification

ce > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None mems: typing.Optional[torch.Tensor] = None perm_mask: typing.Optional[torch.Tensor] = None target_mapping: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None input_mask: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None use_mems: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None **kwargs  ) → transformers.models.xlnet.modeling_xlnet.XLNetForTokenClassificationOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabu


XLNetForTokenClassification

lary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   mems (List[torch.FloatTensor] of length config.n_layers) —
Contains pre-computed hidden-states (see mems output below) . Can be used to speed up sequential
decoding. The token ids which have their past given to this model should not be passed as input_ids as
they have already been computed.
use_mems has to be set to True to make use of mems.   perm_mask (torch.FloatTensor of shape (batch_size, sequence_length, sequence_length), optional) —
Mask to indicate the attention pattern for each input token w


XLNetForTokenClassification

ith values selected in [0, 1]:
if perm_mask[k, i, j] = 0, i attend to j in batch k;
if perm_mask[k, i, j] = 1, i does not attend to j in batch k.
If not set, each token attends to all the others (full bidirectional attention). Only used during
pretraining (to define factorization order) or for sequential decoding (generation).   target_mapping (torch.FloatTensor of shape (batch_size, num_predict, sequence_length), optional) —
Mask to indicate the output tokens to use. If target_mapping[k, i, j] = 1, the i-th predict in batch k is
on the j-th token. Only used during pretraining for partial prediction or for sequential decoding
(generation).   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are t


XLNetForTokenClassification

oken type IDs?   input_mask (torch.FloatTensor of shape batch_size, sequence_length, optional) —
Mask to avoid performing attention on padding token indices. Negative of attention_mask, i.e. with 0 for
real tokens and 1 for padding which is kept for compatibility with the original code base.
Mask values selected in [0, 1]:
1 for tokens that are masked,
0 for tokens that are not masked.
You can only uses one of input_mask and attention_mask.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you wa


XLNetForTokenClassification

nt more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the multiple choice classification loss. Indices should be in [0, ..., num_choices]
where num_choices is the size of the second dimension of the input tensors. (see input_ids above)   Returns
transformers.models.xlnet.modeling_xlnet.XLNetForTokenClassificationOutput or tuple(torch.FloatTe


XLNetForTokenClassification

nsor)

A transformers.models.xlnet.modeling_xlnet.XLNetForTokenClassificationOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (XLNetConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided)  — Classification loss.
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.num_labels)) — Classification scores (before SoftMax).
mems (List[torch.FloatTensor] of length config.n_layers) — Contains pre-computed hidden-states. Can be used (see mems input) to speed up sequential decoding. The
token ids which have their past given to this model should not be passed as input_ids as they have
already been computed.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=Tr


XLNetForTokenClassification

ue) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The XLNetForTokenClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while


XLNetForTokenClassification

the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, XLNetForTokenClassification
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("xlnet-base-cased")
>>> model = XLNetForTokenClassification.from_pretrained("xlnet-base-cased")
>>> inputs = tokenizer(
...     "HuggingFace is a company based in Paris and New York", add_special_tokens=False, return_tensors="pt"
... )
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_token_class_ids = logits.argmax(-1)
>>> # Note that tokens are classified rather then input words which means that
>>> # there might be more predicted token classes than words.
>>> # Multiple token classes might account for the same word
>>> predicted_tokens_classes = [model.config.id2label[t.item()] for t in predicted_token_class_ids[0]]
>>> labels = predicted_token_class_ids
>>> loss = model(**inputs, labels=labels)


XLNetForQuestionAnsweringSimple

class transformers.XLNetForQuestionAnsweringSimple  < source > ( config  )   Parameters   config (XLNetConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    XLNet Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear
layers on top of the hidden-states output to compute span start logits and span end logits). This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch do


XLNetForQuestionAnsweringSimple

cumentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None mems: typing.Optional[torch.Tensor] = None perm_mask: typing.Optional[torch.Tensor] = None target_mapping: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None input_mask: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None start_positions: typing.Optional[torch.Tensor] = None end_positions: typing.Optional[torch.Tensor] = None use_mems: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None **kwargs  ) → transformers.models.xlnet.modeling_xlnet.XLNetForQuestionAnswering


XLNetForQuestionAnsweringSimple

SimpleOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   mems (List[torch.FloatTensor] of length config.n_layers) —
Contains pre-computed hidden-states (see mems output below) . Can be used to speed up sequential
decoding. The token ids which have their past given to this model should not be passed as input_ids as
they have already been computed.
use_mems has to be


XLNetForQuestionAnsweringSimple

 set to True to make use of mems.   perm_mask (torch.FloatTensor of shape (batch_size, sequence_length, sequence_length), optional) —
Mask to indicate the attention pattern for each input token with values selected in [0, 1]:
if perm_mask[k, i, j] = 0, i attend to j in batch k;
if perm_mask[k, i, j] = 1, i does not attend to j in batch k.
If not set, each token attends to all the others (full bidirectional attention). Only used during
pretraining (to define factorization order) or for sequential decoding (generation).   target_mapping (torch.FloatTensor of shape (batch_size, num_predict, sequence_length), optional) —
Mask to indicate the output tokens to use. If target_mapping[k, i, j] = 1, the i-th predict in batch k is
on the j-th token. Only used during pretraining for partial prediction or for sequential decoding
(generation).   token_type_ids (torch.LongTensor of shape (batch_size, se


XLNetForQuestionAnsweringSimple

quence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   input_mask (torch.FloatTensor of shape batch_size, sequence_length, optional) —
Mask to avoid performing attention on padding token indices. Negative of attention_mask, i.e. with 0 for
real tokens and 1 for padding which is kept for compatibility with the original code base.
Mask values selected in [0, 1]:
1 for tokens that are masked,
0 for tokens that are not masked.
You can only uses one of input_mask and attention_mask.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the


XLNetForQuestionAnsweringSimple

 head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   start_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the


XLNetForQuestionAnsweringSimple

start of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
are not taken into account for computing the loss.   end_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the end of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
are not taken into account for computing the loss.   Returns
transformers.models.xlnet.modeling_xlnet.XLNetForQuestionAnsweringSimpleOutput or tuple(torch.FloatTensor)

A transformers.models.xlnet.modeling_xlnet.XLNetForQuestionAnsweringSimpleOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the


XLNetForQuestionAnsweringSimple

configuration (XLNetConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Total span extraction loss is the sum of a Cross-Entropy for the start and end positions.
start_logits (torch.FloatTensor of shape (batch_size, sequence_length,)) — Span-start scores (before SoftMax).
end_logits (torch.FloatTensor of shape (batch_size, sequence_length,)) — Span-end scores (before SoftMax).
mems (List[torch.FloatTensor] of length config.n_layers) — Contains pre-computed hidden-states. Can be used (see mems input) to speed up sequential decoding. The
token ids which have their past given to this model should not be passed as input_ids as they have
already been computed.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the


XLNetForQuestionAnsweringSimple

output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The XLNetForQuestionAnsweringSimple forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silent


XLNetForQuestionAnsweringSimple

ly ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, XLNetForQuestionAnsweringSimple
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("xlnet-base-cased")
>>> model = XLNetForQuestionAnsweringSimple.from_pretrained("xlnet-base-cased")
>>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
>>> inputs = tokenizer(question, text, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> answer_start_index = outputs.start_logits.argmax()
>>> answer_end_index = outputs.end_logits.argmax()
>>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
>>> # target is "nice puppet"
>>> target_start_index = torch.tensor([14])
>>> target_end_index = torch.tensor([15])
>>> outputs = model(**inputs, start_positions=target_start_index, end_positions=target_end_index)
>>> loss = outputs.lo


XLNetForQuestionAnswering

class transformers.XLNetForQuestionAnswering  < source > ( config  )   Parameters   config (XLNetConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    XLNet Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear
layers on top of the hidden-states output to compute span start logits and span end logits). This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usag


XLNetForQuestionAnswering

e
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None mems: typing.Optional[torch.Tensor] = None perm_mask: typing.Optional[torch.Tensor] = None target_mapping: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None input_mask: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None start_positions: typing.Optional[torch.Tensor] = None end_positions: typing.Optional[torch.Tensor] = None is_impossible: typing.Optional[torch.Tensor] = None cls_index: typing.Optional[torch.Tensor] = None p_mask: typing.Optional[torch.Tensor] = None use_mems: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None **kwargs  ) → transformers.mo


XLNetForQuestionAnswering

dels.xlnet.modeling_xlnet.XLNetForQuestionAnsweringOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   mems (List[torch.FloatTensor] of length config.n_layers) —
Contains pre-computed hidden-states (see mems output below) . Can be used to speed up sequential
decoding. The token ids which have their past given to this model should not be passed as input_ids as
they have already been computed.
use_mems has to b


XLNetForQuestionAnswering

e set to True to make use of mems.   perm_mask (torch.FloatTensor of shape (batch_size, sequence_length, sequence_length), optional) —
Mask to indicate the attention pattern for each input token with values selected in [0, 1]:
if perm_mask[k, i, j] = 0, i attend to j in batch k;
if perm_mask[k, i, j] = 1, i does not attend to j in batch k.
If not set, each token attends to all the others (full bidirectional attention). Only used during
pretraining (to define factorization order) or for sequential decoding (generation).   target_mapping (torch.FloatTensor of shape (batch_size, num_predict, sequence_length), optional) —
Mask to indicate the output tokens to use. If target_mapping[k, i, j] = 1, the i-th predict in batch k is
on the j-th token. Only used during pretraining for partial prediction or for sequential decoding
(generation).   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token i


XLNetForQuestionAnswering

ndices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   input_mask (torch.FloatTensor of shape batch_size, sequence_length, optional) —
Mask to avoid performing attention on padding token indices. Negative of attention_mask, i.e. with 0 for
real tokens and 1 for padding which is kept for compatibility with the original code base.
Mask values selected in [0, 1]:
1 for tokens that are masked,
0 for tokens that are not masked.
You can only uses one of input_mask and attention_mask.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_len


XLNetForQuestionAnswering

gth, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   start_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the start of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequen


XLNetForQuestionAnswering

ce_length). Position outside of the sequence
are not taken into account for computing the loss.   end_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the end of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
are not taken into account for computing the loss.   is_impossible (torch.LongTensor of shape (batch_size,), optional) —
Labels whether a question has an answer or no answer (SQuAD 2.0)   cls_index (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the classification token to use as input for computing plausibility of the
answer.   p_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Optional mask of tokens which can’t be in answers (e.g. [CLS], [PAD], …). 1.0 means token should be
masked. 0.0 mean token is


XLNetForQuestionAnswering

 not masked.   Returns
transformers.models.xlnet.modeling_xlnet.XLNetForQuestionAnsweringOutput or tuple(torch.FloatTensor)

A transformers.models.xlnet.modeling_xlnet.XLNetForQuestionAnsweringOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (XLNetConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned if both start_positions and end_positions are provided) — Classification loss as the sum of start token, end token (and is_impossible if provided) classification
losses.
start_top_log_probs (torch.FloatTensor of shape (batch_size, config.start_n_top), optional, returned if start_positions or end_positions is not provided) — Log probabilities for the top config.start_n_top start token possibilities (beam-search).
start_top_index (torch.LongTensor of shape (batch_size, config.start_n_top), optional,


XLNetForQuestionAnswering

 returned if start_positions or end_positions is not provided) — Indices for the top config.start_n_top start token possibilities (beam-search).
end_top_log_probs (torch.FloatTensor of shape (batch_size, config.start_n_top * config.end_n_top), optional, returned if start_positions or end_positions is not provided) — Log probabilities for the top config.start_n_top * config.end_n_top end token possibilities
(beam-search).
end_top_index (torch.LongTensor of shape (batch_size, config.start_n_top * config.end_n_top), optional, returned if start_positions or end_positions is not provided) — Indices for the top config.start_n_top * config.end_n_top end token possibilities (beam-search).
cls_logits (torch.FloatTensor of shape (batch_size,), optional, returned if start_positions or end_positions is not provided) — Log probabilities for the is_impossible label of the answers.
mems (List[torch.FloatTensor] of length config.n_layers) — Contain


XLNetForQuestionAnswering

s pre-computed hidden-states. Can be used (see mems input) to speed up sequential decoding. The
token ids which have their past given to this model should not be passed as input_ids as they have
already been computed.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average i


XLNetForQuestionAnswering

n the self-attention
heads.
  The XLNetForQuestionAnswering forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, XLNetForQuestionAnswering
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("xlnet-base-cased")
>>> model = XLNetForQuestionAnswering.from_pretrained("xlnet-base-cased")
>>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(
...     0
... )  # Batch size 1
>>> start_positions = torch.tensor([1])
>>> end_positions = torch.tensor([3])
>>> outputs = model(input_ids, start_positions=start_positions, end_positions=end_positions)
>>> loss = outputs.


TFXLNetModel

class transformers.TFXLNetModel  < source > ( *args **kwargs  )   Parameters   config (XLNetConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare XLNet Model transformer outputting raw hidden-states without any specific head on top. This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior. TensorFlow models and layers in transformers accept two formats as input: having all inputs as keyword arguments (like PyTorch


TFXLNetModel

 models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_mask]) or model([input_ids, attention_mask, token_type_ids]) a dictionary with one or several input Tens


TFXLNetModel

ors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None mems: np.ndarray | tf.Tensor | None = None perm_mask: np.ndarray | tf.Tensor | None = None target_mapping: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None input_mask: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None use_mems: Optional[bool] = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None training: bool = False  ) → transformers.models.xlnet


TFXLNetModel

.modeling_tf_xlnet.TFXLNetModelOutput or tuple(tf.Tensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   mems (List[torch.FloatTensor] of length config.n_layers) —
Contains pre-computed hidden-states (see mems output below) . Can be used to speed up sequential
decoding. The token ids which have their past given to this model should not be passed as input_ids as
they have already been computed.
use_mems has to be set to True to make use of mems.   perm_mask (torc


TFXLNetModel

h.FloatTensor of shape (batch_size, sequence_length, sequence_length), optional) —
Mask to indicate the attention pattern for each input token with values selected in [0, 1]:
if perm_mask[k, i, j] = 0, i attend to j in batch k;
if perm_mask[k, i, j] = 1, i does not attend to j in batch k.
If not set, each token attends to all the others (full bidirectional attention). Only used during
pretraining (to define factorization order) or for sequential decoding (generation).   target_mapping (torch.FloatTensor of shape (batch_size, num_predict, sequence_length), optional) —
Mask to indicate the output tokens to use. If target_mapping[k, i, j] = 1, the i-th predict in batch k is
on the j-th token. Only used during pretraining for partial prediction or for sequential decoding
(generation).   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are sele


TFXLNetModel

cted in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   input_mask (torch.FloatTensor of shape batch_size, sequence_length, optional) —
Mask to avoid performing attention on padding token indices. Negative of attention_mask, i.e. with 0 for
real tokens and 1 for padding which is kept for compatibility with the original code base.
Mask values selected in [0, 1]:
1 for tokens that are masked,
0 for tokens that are not masked.
You can only uses one of input_mask and attention_mask.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly p


TFXLNetModel

ass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.models.xlnet.modeling_tf_xlnet.TFXLNetModelOutput or tuple(tf.Tensor)

A transformers.models.xlnet.modeling_tf_xlnet.TFXLNetModelOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (XLNetConfig) and inputs.
last_hidden_state (tf.Ten


TFXLNetModel

sor of shape (batch_size, num_predict, hidden_size)) — Sequence of hidden-states at the last layer of the model.
num_predict corresponds to target_mapping.shape[1]. If target_mapping is None, then num_predict
corresponds to sequence_length.
mems (List[tf.Tensor] of length config.n_layers) — Contains pre-computed hidden-states. Can be used (see mems input) to speed up sequential decoding. The
token ids which have their past given to this model should not be passed as input_ids as they have
already been computed.
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is


TFXLNetModel

 passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The TFXLNetModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, TFXLNetModel
>>> import tensorflow as tf
>>> tokenizer = AutoTokenizer.from_pretrained("xlnet-base-cased")
>>> model = TFXLNetModel.from_pretrained("xlnet-base-cased")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="tf")
>>> outputs = model(inputs)
>>> last_hidden_states = outputs.last_hidd


TFXLNetLMHeadModel

class transformers.TFXLNetLMHeadModel  < source > ( *args **kwargs  )   Parameters   config (XLNetConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    XLNet Model with a language modeling head on top (linear layer with weights tied to the input embeddings). This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior. TensorFlow models and layers in transformers accept


TFXLNetLMHeadModel

 two formats as input: having all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:


TFXLNetLMHeadModel

model([input_ids, attention_mask]) or model([input_ids, attention_mask, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None mems: np.ndarray | tf.Tensor | None = None perm_mask: np.ndarray | tf.Tensor | None = None target_mapping: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None input_mask: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None use_mems: Opti


TFXLNetLMHeadModel

onal[bool] = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None labels: np.ndarray | tf.Tensor | None = None training: bool = False  ) → transformers.models.xlnet.modeling_tf_xlnet.TFXLNetLMHeadModelOutput or tuple(tf.Tensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   mems (List[torch.FloatTensor] of length config.n_layers) —
Co


TFXLNetLMHeadModel

ntains pre-computed hidden-states (see mems output below) . Can be used to speed up sequential
decoding. The token ids which have their past given to this model should not be passed as input_ids as
they have already been computed.
use_mems has to be set to True to make use of mems.   perm_mask (torch.FloatTensor of shape (batch_size, sequence_length, sequence_length), optional) —
Mask to indicate the attention pattern for each input token with values selected in [0, 1]:
if perm_mask[k, i, j] = 0, i attend to j in batch k;
if perm_mask[k, i, j] = 1, i does not attend to j in batch k.
If not set, each token attends to all the others (full bidirectional attention). Only used during
pretraining (to define factorization order) or for sequential decoding (generation).   target_mapping (torch.FloatTensor of shape (batch_size, num_predict, sequence_length), optional) —
Mask to indicate the output tokens to


TFXLNetLMHeadModel

use. If target_mapping[k, i, j] = 1, the i-th predict in batch k is
on the j-th token. Only used during pretraining for partial prediction or for sequential decoding
(generation).   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   input_mask (torch.FloatTensor of shape batch_size, sequence_length, optional) —
Mask to avoid performing attention on padding token indices. Negative of attention_mask, i.e. with 0 for
real tokens and 1 for padding which is kept for compatibility with the original code base.
Mask values selected in [0, 1]:
1 for tokens that are masked,
0 for tokens that are not masked.
You can only uses one of input_mask and attention_mask.   head_ma


TFXLNetLMHeadModel

sk (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for


TFXLNetLMHeadModel

more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (tf.Tensor of shape (batch_size, sequence_length), optional) —
Labels for computing the cross entropy classification loss. Indices should be in [0, ..., config.vocab_size - 1].   Returns
transformers.models.xlnet.modeling_tf_xlnet.TFXLNetLMHeadModelOutput or tuple(tf.Tensor)

A transformers.models.xlnet.modeling_tf_xlnet.TFXLNetLMHeadModelOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (XLNetConfig) and inputs.
loss (tf.Tensor of shape (1,), optional, returned when labels is provided)
Language modeling loss (for next-token prediction).
logits (tf.Tensor of shape (batch_size, num_predict, config.vocab_size)) — Prediction scores of the language modeling head (scores for eac


TFXLNetLMHeadModel

h vocabulary token before SoftMax).
num_predict corresponds to target_mapping.shape[1]. If target_mapping is None, then num_predict
corresponds to sequence_length.
mems (List[tf.Tensor] of length config.n_layers) — Contains pre-computed hidden-states. Can be used (see mems input) to speed up sequential decoding. The
token ids which have their past given to this model should not be passed as input_ids as they have
already been computed.
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when con


TFXLNetLMHeadModel

fig.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The TFXLNetLMHeadModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> import tensorflow as tf
>>> import numpy as np
>>> from transformers import AutoTokenizer, TFXLNetLMHeadModel
>>> tokenizer = AutoTokenizer.from_pretrained("xlnet-large-cased")
>>> model = TFXLNetLMHeadModel.from_pretrained("xlnet-large-cased")
>>> # We show how to setup inputs to predict a next toke


TFXLNetLMHeadModel

n using a bi-directional context.
>>> input_ids = tf.constant(tokenizer.encode("Hello, my dog is very <mask>", add_special_tokens=True))[
...     None, :
... ]  # We will predict the masked token
>>> perm_mask = np.zeros((1, input_ids.shape[1], input_ids.shape[1]))
>>> perm_mask[:, :, -1] = 1.0  # Previous tokens don't see last token
>>> target_mapping = np.zeros(
...     (1, 1, input_ids.shape[1])
... )  # Shape [1, 1, seq_length] => let's predict one token
>>> target_mapping[
...     0, 0, -1
... ] = 1.0  # Our first (and only) prediction will be the last token of the sequence (the masked token)
>>> outputs = model(
...     input_ids,
...     perm_mask=tf.constant(perm_mask, dtype=tf.float32),
...     target_mapping=tf.constant(target_mapping, dtype=tf.float32),
... )
>>> next_token_logits = outputs[
...     0
... ]  # Output has shape [target_mapping.size(0), target_mapping.size(1), config.vocab_


TFXLNetForSequenceClassification

class transformers.TFXLNetForSequenceClassification  < source > ( *args **kwargs  )   Parameters   config (XLNetConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    XLNet Model with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g.
for GLUE tasks. This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior. TensorFlow models and layers in transformers accept two formats as


TFXLNetForSequenceClassification

 input: having all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_mask]) or model([input_ids, attention_ma


TFXLNetForSequenceClassification

sk, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None mems: np.ndarray | tf.Tensor | None = None perm_mask: np.ndarray | tf.Tensor | None = None target_mapping: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None input_mask: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None use_mems: Optional[bool] = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optiona


TFXLNetForSequenceClassification

l[bool] = None labels: np.ndarray | tf.Tensor | None = None training: bool = False  ) → transformers.models.xlnet.modeling_tf_xlnet.TFXLNetForSequenceClassificationOutput or tuple(tf.Tensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   mems (List[torch.FloatTensor] of length config.n_layers) —
Contains pre-computed hidden-states (see mems output below) . Can be used to speed up sequential
decoding. The token ids which have their past given to this model sh


TFXLNetForSequenceClassification

ould not be passed as input_ids as
they have already been computed.
use_mems has to be set to True to make use of mems.   perm_mask (torch.FloatTensor of shape (batch_size, sequence_length, sequence_length), optional) —
Mask to indicate the attention pattern for each input token with values selected in [0, 1]:
if perm_mask[k, i, j] = 0, i attend to j in batch k;
if perm_mask[k, i, j] = 1, i does not attend to j in batch k.
If not set, each token attends to all the others (full bidirectional attention). Only used during
pretraining (to define factorization order) or for sequential decoding (generation).   target_mapping (torch.FloatTensor of shape (batch_size, num_predict, sequence_length), optional) —
Mask to indicate the output tokens to use. If target_mapping[k, i, j] = 1, the i-th predict in batch k is
on the j-th token. Only used during pretraining for partial prediction or for sequential decoding
(generation).   token_type_ids (torch.LongTensor of


TFXLNetForSequenceClassification

 shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   input_mask (torch.FloatTensor of shape batch_size, sequence_length, optional) —
Mask to avoid performing attention on padding token indices. Negative of attention_mask, i.e. with 0 for
real tokens and 1 for padding which is kept for compatibility with the original code base.
Mask values selected in [0, 1]:
1 for tokens that are masked,
0 for tokens that are not masked.
You can only uses one of input_mask and attention_mask.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.F


TFXLNetForSequenceClassification

loatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (tf.Tensor of shape (batch_size,), optional) —
Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regressio


TFXLNetForSequenceClassification

n loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-Entropy).   Returns
transformers.models.xlnet.modeling_tf_xlnet.TFXLNetForSequenceClassificationOutput or tuple(tf.Tensor)

A transformers.models.xlnet.modeling_tf_xlnet.TFXLNetForSequenceClassificationOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (XLNetConfig) and inputs.
loss (tf.Tensor of shape (1,), optional, returned when label is provided) — Classification (or regression if config.num_labels==1) loss.
logits (tf.Tensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
mems (List[tf.Tensor] of length config.n_layers) — Contains pre-computed hidden-states. Can be used (see mems input) to speed up sequential decoding. The
token ids which have their past given to


TFXLNetForSequenceClassification

this model should not be passed as input_ids as they have
already been computed.
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The TFXLNetForSequenceClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within thi


TFXLNetForSequenceClassification

s function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, TFXLNetForSequenceClassification
>>> import tensorflow as tf
>>> tokenizer = AutoTokenizer.from_pretrained("xlnet-base-cased")
>>> model = TFXLNetForSequenceClassification.from_pretrained("xlnet-base-cased")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="tf")
>>> logits = model(**inputs).logits
>>> predicted_class_id = int(tf.math.argmax(logits, axis=-1)[0])    Copied >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`
>>> num_labels = len(model.config.id2label)
>>> model = TFXLNetForSequenceClassification.from_pretrained("xlnet-base-cased", num_labels=num_labels)
>>> labels = tf.constant(1)
>>> loss = model(**inputs, labels=label


TFLNetForMultipleChoice

class transformers.TFXLNetForMultipleChoice  < source > ( *args **kwargs  )   Parameters   config (XLNetConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    XLNET Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a
softmax) e.g. for RocStories/SWAG tasks. This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior. TensorFlow models and layers in transformers acc


TFLNetForMultipleChoice

ept two formats as input: having all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_mask]) or model([