LongformerForSequenceClassification

bels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-Entropy).   Returns
transformers.models.longformer.modeling_longformer.LongformerSequenceClassifierOutput or tuple(torch.FloatTensor)

A transformers.models.longformer.modeling_longformer.LongformerSequenceClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (LongformerConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before Soft


LongformerForSequenceClassification

Max).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x + attention_window + 1), where x is the number of tokens with global attention mask.
Local attentions weights after the attention softmax, used to compute the weighted average in the
self-attention heads. Those are the attention weights from every token in the sequence to every token with
global attention (first x values) and to e


LongformerForSequenceClassification

very token in the attention window (remaining `attention_window
1values). Note that the firstxvalues refer to tokens with fixed positions in the text, but the remainingattention_window + 1values refer to tokens with relative positions: the attention weight of a token to itself is located at indexx + attention_window / 2and theattention_window / 2preceding (succeeding) values are the attention weights to theattention_window / 2preceding (succeeding) tokens. If the attention window contains a token with global attention, the attention weight at the corresponding index is set to 0; the value should be accessed from the firstxattention weights. If a token has global attention, the attention weights to all other tokens inattentionsis set to 0, the values should be accessed fromglobal_attentions`.
global_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.Flo


LongformerForSequenceClassification

atTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x),
where x is the number of tokens with global attention mask.
Global attentions weights after the attention softmax, used to compute the weighted average in the
self-attention heads. Those are the attention weights from every token with global attention to every token
in the sequence.
  The LongformerForSequenceClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example of single-label classification:   Copied >>> import torch
>>> from transformers import AutoTokenizer, LongformerForSequenceClassification
>>> tokenizer = AutoTokenizer.from_pretrained("jpwahle/longformer-base-plagiarism-detection")
>>>


LongformerForSequenceClassification

 model = LongformerForSequenceClassification.from_pretrained("jpwahle/longformer-base-plagiarism-detection")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_class_id = logits.argmax().item()
>>> model.config.id2label[predicted_class_id]
'ORIGINAL'
>>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`
>>> num_labels = len(model.config.id2label)
>>> model = LongformerForSequenceClassification.from_pretrained("jpwahle/longformer-base-plagiarism-detection", num_labels=num_labels)
>>> labels = torch.tensor([1])
>>> loss = model(**inputs, labels=labels).loss
>>> round(loss.item(), 2)
5.44  Example of multi-label classification:   Copied >>> import torch
>>> from transformers import AutoTokenizer, LongformerForSequenceClassification
>>> tokenizer = AutoTokenizer.from_pretrained("jpwahle/longformer-bas


LongformerForSequenceClassification

e-plagiarism-detection")
>>> model = LongformerForSequenceClassification.from_pretrained("jpwahle/longformer-base-plagiarism-detection", problem_type="multi_label_classification")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_class_ids = torch.arange(0, logits.shape[-1])[torch.sigmoid(logits).squeeze(dim=0) > 0.5]
>>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`
>>> num_labels = len(model.config.id2label)
>>> model = LongformerForSequenceClassification.from_pretrained(
...     "jpwahle/longformer-base-plagiarism-detection", num_labels=num_labels, problem_type="multi_label_classification"
... )
>>> labels = torch.sum(
...     torch.nn.functional.one_hot(predicted_class_ids[None, :].clone(), num_classes=num_labels), dim=1
... ).to(torch.float)
>>> loss = model(**inputs, labels=labels).loss


LongformerForMultipleChoice

class transformers.LongformerForMultipleChoice  < source > ( config  )   Parameters   config (LongformerConfig) — Model configuration class with all the parameters of the
model. Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Longformer Model with a multiple choice classification head on top (a linear layer on top of the pooled output and
a softmax) e.g. for RocStories/SWAG tasks. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  <


LongformerForMultipleChoice

 source > ( input_ids: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None global_attention_mask: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.longformer.modeling_longformer.LongformerMultipleChoiceModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, num_choices, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer


LongformerForMultipleChoice

.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, num_choices, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   global_attention_mask (torch.FloatTensor of shape (batch_size, num_choices, sequence_length), optional) —
Mask to decide the attention given on each token, local attention or global attention. Tokens with global
attention attends to all other tokens, and all other tokens attend to them. This is important for
task-specific finetuning because it makes the model more flexible at representing the task. For example,
for classification, the  token should be given global attention. For QA, all question tokens should also
have global attention. Please refer to the Longformer paper for more
details. Mask val


LongformerForMultipleChoice

ues selected in [0, 1]:
0 for local attention (a sliding window attention),
1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).
   head_mask (torch.Tensor of shape (num_layers, num_heads), optional) —
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   decoder_head_mask (torch.Tensor of shape (num_layers, num_heads), optional) —
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   token_type_ids (torch.LongTensor of shape (batch_size, num_choices, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 correspond


LongformerForMultipleChoice

s to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, num_choices, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   inputs_embeds (torch.FloatTensor of shape (batch_size, num_choices, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states o


LongformerForMultipleChoice

f all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the multiple choice classification loss. Indices should be in [0, ..., num_choices-1] where num_choices is the size of the second dimension of the input tensors. (See
input_ids above)   Returns
transformers.models.longformer.modeling_longformer.LongformerMultipleChoiceModelOutput or tuple(torch.FloatTensor)

A transformers.models.longformer.modeling_longformer.LongformerMultipleChoiceModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (LongformerConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Clas


LongformerForMultipleChoice

sification loss.
logits (torch.FloatTensor of shape (batch_size, num_choices)) — num_choices is the second dimension of the input tensors. (see input_ids above).
Classification scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x + attention_window + 1), where x is the number of tokens with global attention mask.
Local attentions wei


LongformerForMultipleChoice

ghts after the attention softmax, used to compute the weighted average in the
self-attention heads. Those are the attention weights from every token in the sequence to every token with
global attention (first x values) and to every token in the attention window (remaining `attention_window
1values). Note that the firstxvalues refer to tokens with fixed positions in the text, but the remainingattention_window + 1values refer to tokens with relative positions: the attention weight of a token to itself is located at indexx + attention_window / 2and theattention_window / 2preceding (succeeding) values are the attention weights to theattention_window / 2preceding (succeeding) tokens. If the attention window contains a token with global attention, the attention weight at the corresponding index is set to 0; the value should be accessed from the firstxattention weights. If a token has global attention, the attention weights to


LongformerForMultipleChoice

 all other tokens inattentionsis set to 0, the values should be accessed fromglobal_attentions`.
global_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x),
where x is the number of tokens with global attention mask.
Global attentions weights after the attention softmax, used to compute the weighted average in the
self-attention heads. Those are the attention weights from every token with global attention to every token
in the sequence.
  The LongformerForMultipleChoice forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps whi


LongformerForMultipleChoice

le
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, LongformerForMultipleChoice
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("allenai/longformer-base-4096")
>>> model = LongformerForMultipleChoice.from_pretrained("allenai/longformer-base-4096")
>>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
>>> choice0 = "It is eaten with a fork and a knife."
>>> choice1 = "It is eaten while held in the hand."
>>> labels = torch.tensor(0).unsqueeze(0)  # choice0 is correct (according to Wikipedia ;)), batch size 1
>>> encoding = tokenizer([prompt, prompt], [choice0, choice1], return_tensors="pt", padding=True)
>>> outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()}, labels=labels)  # batch size is 1
>>> # the linear classifier still needs to be trained
>>> loss = outputs.loss
>>> logits = outputs.lo


LongformerForTokenClassification

class transformers.LongformerForTokenClassification  < source > ( config  )   Parameters   config (LongformerConfig) — Model configuration class with all the parameters of the
model. Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Longformer Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g.
for Named-Entity-Recognition (NER) tasks. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward


LongformerForTokenClassification

< source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None global_attention_mask: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.longformer.modeling_longformer.LongformerTokenClassifierOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for detail


LongformerForTokenClassification

s.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   global_attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to decide the attention given on each token, local attention or global attention. Tokens with global
attention attends to all other tokens, and all other tokens attend to them. This is important for
task-specific finetuning because it makes the model more flexible at representing the task. For example,
for classification, the  token should be given global attention. For QA, all question tokens should also
have global attention. Please refer to the Longformer paper for more
details. Mask values selected in [0, 1]:
0 for local attention


LongformerForTokenClassification

(a sliding window attention),
1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).
   head_mask (torch.Tensor of shape (num_layers, num_heads), optional) —
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   decoder_head_mask (torch.Tensor of shape (num_layers, num_heads), optional) —
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_


LongformerForTokenClassification

ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bo


LongformerForTokenClassification

ol, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Labels for computing the token classification loss. Indices should be in [0, ..., config.num_labels - 1].   Returns
transformers.models.longformer.modeling_longformer.LongformerTokenClassifierOutput or tuple(torch.FloatTensor)

A transformers.models.longformer.modeling_longformer.LongformerTokenClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (LongformerConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided)  — Classification loss.
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.num_labels)) — Classification scores (before SoftMax).
hidden_states (tuple(torch.FloatTenso


LongformerForTokenClassification

r), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x + attention_window + 1), where x is the number of tokens with global attention mask.
Local attentions weights after the attention softmax, used to compute the weighted average in the
self-attention heads. Those are the attention weights from every token in the sequence to every token with
global attention (first x values) and to every token in


LongformerForTokenClassification

the attention window (remaining `attention_window
1values). Note that the firstxvalues refer to tokens with fixed positions in the text, but the remainingattention_window + 1values refer to tokens with relative positions: the attention weight of a token to itself is located at indexx + attention_window / 2and theattention_window / 2preceding (succeeding) values are the attention weights to theattention_window / 2preceding (succeeding) tokens. If the attention window contains a token with global attention, the attention weight at the corresponding index is set to 0; the value should be accessed from the firstxattention weights. If a token has global attention, the attention weights to all other tokens inattentionsis set to 0, the values should be accessed fromglobal_attentions`.
global_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tup


LongformerForTokenClassification

le of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x),
where x is the number of tokens with global attention mask.
Global attentions weights after the attention softmax, used to compute the weighted average in the
self-attention heads. Those are the attention weights from every token with global attention to every token
in the sequence.
  The LongformerForTokenClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, LongformerForTokenClassification
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("brad1141/Longformer-finetuned-norm")
>>> mo


LongformerForTokenClassification

del = LongformerForTokenClassification.from_pretrained("brad1141/Longformer-finetuned-norm")
>>> inputs = tokenizer(
...     "HuggingFace is a company based in Paris and New York", add_special_tokens=False, return_tensors="pt"
... )
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_token_class_ids = logits.argmax(-1)
>>> # Note that tokens are classified rather then input words which means that
>>> # there might be more predicted token classes than words.
>>> # Multiple token classes might account for the same word
>>> predicted_tokens_classes = [model.config.id2label[t.item()] for t in predicted_token_class_ids[0]]
>>> predicted_tokens_classes
['Evidence', 'Evidence', 'Evidence', 'Evidence', 'Evidence', 'Evidence', 'Evidence', 'Evidence', 'Evidence', 'Evidence', 'Evidence', 'Evidence']
>>> labels = predicted_token_class_ids
>>> loss = model(**inputs, labels=labels).loss
>>> round(loss.item()


LongformerForQuestionAnswering

class transformers.LongformerForQuestionAnswering  < source > ( config  )   Parameters   config (LongformerConfig) — Model configuration class with all the parameters of the
model. Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Longformer Model with a span classification head on top for extractive question-answering tasks like SQuAD /
TriviaQA (a linear layers on top of the hidden-states output to compute span start logits and span end logits). This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTor


LongformerForQuestionAnswering

ch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None global_attention_mask: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None start_positions: typing.Optional[torch.Tensor] = None end_positions: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.longformer.modeling_longformer.LongformerQuestionAnsweringModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indi


LongformerForQuestionAnswering

ces of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   global_attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to decide the attention given on each token, local attention or global attention. Tokens with global
attention attends to all other tokens, and all other tokens attend to them. This is important for
task-specific finetuning because it makes the model more flexible at representing the task. For example,
for classification, the  token should be given globa


LongformerForQuestionAnswering

l attention. For QA, all question tokens should also
have global attention. Please refer to the Longformer paper for more
details. Mask values selected in [0, 1]:
0 for local attention (a sliding window attention),
1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).
   head_mask (torch.Tensor of shape (num_layers, num_heads), optional) —
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   decoder_head_mask (torch.Tensor of shape (num_layers, num_heads), optional) —
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token


LongformerForQuestionAnswering

indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See atten


LongformerForQuestionAnswering

tions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   start_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the start of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
are not taken into account for computing the loss.   end_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the end of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
ar


LongformerForQuestionAnswering

e not taken into account for computing the loss.   Returns
transformers.models.longformer.modeling_longformer.LongformerQuestionAnsweringModelOutput or tuple(torch.FloatTensor)

A transformers.models.longformer.modeling_longformer.LongformerQuestionAnsweringModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (LongformerConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Total span extraction loss is the sum of a Cross-Entropy for the start and end positions.
start_logits (torch.FloatTensor of shape (batch_size, sequence_length)) — Span-start scores (before SoftMax).
end_logits (torch.FloatTensor of shape (batch_size, sequence_length)) — Span-end scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when


LongformerForQuestionAnswering

output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x + attention_window + 1), where x is the number of tokens with global attention mask.
Local attentions weights after the attention softmax, used to compute the weighted average in the
self-attention heads. Those are the attention weights from every token in the sequence to every token with
global attention (first x values) and to every token in the attent


LongformerForQuestionAnswering

ion window (remaining `attention_window
1values). Note that the firstxvalues refer to tokens with fixed positions in the text, but the remainingattention_window + 1values refer to tokens with relative positions: the attention weight of a token to itself is located at indexx + attention_window / 2and theattention_window / 2preceding (succeeding) values are the attention weights to theattention_window / 2preceding (succeeding) tokens. If the attention window contains a token with global attention, the attention weight at the corresponding index is set to 0; the value should be accessed from the firstxattention weights. If a token has global attention, the attention weights to all other tokens inattentionsis set to 0, the values should be accessed fromglobal_attentions`.
global_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=Tru


LongformerForQuestionAnswering

e) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x),
where x is the number of tokens with global attention mask.
Global attentions weights after the attention softmax, used to compute the weighted average in the
self-attention heads. Those are the attention weights from every token with global attention to every token
in the sequence.
  The LongformerForQuestionAnswering forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoTokenizer, LongformerForQuestionAnswering
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("allenai/longformer-l


LongformerForQuestionAnswering

arge-4096-finetuned-triviaqa")
>>> model = LongformerForQuestionAnswering.from_pretrained("allenai/longformer-large-4096-finetuned-triviaqa")
>>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
>>> encoding = tokenizer(question, text, return_tensors="pt")
>>> input_ids = encoding["input_ids"]
>>> # default is local attention everywhere
>>> # the forward method will automatically set global attention on question tokens
>>> attention_mask = encoding["attention_mask"]
>>> outputs = model(input_ids, attention_mask=attention_mask)
>>> start_logits = outputs.start_logits
>>> end_logits = outputs.end_logits
>>> all_tokens = tokenizer.convert_ids_to_tokens(input_ids[0].tolist())
>>> answer_tokens = all_tokens[torch.argmax(start_logits) : torch.argmax(end_logits) + 1]
>>> answer = tokenizer.decode(
...     tokenizer.convert_tokens_to_ids(answer_tokens)
... )  # remove space prepending space


TFLongformerModel

class transformers.TFLongformerModel  < source > ( *args **kwargs  )   Parameters   config (LongformerConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare Longformer Model outputting raw hidden-states without any specific head on top. This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior. TensorFlow models and layers in transformers accept two formats as input: having all inputs


TFLongformerModel

as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_mask]) or model([input_ids, attention_m


TFLongformerModel

ask, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function! This class copies code from TFRobertaModel and overwrites standard self-attention with longformer
self-attention to provide the ability to process long sequences following the self-attention approach described in
Longformer: the Long-Document Transformer by Iz Beltagy, Matthew E. Peters, and
Arman Cohan. Longformer self-attention combines a local (sliding window) and global attention to extend to long
documents without the O(n^2) increase in memory and compute. The self-attention module TFLongformerSelfAttention implemented here supports the combination of local a


TFLongformerModel

nd global
attention but it lacks support for autoregressive attention and dilated attention. Autoregressive and dilated
attention are more relevant for autoregressive language modeling than finetuning on downstream tasks. Future
release will add support for autoregressive attention, but the support for dilated attention requires a custom CUDA
kernel to be memory and compute efficient.  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None global_attention_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None training: Optional[bool] = False  )   Parameters   i


TFLongformerModel

nput_ids (np.ndarray or tf.Tensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and
PreTrainedTokenizer.encode() for details.
What are input IDs?   attention_mask (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   head_mask (np.ndarray or tf.Tensor of shape (encoder_layers, encoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   global_attention_mask (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Mask to decide the attention giv


TFLongformerModel

en on each token, local attention or global attention. Tokens with global
attention attends to all other tokens, and all other tokens attend to them. This is important for
task-specific finetuning because it makes the model more flexible at representing the task. For example,
for classification, the  token should be given global attention. For QA, all question tokens should also
have global attention. Please refer to the Longformer paper for more
details. Mask values selected in [0, 1]:
0 for local attention (a sliding window attention),
1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).
   token_type_ids (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token ty


TFLongformerModel

pe IDs?   position_ids (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   inputs_embeds (np.ndarray or tf.Tensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the
config will be used instead.   output_hidden_states (bool, optional) —


TFLongformerModel

Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail. This argument can be used only in eager mode, in graph mode the value in the config will be
used instead.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in
eager mode, in graph mode the value will always be set to True.   training (bool, optional, defaults to False) —
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).    The TFLongformerModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them


TFLongformerForMaskedLM

class transformers.TFLongformerForMaskedLM  < source > ( *args **kwargs  )   Parameters   config (LongformerConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Longformer Model with a language modeling head on top. This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior. TensorFlow models and layers in transformers accept two formats as input: having all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple o


TFLongformerForMaskedLM

r dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_mask]) or model([input_ids, attention_mask, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_i


TFLongformerForMaskedLM

ds": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None global_attention_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None labels: np.ndarray | tf.Tensor | None = None training: Optional[bool] = False  ) → transformers.models.longformer.modeling_tf_longformer.TFLongformerMaskedLMOutput or tuple(tf.Tensor)  Parameters   input_ids (np.ndarray or tf.Tensor of shape (batch_size, sequence_


TFLongformerForMaskedLM

length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and
PreTrainedTokenizer.encode() for details.
What are input IDs?   attention_mask (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   head_mask (np.ndarray or tf.Tensor of shape (encoder_layers, encoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   global_attention_mask (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Mask to decide the attention given on each token, local attention or global attention. Tokens with global
attention attends to all other tokens, and all


TFLongformerForMaskedLM

 other tokens attend to them. This is important for
task-specific finetuning because it makes the model more flexible at representing the task. For example,
for classification, the  token should be given global attention. For QA, all question tokens should also
have global attention. Please refer to the Longformer paper for more
details. Mask values selected in [0, 1]:
0 for local attention (a sliding window attention),
1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).
   token_type_ids (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddin


TFLongformerForMaskedLM

gs. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   inputs_embeds (np.ndarray or tf.Tensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the
config will be used instead.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail. This argument can be used only in eager mode, in graph mode the value in the config will be
used instead.   retur


TFLongformerForMaskedLM

n_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in
eager mode, in graph mode the value will always be set to True.   training (bool, optional, defaults to False) —
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).   labels (tf.Tensor of shape (batch_size, sequence_length), optional) —
Labels for computing the masked language modeling loss. Indices should be in [-100, 0, ..., config.vocab_size] (see input_ids docstring) Tokens with indices set to -100 are ignored (masked), the
loss is only computed for the tokens with labels in [0, ..., config.vocab_size]   Returns
transformers.models.longformer.modeling_tf_longformer.TFLongformerMaskedLMOutput or tuple(tf.Tensor)

A transformers.models.longformer.modeling_tf_longformer.TFLongformerMaskedLMOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.re


TFLongformerForMaskedLM

turn_dict=False) comprising various elements depending on the
configuration (LongformerConfig) and inputs.
loss (tf.Tensor of shape (1,), optional, returned when labels is provided) — Masked language modeling (MLM) loss.
logits (tf.Tensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x +


TFLongformerForMaskedLM

 attention_window + 1), where x is the number of tokens with global attention mask.
Local attentions weights after the attention softmax, used to compute the weighted average in the
self-attention heads. Those are the attention weights from every token in the sequence to every token with
global attention (first x values) and to every token in the attention window (remaining `attention_window
1values). Note that the firstxvalues refer to tokens with fixed positions in the text, but the remainingattention_window + 1values refer to tokens with relative positions: the attention weight of a token to itself is located at indexx + attention_window / 2and theattention_window / 2preceding (succeeding) values are the attention weights to theattention_window / 2preceding (succeeding) tokens. If the attention window contains a token with global attention, the attention weight at the corresponding index is set to 0; the value should be accessed from the firstxattention weights. If a token has


TFLongformerForMaskedLM

 global attention, the attention weights to all other tokens inattentionsis set to 0, the values should be accessed fromglobal_attentions`.
global_attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x), where x
is the number of tokens with global attention mask.
Global attentions weights after the attention softmax, used to compute the weighted average in the
self-attention heads. Those are the attention weights from every token with global attention to every token
in the sequence.
  The TFLongformerForMaskedLM forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.


TFLongformerForMaskedLM

Example:   Copied >>> from transformers import AutoTokenizer, TFLongformerForMaskedLM
>>> import tensorflow as tf
>>> tokenizer = AutoTokenizer.from_pretrained("allenai/longformer-base-4096")
>>> model = TFLongformerForMaskedLM.from_pretrained("allenai/longformer-base-4096")
>>> inputs = tokenizer("The capital of France is <mask>.", return_tensors="tf")
>>> logits = model(**inputs).logits
>>> # retrieve index of <mask>
>>> mask_token_index = tf.where((inputs.input_ids == tokenizer.mask_token_id)[0])
>>> selected_logits = tf.gather_nd(logits[0], indices=mask_token_index)
>>> predicted_token_id = tf.math.argmax(selected_logits, axis=-1)
>>> tokenizer.decode(predicted_token_id)
' Paris'    Copied >>> labels = tokenizer("The capital of France is Paris.", return_tensors="tf")["input_ids"]
>>> # mask labels of non-<mask> tokens
>>> labels = tf.where(inputs.input_ids == tokenizer.mask_token_id, labels, -100)
>>> outputs = model(**inputs, labels=labels)
>>> round(float(outputs.loss), 2)


TFLongformerForQuestionAnswering

class transformers.TFLongformerForQuestionAnswering  < source > ( *args **kwargs  )   Parameters   config (LongformerConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Longformer Model with a span classification head on top for extractive question-answering tasks like SQuAD /
TriviaQA (a linear layer on top of the hidden-states output to compute span start logits and span end logits). This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general u


TFLongformerForQuestionAnswering

sage and
behavior. TensorFlow models and layers in transformers accept two formats as input: having all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a list of varying length with one or several input Tensors IN THE ORDER given in the do


TFLongformerForQuestionAnswering

cstring:
model([input_ids, attention_mask]) or model([input_ids, attention_mask, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None global_attention_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None start_positions: np.ndarray |


TFLongformerForQuestionAnswering

tf.Tensor | None = None end_positions: np.ndarray | tf.Tensor | None = None training: Optional[bool] = False  ) → transformers.models.longformer.modeling_tf_longformer.TFLongformerQuestionAnsweringModelOutput or tuple(tf.Tensor)  Parameters   input_ids (np.ndarray or tf.Tensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and
PreTrainedTokenizer.encode() for details.
What are input IDs?   attention_mask (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   head_mask (np.ndarray or tf.Tensor of shape (encoder_layers, encoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules. Mask values selected in [


TFLongformerForQuestionAnswering

0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   global_attention_mask (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Mask to decide the attention given on each token, local attention or global attention. Tokens with global
attention attends to all other tokens, and all other tokens attend to them. This is important for
task-specific finetuning because it makes the model more flexible at representing the task. For example,
for classification, the  token should be given global attention. For QA, all question tokens should also
have global attention. Please refer to the Longformer paper for more
details. Mask values selected in [0, 1]:
0 for local attention (a sliding window attention),
1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).
   token_type_ids (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Segment token indices to


TFLongformerForQuestionAnswering

 indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   inputs_embeds (np.ndarray or tf.Tensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail. This arg


TFLongformerForQuestionAnswering

ument can be used only in eager mode, in graph mode the value in the
config will be used instead.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail. This argument can be used only in eager mode, in graph mode the value in the config will be
used instead.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in
eager mode, in graph mode the value will always be set to True.   training (bool, optional, defaults to False) —
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).   start_positions (tf.Tensor of shape (batch_size,), optional) —
Labels for position (index) of the start of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequ


TFLongformerForQuestionAnswering

ence_length). Position outside of the sequence
are not taken into account for computing the loss.   end_positions (tf.Tensor of shape (batch_size,), optional) —
Labels for position (index) of the end of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
are not taken into account for computing the loss.   Returns
transformers.models.longformer.modeling_tf_longformer.TFLongformerQuestionAnsweringModelOutput or tuple(tf.Tensor)

A transformers.models.longformer.modeling_tf_longformer.TFLongformerQuestionAnsweringModelOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (LongformerConfig) and inputs.
loss (tf.Tensor of shape (1,), optional, returned when labels is provided) — Total span extraction loss is the sum of a Cross-Entropy for the start


TFLongformerForQuestionAnswering

 and end positions.
start_logits (tf.Tensor of shape (batch_size, sequence_length)) — Span-start scores (before SoftMax).
end_logits (tf.Tensor of shape (batch_size, sequence_length)) — Span-end scores (before SoftMax).
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x + attention_window + 1), where x is the number of tokens with global attention mask.
Local attentions weights after the attention softmax, used to compute the w


TFLongformerForQuestionAnswering

eighted average in the
self-attention heads. Those are the attention weights from every token in the sequence to every token with
global attention (first x values) and to every token in the attention window (remaining `attention_window
1values). Note that the firstxvalues refer to tokens with fixed positions in the text, but the remainingattention_window + 1values refer to tokens with relative positions: the attention weight of a token to itself is located at indexx + attention_window / 2and theattention_window / 2preceding (succeeding) values are the attention weights to theattention_window / 2preceding (succeeding) tokens. If the attention window contains a token with global attention, the attention weight at the corresponding index is set to 0; the value should be accessed from the firstxattention weights. If a token has global attention, the attention weights to all other tokens inattentionsis set to 0, the values should be accessed fromglobal_attentions`.


TFLongformerForQuestionAnswering

global_attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x), where x
is the number of tokens with global attention mask.
Global attentions weights after the attention softmax, used to compute the weighted average in the
self-attention heads. Those are the attention weights from every token with global attention to every token
in the sequence.
  The TFLongformerForQuestionAnswering forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, TFLongformerForQuestionAnswering
>>> import tens


TFLongformerForQuestionAnswering

orflow as tf
>>> tokenizer = AutoTokenizer.from_pretrained("allenai/longformer-large-4096-finetuned-triviaqa")
>>> model = TFLongformerForQuestionAnswering.from_pretrained("allenai/longformer-large-4096-finetuned-triviaqa")
>>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
>>> inputs = tokenizer(question, text, return_tensors="tf")
>>> outputs = model(**inputs)
>>> answer_start_index = int(tf.math.argmax(outputs.start_logits, axis=-1)[0])
>>> answer_end_index = int(tf.math.argmax(outputs.end_logits, axis=-1)[0])
>>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
>>> tokenizer.decode(predict_answer_tokens)
' puppet'    Copied >>> # target is "nice puppet"
>>> target_start_index = tf.constant([14])
>>> target_end_index = tf.constant([15])
>>> outputs = model(**inputs, start_positions=target_start_index, end_positions=target_end_index)
>>> loss = tf.math.reduce_mean(outputs.loss)
>>> round(float(los


TFLongformerForSequenceClassification

class transformers.TFLongformerForSequenceClassification  < source > ( *args **kwargs  )   Parameters   config (LongformerConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Longformer Model transformer with a sequence classification/regression head on top (a linear layer on top of the
pooled output) e.g. for GLUE tasks. This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior. TensorFlow models and layers in t


TFLongformerForSequenceClassification

ransformers accept two formats as input: having all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_m


TFLongformerForSequenceClassification

ask]) or model([input_ids, attention_mask, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None global_attention_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None labels: np.ndarray | tf.Tensor | None = None training:


TFLongformerForSequenceClassification

 Optional[bool] = False  ) → transformers.models.longformer.modeling_tf_longformer.TFLongformerSequenceClassifierOutput or tuple(tf.Tensor)  Parameters   input_ids (np.ndarray or tf.Tensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and
PreTrainedTokenizer.encode() for details.
What are input IDs?   attention_mask (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   head_mask (np.ndarray or tf.Tensor of shape (encoder_layers, encoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.


TFLongformerForSequenceClassification

   global_attention_mask (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Mask to decide the attention given on each token, local attention or global attention. Tokens with global
attention attends to all other tokens, and all other tokens attend to them. This is important for
task-specific finetuning because it makes the model more flexible at representing the task. For example,
for classification, the  token should be given global attention. For QA, all question tokens should also
have global attention. Please refer to the Longformer paper for more
details. Mask values selected in [0, 1]:
0 for local attention (a sliding window attention),
1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).
   token_type_ids (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices ar


TFLongformerForSequenceClassification

e selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   inputs_embeds (np.ndarray or tf.Tensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail. This argument can be used only in eager mode, in graph


TFLongformerForSequenceClassification

mode the value in the
config will be used instead.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail. This argument can be used only in eager mode, in graph mode the value in the config will be
used instead.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in
eager mode, in graph mode the value will always be set to True.   training (bool, optional, defaults to False) —
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).   Returns
transformers.models.longformer.modeling_tf_longformer.TFLongformerSequenceClassifierOutput or tuple(tf.Tensor)

A transformers.models.longformer.modeling_tf_longformer.TFLongformerSequenceClassifierOutput or a tuple of tf.Tensor (if
return_dict=Fals


TFLongformerForSequenceClassification

e is passed or when config.return_dict=False) comprising various elements depending on the
configuration (LongformerConfig) and inputs.
loss (tf.Tensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (tf.Tensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of


TFLongformerForSequenceClassification

 shape (batch_size, num_heads, sequence_length, x + attention_window + 1), where x is the number of tokens with global attention mask.
Local attentions weights after the attention softmax, used to compute the weighted average in the
self-attention heads. Those are the attention weights from every token in the sequence to every token with
global attention (first x values) and to every token in the attention window (remaining `attention_window
1values). Note that the firstxvalues refer to tokens with fixed positions in the text, but the remainingattention_window + 1values refer to tokens with relative positions: the attention weight of a token to itself is located at indexx + attention_window / 2and theattention_window / 2preceding (succeeding) values are the attention weights to theattention_window / 2preceding (succeeding) tokens. If the attention window contains a token with global attention, the attention weight at the corresponding index is se


TFLongformerForSequenceClassification

t to 0; the value should be accessed from the firstxattention weights. If a token has global attention, the attention weights to all other tokens inattentionsis set to 0, the values should be accessed fromglobal_attentions`.
global_attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x), where x
is the number of tokens with global attention mask.
Global attentions weights after the attention softmax, used to compute the weighted average in the
self-attention heads. Those are the attention weights from every token with global attention to every token
in the sequence.
  The TFLongformerForSequenceClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwar


TFLongformerForSequenceClassification

ds instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, TFLongformerForSequenceClassification
>>> import tensorflow as tf
>>> tokenizer = AutoTokenizer.from_pretrained("allenai/longformer-base-4096")
>>> model = TFLongformerForSequenceClassification.from_pretrained("allenai/longformer-base-4096")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="tf")
>>> logits = model(**inputs).logits
>>> predicted_class_id = int(tf.math.argmax(logits, axis=-1)[0])    Copied >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`
>>> num_labels = len(model.config.id2label)
>>> model = TFLongformerForSequenceClassification.from_pretrained("allenai/longformer-base-4096", num_labels=num_labels)
>>> labels = tf.constant(1)
>>> loss = model(**inputs, labels=labe


TFLongformerForTokenClassification

class transformers.TFLongformerForTokenClassification  < source > ( *args **kwargs  )   Parameters   config (LongformerConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Longformer Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g.
for Named-Entity-Recognition (NER) tasks. This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior. TensorFlow models and layers in transformers accept


TFLongformerForTokenClassification

two formats as input: having all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_mask]) or model([input_ids, attention_mask,


TFLongformerForTokenClassification

 token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None global_attention_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None labels: Optional[Union[np.array, tf.Tensor]] = None training: Optional[bool] = False  ) → transformers.models.longforme


TFLongformerForTokenClassification

r.modeling_tf_longformer.TFLongformerTokenClassifierOutput or tuple(tf.Tensor)  Parameters   input_ids (np.ndarray or tf.Tensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and
PreTrainedTokenizer.encode() for details.
What are input IDs?   attention_mask (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   head_mask (np.ndarray or tf.Tensor of shape (encoder_layers, encoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   global_attention_mask (np.ndarray or tf.Tensor of shape (batch_size, sequence_len


TFLongformerForTokenClassification

gth), optional) —
Mask to decide the attention given on each token, local attention or global attention. Tokens with global
attention attends to all other tokens, and all other tokens attend to them. This is important for
task-specific finetuning because it makes the model more flexible at representing the task. For example,
for classification, the  token should be given global attention. For QA, all question tokens should also
have global attention. Please refer to the Longformer paper for more
details. Mask values selected in [0, 1]:
0 for local attention (a sliding window attention),
1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).
   token_type_ids (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are to


TFLongformerForTokenClassification

ken type IDs?   position_ids (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   inputs_embeds (np.ndarray or tf.Tensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the
config will be used instead.   output_hidden_states (bool, optional) —
Whether or not to return the hidden s


TFLongformerForTokenClassification

tates of all layers. See hidden_states under returned tensors for
more detail. This argument can be used only in eager mode, in graph mode the value in the config will be
used instead.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in
eager mode, in graph mode the value will always be set to True.   training (bool, optional, defaults to False) —
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).   labels (tf.Tensor of shape (batch_size, sequence_length), optional) —
Labels for computing the token classification loss. Indices should be in [0, ..., config.num_labels - 1].   Returns
transformers.models.longformer.modeling_tf_longformer.TFLongformerTokenClassifierOutput or tuple(tf.Tensor)

A transformers.models.longformer.modeling_tf_longformer.TFLongformerTokenClassifierOutput or a tuple of tf.Tensor (


TFLongformerForTokenClassification

if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (LongformerConfig) and inputs.
loss (tf.Tensor of shape (1,), optional, returned when labels is provided)  — Classification loss.
logits (tf.Tensor of shape (batch_size, sequence_length, config.num_labels)) — Classification scores (before SoftMax).
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x + attention_wind


TFLongformerForTokenClassification

ow + 1), where x is the number of tokens with global attention mask.
Local attentions weights after the attention softmax, used to compute the weighted average in the
self-attention heads. Those are the attention weights from every token in the sequence to every token with
global attention (first x values) and to every token in the attention window (remaining `attention_window
1values). Note that the firstxvalues refer to tokens with fixed positions in the text, but the remainingattention_window + 1values refer to tokens with relative positions: the attention weight of a token to itself is located at indexx + attention_window / 2and theattention_window / 2preceding (succeeding) values are the attention weights to theattention_window / 2preceding (succeeding) tokens. If the attention window contains a token with global attention, the attention weight at the corresponding index is set to 0; the value should be accessed from the firstxattention weights. If a token has glo


TFLongformerForTokenClassification

bal attention, the attention weights to all other tokens inattentionsis set to 0, the values should be accessed fromglobal_attentions`.
global_attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x), where x
is the number of tokens with global attention mask.
Global attentions weights after the attention softmax, used to compute the weighted average in the
self-attention heads. Those are the attention weights from every token with global attention to every token
in the sequence.
  The TFLongformerForTokenClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silent


TFLongformerForTokenClassification

ly ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, TFLongformerForTokenClassification
>>> import tensorflow as tf
>>> tokenizer = AutoTokenizer.from_pretrained("allenai/longformer-base-4096")
>>> model = TFLongformerForTokenClassification.from_pretrained("allenai/longformer-base-4096")
>>> inputs = tokenizer(
...     "HuggingFace is a company based in Paris and New York", add_special_tokens=False, return_tensors="tf"
... )
>>> logits = model(**inputs).logits
>>> predicted_token_class_ids = tf.math.argmax(logits, axis=-1)
>>> # Note that tokens are classified rather then input words which means that
>>> # there might be more predicted token classes than words.
>>> # Multiple token classes might account for the same word
>>> predicted_tokens_classes = [model.config.id2label[t] for t in predicted_token_class_ids[0].numpy().tolist()]    Copied >>> labels = predicted_token_class_ids
>>> loss = tf.math.reduce_mean(model(**inputs, labels=labels).l


TFLongformerForMultipleChoice

class transformers.TFLongformerForMultipleChoice  < source > ( *args **kwargs  )   Parameters   config (LongformerConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Longformer Model with a multiple choice classification head on top (a linear layer on top of the pooled output and
a softmax) e.g. for RocStories/SWAG tasks. This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior. TensorFlow models and layers in transformers accept two fo


TFLongformerForMultipleChoice

rmats as input: having all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_mask]) or model([input_ids, attention_mask, token_ty


TFLongformerForMultipleChoice

pe_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None global_attention_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None labels: np.ndarray | tf.Tensor | None = None training: Optional[bool] = False  ) → transformers.models.longformer.modeling_tf_longf


TFLongformerForMultipleChoice

ormer.TFLongformerMultipleChoiceModelOutput or tuple(tf.Tensor)  Parameters   input_ids (np.ndarray or tf.Tensor of shape (batch_size, num_choices, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and
PreTrainedTokenizer.encode() for details.
What are input IDs?   attention_mask (np.ndarray or tf.Tensor of shape (batch_size, num_choices, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   head_mask (np.ndarray or tf.Tensor of shape (encoder_layers, encoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   global_attention_mask (np.ndarray or tf.Tensor of shape (batch_size, num_


TFLongformerForMultipleChoice

choices, sequence_length), optional) —
Mask to decide the attention given on each token, local attention or global attention. Tokens with global
attention attends to all other tokens, and all other tokens attend to them. This is important for
task-specific finetuning because it makes the model more flexible at representing the task. For example,
for classification, the  token should be given global attention. For QA, all question tokens should also
have global attention. Please refer to the Longformer paper for more
details. Mask values selected in [0, 1]:
0 for local attention (a sliding window attention),
1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).
   token_type_ids (np.ndarray or tf.Tensor of shape (batch_size, num_choices, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to


TFLongformerForMultipleChoice

a sentence B token.
What are token type IDs?   position_ids (np.ndarray or tf.Tensor of shape (batch_size, num_choices, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   inputs_embeds (np.ndarray or tf.Tensor of shape (batch_size, num_choices, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the
config will be used instead.   output_hidden_states (b


TFLongformerForMultipleChoice

ool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail. This argument can be used only in eager mode, in graph mode the value in the config will be
used instead.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in
eager mode, in graph mode the value will always be set to True.   training (bool, optional, defaults to False) —
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).   labels (tf.Tensor of shape (batch_size,), optional) —
Labels for computing the multiple choice classification loss. Indices should be in [0, ..., num_choices]
where num_choices is the size of the second dimension of the input tensors. (See input_ids above)   Returns
transformers.models.longformer.modeling_tf_longformer.TFLongformerMultipleChoiceModelOutput or


TFLongformerForMultipleChoice

 tuple(tf.Tensor)

A transformers.models.longformer.modeling_tf_longformer.TFLongformerMultipleChoiceModelOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (LongformerConfig) and inputs.
loss (tf.Tensor of shape (1,), optional, returned when labels is provided) — Classification loss.
logits (tf.Tensor of shape (batch_size, num_choices)) — num_choices is the second dimension of the input tensors. (see input_ids above).
Classification scores (before SoftMax).
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, re


TFLongformerForMultipleChoice

turned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x + attention_window + 1), where x is the number of tokens with global attention mask.
Local attentions weights after the attention softmax, used to compute the weighted average in the
self-attention heads. Those are the attention weights from every token in the sequence to every token with
global attention (first x values) and to every token in the attention window (remaining `attention_window
1values). Note that the firstxvalues refer to tokens with fixed positions in the text, but the remainingattention_window + 1values refer to tokens with relative positions: the attention weight of a token to itself is located at indexx + attention_window / 2and theattention_window / 2preceding (succeeding) values are the attention weights to theattention_window / 2preceding (succeeding) tokens. If the attentio


TFLongformerForMultipleChoice

n window contains a token with global attention, the attention weight at the corresponding index is set to 0; the value should be accessed from the firstxattention weights. If a token has global attention, the attention weights to all other tokens inattentionsis set to 0, the values should be accessed fromglobal_attentions`.
global_attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x), where x
is the number of tokens with global attention mask.
Global attentions weights after the attention softmax, used to compute the weighted average in the
self-attention heads. Those are the attention weights from every token with global attention to every token
in the sequence.
  The TFLongformerForMultipleChoice forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be define