LukeForEntityPairClassification

ould call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoTokenizer, LukeForEntityPairClassification
>>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-large-finetuned-tacred")
>>> model = LukeForEntityPairClassification.from_pretrained("studio-ousia/luke-large-finetuned-tacred")
>>> text = "Beyoncé lives in Los Angeles."
>>> entity_spans = [
...     (0, 7),
...     (17, 28),
... ]  # character-based entity spans corresponding to "Beyoncé" and "Los Angeles"
>>> inputs = tokenizer(text, entity_spans=entity_spans, return_tensors="pt")
>>> outputs = model(**inputs)
>>> logits = outputs.logits
>>> predicted_class_idx = logits.argmax(-1).item()
>>> print("Predicted class:", model.config.id2label[predicted_class_idx])
Predicted class: per:cities_of_resi


LukeForEntitySpanClassification

class transformers.LukeForEntitySpanClassification  < source > ( config  )   Parameters   config (LukeConfig) — Model configuration class with all the parameters of the
model. Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The LUKE model with a span classification head on top (a linear layer on top of the hidden states output) for tasks
such as named entity recognition. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  <


LukeForEntitySpanClassification

source > ( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.LongTensor] = None entity_ids: typing.Optional[torch.LongTensor] = None entity_attention_mask: typing.Optional[torch.LongTensor] = None entity_token_type_ids: typing.Optional[torch.LongTensor] = None entity_position_ids: typing.Optional[torch.LongTensor] = None entity_start_positions: typing.Optional[torch.LongTensor] = None entity_end_positions: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.LongTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.


LukeForEntitySpanClassification

models.luke.modeling_luke.EntitySpanClassificationOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type I


LukeForEntitySpanClassification

Ds?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   entity_ids (torch.LongTensor of shape (batch_size, entity_length)) —
Indices of entity tokens in the entity vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.   entity_attention_mask (torch.FloatTensor of shape (batch_size, entity_length), optional) —
Mask to avoid performing attention on padding entity token indices. Mask values selected in [0, 1]:
1 for entity tokens that are not masked,
0 for entity tokens that are masked.
   entity_token_type_ids (torch.LongTensor of shape (batch_size, entity_length), optional) —
Segment token indices to indicate first and second portions o


LukeForEntitySpanClassification

f the entity token inputs. Indices are
selected in [0, 1]:
0 corresponds to a portion A entity token,
1 corresponds to a portion B entity token.
   entity_position_ids (torch.LongTensor of shape (batch_size, entity_length, max_mention_length), optional) —
Indices of positions of each input entity in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [


LukeForEntitySpanClassification

0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   entity_start_positions (torch.LongTensor) —
The start positions of entities in the word token sequence.   entity_end_positions (torch.LongTensor) —
The end positions of entities in the word token sequence.   labels (torch.LongTensor of shape (batch_size, entity_length) or (batch_size, entity_length, num_labels), optional) —
Labels for computing the classification loss. If the shape is (batch_size, entity_length), the c


LukeForEntitySpanClassification

ross
entropy loss is used for the single-label classification. In this case, labels should contain the indices
that should be in [0, ..., config.num_labels - 1]. If the shape is (batch_size, entity_length, num_labels), the binary cross entropy loss is used for the multi-label classification. In this case,
labels should only contain [0, 1], where 0 and 1 indicate false and true, respectively.   Returns
transformers.models.luke.modeling_luke.EntitySpanClassificationOutput or tuple(torch.FloatTensor)

A transformers.models.luke.modeling_luke.EntitySpanClassificationOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (LukeConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification loss.
logits (torch.FloatTensor of shape (batch_size, entity_l


LukeForEntitySpanClassification

ength, config.num_labels)) — Classification scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size). Hidden-states of the model at the output of each layer
plus the initial embedding outputs.
entity_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, entity_length, hidden_size). Entity hidden-states of the model at the output of each
layer plus the initial entity embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returne


LukeForEntitySpanClassification

d when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Attentions weights after the attention softmax, used to compute the weighted average in
the self-attention heads.
  The LukeForEntitySpanClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoTokenizer, LukeForEntitySpanClassification
>>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-large-finetuned-conll-2003")
>>> model = LukeForEntitySpanClassification.from_pretrained("studio-ousia/


LukeForEntitySpanClassification

luke-large-finetuned-conll-2003")
>>> text = "Beyoncé lives in Los Angeles"
# List all possible entity spans in the text
>>> word_start_positions = [0, 8, 14, 17, 21]  # character-based start positions of word tokens
>>> word_end_positions = [7, 13, 16, 20, 28]  # character-based end positions of word tokens
>>> entity_spans = []
>>> for i, start_pos in enumerate(word_start_positions):
...     for end_pos in word_end_positions[i:]:
...         entity_spans.append((start_pos, end_pos))
>>> inputs = tokenizer(text, entity_spans=entity_spans, return_tensors="pt")
>>> outputs = model(**inputs)
>>> logits = outputs.logits
>>> predicted_class_indices = logits.argmax(-1).squeeze().tolist()
>>> for span, predicted_class_idx in zip(entity_spans, predicted_class_indices):
...     if predicted_class_idx != 0:
...         print(text[span[0] : span[1]], model.config.id2label[predicted_class_idx])
Beyoncé PER
Los Angeles LO


LukeForSequenceClassification

class transformers.LukeForSequenceClassification  < source > ( config  )   Parameters   config (LukeConfig) — Model configuration class with all the parameters of the
model. Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The LUKE Model transformer with a sequence classification/regression head on top (a linear layer on top of the
pooled output) e.g. for GLUE tasks. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typ


LukeForSequenceClassification

ing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.LongTensor] = None entity_ids: typing.Optional[torch.LongTensor] = None entity_attention_mask: typing.Optional[torch.FloatTensor] = None entity_token_type_ids: typing.Optional[torch.LongTensor] = None entity_position_ids: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.luke.modeling_luke.LukeSequenceClassifierOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indice


LukeForSequenceClassification

s of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, co


LukeForSequenceClassification

nfig.max_position_embeddings - 1].
What are position IDs?   entity_ids (torch.LongTensor of shape (batch_size, entity_length)) —
Indices of entity tokens in the entity vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.   entity_attention_mask (torch.FloatTensor of shape (batch_size, entity_length), optional) —
Mask to avoid performing attention on padding entity token indices. Mask values selected in [0, 1]:
1 for entity tokens that are not masked,
0 for entity tokens that are masked.
   entity_token_type_ids (torch.LongTensor of shape (batch_size, entity_length), optional) —
Segment token indices to indicate first and second portions of the entity token inputs. Indices are
selected in [0, 1]:
0 corresponds to a portion A entity token,
1 corresponds to a portion B entity token.
   entity_position_ids (torch.LongTensor of shape (batch_size, entity_le


LukeForSequenceClassification

ngth, max_mention_length), optional) —
Indices of positions of each input entity in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more det


LukeForSequenceClassification

ail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-Entropy).   Returns
transformers.models.luke.modeling_luke.LukeSequenceClassifierOutput or tuple(torch.FloatTensor)

A transformers.models.luke.modeling_luke.LukeSequenceClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (LukeConfig


LukeForSequenceClassification

) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
entity_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the outpu


LukeForSequenceClassification

t of the embeddings + one for the output of each layer) of
shape (batch_size, entity_length, hidden_size). Entity hidden-states of the model at the output of each
layer plus the initial entity embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The LukeForSequenceClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example of single-l


LukeForSequenceClassification

abel classification:   Copied >>> import torch
>>> from transformers import AutoTokenizer, LukeForSequenceClassification
>>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-base")
>>> model = LukeForSequenceClassification.from_pretrained("studio-ousia/luke-base")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_class_id = logits.argmax().item()
>>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`
>>> num_labels = len(model.config.id2label)
>>> model = LukeForSequenceClassification.from_pretrained("studio-ousia/luke-base", num_labels=num_labels)
>>> labels = torch.tensor([1])
>>> loss = model(**inputs, labels=labels).loss  Example of multi-label classification:   Copied >>> import torch
>>> from transformers import AutoTokenizer, LukeForSequenceClassification
>>>


LukeForSequenceClassification

 tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-base")
>>> model = LukeForSequenceClassification.from_pretrained("studio-ousia/luke-base", problem_type="multi_label_classification")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_class_ids = torch.arange(0, logits.shape[-1])[torch.sigmoid(logits).squeeze(dim=0) > 0.5]
>>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`
>>> num_labels = len(model.config.id2label)
>>> model = LukeForSequenceClassification.from_pretrained(
...     "studio-ousia/luke-base", num_labels=num_labels, problem_type="multi_label_classification"
... )
>>> labels = torch.sum(
...     torch.nn.functional.one_hot(predicted_class_ids[None, :].clone(), num_classes=num_labels), dim=1
... ).to(torch.float)
>>> loss = model(**inputs, labels=labe


LukeForMultipleChoice

class transformers.LukeForMultipleChoice  < source > ( config  )   Parameters   config (LukeConfig) — Model configuration class with all the parameters of the
model. Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The LUKE Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a
softmax) e.g. for RocStories/SWAG tasks. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( in


LukeForMultipleChoice

put_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.LongTensor] = None entity_ids: typing.Optional[torch.LongTensor] = None entity_attention_mask: typing.Optional[torch.FloatTensor] = None entity_token_type_ids: typing.Optional[torch.LongTensor] = None entity_position_ids: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.luke.modeling_luke.LukeMultipleChoiceModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, nu


LukeForMultipleChoice

m_choices, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, num_choices, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, num_choices, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, num_choices, sequence_length), optional) —
Indices of positions


LukeForMultipleChoice

 of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   entity_ids (torch.LongTensor of shape (batch_size, entity_length)) —
Indices of entity tokens in the entity vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.   entity_attention_mask (torch.FloatTensor of shape (batch_size, entity_length), optional) —
Mask to avoid performing attention on padding entity token indices. Mask values selected in [0, 1]:
1 for entity tokens that are not masked,
0 for entity tokens that are masked.
   entity_token_type_ids (torch.LongTensor of shape (batch_size, entity_length), optional) —
Segment token indices to indicate first and second portions of the entity token inputs. Indices are
selected in [0, 1]:
0 corresponds to a portion A entity token,
1 corresponds to


LukeForMultipleChoice

a portion B entity token.
   entity_position_ids (torch.LongTensor of shape (batch_size, entity_length, max_mention_length), optional) —
Indices of positions of each input entity in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].   inputs_embeds (torch.FloatTensor of shape (batch_size, num_choices, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   output_attentions (bool, optional) —
W


LukeForMultipleChoice

hether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the multiple choice classification loss. Indices should be in [0, ..., num_choices-1] where num_choices is the size of the second dimension of the input tensors. (See
input_ids above)   Returns
transformers.models.luke.modeling_luke.LukeMultipleChoiceModelOutput or tuple(torch.FloatTensor)

A transformers.models.luke.modeling_luke.LukeMultipleChoiceModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) co


LukeForMultipleChoice

mprising various
elements depending on the configuration (LukeConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification loss.
logits (torch.FloatTensor of shape (batch_size, num_choices)) — num_choices is the second dimension of the input tensors. (see input_ids above).
Classification scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
entity_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.


LukeForMultipleChoice

output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, entity_length, hidden_size). Entity hidden-states of the model at the output of each
layer plus the initial entity embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The LukeForMultipleChoice forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post


LukeForMultipleChoice

processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, LukeForMultipleChoice
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-base")
>>> model = LukeForMultipleChoice.from_pretrained("studio-ousia/luke-base")
>>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
>>> choice0 = "It is eaten with a fork and a knife."
>>> choice1 = "It is eaten while held in the hand."
>>> labels = torch.tensor(0).unsqueeze(0)  # choice0 is correct (according to Wikipedia ;)), batch size 1
>>> encoding = tokenizer([prompt, prompt], [choice0, choice1], return_tensors="pt", padding=True)
>>> outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()}, labels=labels)  # batch size is 1
>>> # the linear classifier still needs to be trained
>>> loss = outputs.loss
>>> logits = outputs.logits


LukeForTokenClassification

class transformers.LukeForTokenClassification  < source > ( config  )   Parameters   config (LukeConfig) — Model configuration class with all the parameters of the
model. Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The LUKE Model with a token classification head on top (a linear layer on top of the hidden-states output). To
solve Named-Entity Recognition (NER) task using LUKE, LukeForEntitySpanClassification is more suitable than this
class. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentat


LukeForTokenClassification

ion for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.LongTensor] = None entity_ids: typing.Optional[torch.LongTensor] = None entity_attention_mask: typing.Optional[torch.FloatTensor] = None entity_token_type_ids: typing.Optional[torch.LongTensor] = None entity_position_ids: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.luke.modeling_luke.LukeTokenClassifierOutput or tuple(tor


LukeForTokenClassification

ch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length),


LukeForTokenClassification

 optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   entity_ids (torch.LongTensor of shape (batch_size, entity_length)) —
Indices of entity tokens in the entity vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.   entity_attention_mask (torch.FloatTensor of shape (batch_size, entity_length), optional) —
Mask to avoid performing attention on padding entity token indices. Mask values selected in [0, 1]:
1 for entity tokens that are not masked,
0 for entity tokens that are masked.
   entity_token_type_ids (torch.LongTensor of shape (batch_size, entity_length), optional) —
Segment token indices to indicate first and second portions of the entity token inputs. Indices are
selected in [0, 1]:
0 corresponds to a porti


LukeForTokenClassification

on A entity token,
1 corresponds to a portion B entity token.
   entity_position_ids (torch.LongTensor of shape (batch_size, entity_length, max_mention_length), optional) —
Indices of positions of each input entity in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   output_atten


LukeForTokenClassification

tions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the multiple choice classification loss. Indices should be in [0, ..., num_choices-1] where num_choices is the size of the second dimension of the input tensors. (See
input_ids above)   Returns
transformers.models.luke.modeling_luke.LukeTokenClassifierOutput or tuple(torch.FloatTensor)

A transformers.models.luke.modeling_luke.LukeTokenClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.


LukeForTokenClassification

return_dict=False) comprising various
elements depending on the configuration (LukeConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided)  — Classification loss.
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.num_labels)) — Classification scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
entity_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tupl


LukeForTokenClassification

e of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, entity_length, hidden_size). Entity hidden-states of the model at the output of each
layer plus the initial entity embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The LukeForTokenClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
th


LukeForTokenClassification

e latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, LukeForTokenClassification
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-base")
>>> model = LukeForTokenClassification.from_pretrained("studio-ousia/luke-base")
>>> inputs = tokenizer(
...     "HuggingFace is a company based in Paris and New York", add_special_tokens=False, return_tensors="pt"
... )
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_token_class_ids = logits.argmax(-1)
>>> # Note that tokens are classified rather then input words which means that
>>> # there might be more predicted token classes than words.
>>> # Multiple token classes might account for the same word
>>> predicted_tokens_classes = [model.config.id2label[t.item()] for t in predicted_token_class_ids[0]]
>>> labels = predicted_token_class_ids
>>> loss = model(**inputs, labels=labe


LukeForQuestionAnswering

class transformers.LukeForQuestionAnswering  < source > ( config  )   Parameters   config (LukeConfig) — Model configuration class with all the parameters of the
model. Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The LUKE Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear
layers on top of the hidden-states output to compute span start logits and span end logits). This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids:


LukeForQuestionAnswering

typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.FloatTensor] = None entity_ids: typing.Optional[torch.LongTensor] = None entity_attention_mask: typing.Optional[torch.FloatTensor] = None entity_token_type_ids: typing.Optional[torch.LongTensor] = None entity_position_ids: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None start_positions: typing.Optional[torch.LongTensor] = None end_positions: typing.Optional[torch.LongTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.luke.modeling_luke.LukeQuestionAnsweringModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, seque


LukeForQuestionAnswering

nce_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings -


LukeForQuestionAnswering

1].
What are position IDs?   entity_ids (torch.LongTensor of shape (batch_size, entity_length)) —
Indices of entity tokens in the entity vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.   entity_attention_mask (torch.FloatTensor of shape (batch_size, entity_length), optional) —
Mask to avoid performing attention on padding entity token indices. Mask values selected in [0, 1]:
1 for entity tokens that are not masked,
0 for entity tokens that are masked.
   entity_token_type_ids (torch.LongTensor of shape (batch_size, entity_length), optional) —
Segment token indices to indicate first and second portions of the entity token inputs. Indices are
selected in [0, 1]:
0 corresponds to a portion A entity token,
1 corresponds to a portion B entity token.
   entity_position_ids (torch.LongTensor of shape (batch_size, entity_length, max_mention_length), optional) —
Indices of positions of each input entity in


LukeForQuestionAnswering

 the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under return


LukeForQuestionAnswering

ed tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   start_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the start of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
are not taken into account for computing the loss.   end_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the end of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
are not taken into account for computing the loss.   Returns
transformers.models.luke.modeling_luke.LukeQuestionAnsweringModelOutput or tuple(torch.FloatTensor)

A transformers.models.luke.modeling_luke.LukeQuestionAnsweringModelOutput or a tuple of
torc


LukeForQuestionAnswering

h.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (LukeConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Total span extraction loss is the sum of a Cross-Entropy for the start and end positions.
start_logits (torch.FloatTensor of shape (batch_size, sequence_length)) — Span-start scores (before SoftMax).
end_logits (torch.FloatTensor of shape (batch_size, sequence_length)) — Span-end scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
enti


LukeForQuestionAnswering

ty_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, entity_length, hidden_size). Entity hidden-states of the model at the output of each
layer plus the initial entity embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The LukeForQuestionAnswering forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of th


LukeForQuestionAnswering

is since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, LukeForQuestionAnswering
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-base")
>>> model = LukeForQuestionAnswering.from_pretrained("studio-ousia/luke-base")
>>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
>>> inputs = tokenizer(question, text, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> answer_start_index = outputs.start_logits.argmax()
>>> answer_end_index = outputs.end_logits.argmax()
>>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
>>> # target is "nice puppet"
>>> target_start_index = torch.tensor([14])
>>> target_end_index = torch.tensor([15])
>>> outputs = model(**inputs, start_positions=target_start_index, end_positions=target_end_index)
>>> loss = outputs


Overview

Dense Passage Retrieval (DPR) is a set of tools and models for state-of-the-art open-domain Q&A research. It was
introduced in Dense Passage Retrieval for Open-Domain Question Answering by
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih.
The abstract from the paper is the following:
Open-domain question answering relies on efficient passage retrieval to select candidate contexts, where traditional
sparse vector space models, such as TF-IDF or BM25, are the de facto method. In this work, we show that retrieval can
be practically implemented using dense representations alone, where embeddings are learned from a small number of
questions a


Overview

nd passages by a simple dual-encoder framework. When evaluated on a wide range of open-domain QA datasets,
our dense retriever outperforms a strong Lucene-BM25 system largely by 9%-19% absolute in terms of top-20 passage
retrieval accuracy, and helps our end-to-end QA system establish new state-of-the-art on multiple open-domain QA
benchmarks.
This model was contributed by lhoestq. The original code can be found here.
Tips:
DPR consists in three models: Question encoder: encode questions as vectors Context encoder: encode contexts as vectors Reader: extract the answer of the questions inside retrieved contexts, along with a relevance score (high if the inferred span actually answers the question)


DPRConfig

class transformers.DPRConfig  < source > ( vocab_size = 30522 hidden_size = 768 num_hidden_layers = 12 num_attention_heads = 12 intermediate_size = 3072 hidden_act = 'gelu' hidden_dropout_prob = 0.1 attention_probs_dropout_prob = 0.1 max_position_embeddings = 512 type_vocab_size = 2 initializer_range = 0.02 layer_norm_eps = 1e-12 pad_token_id = 0 position_embedding_type = 'absolute' projection_dim: int = 0 **kwargs  )   Parameters   vocab_size (int, optional, defaults to 30522) —
Vocabulary size of the DPR model. Defines the different tokens that can be represented by the inputs_ids
passed to the forward method of BertModel.   hidden_size (int, optional, defaults to 768) —
Dimensionality of the encoder layers and the pooler layer.   num_hidden_layers (int, optional, defaults to 12) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 12) —
Number of attention heads for each attention layer in the Transformer enco


DPRConfig

der.   intermediate_size (int, optional, defaults to 3072) —
Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.   hidden_act (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "silu" and "gelu_new" are supported.   hidden_dropout_prob (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.   attention_probs_dropout_prob (float, optional, defaults to 0.1) —
The dropout ratio for the attention probabilities.   max_position_embeddings (int, optional, defaults to 512) —
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).   type_vocab_size (int, optional, defaults to 2) —
The vocabulary size of the token_type_ids passed into BertModel.   initializer_r


DPRConfig

ange (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   layer_norm_eps (float, optional, defaults to 1e-12) —
The epsilon used by the layer normalization layers.   position_embedding_type (str, optional, defaults to "absolute") —
Type of position embedding. Choose one of "absolute", "relative_key", "relative_key_query". For
positional embeddings use "absolute". For more information on "relative_key", please refer to
Self-Attention with Relative Position Representations (Shaw et al.).
For more information on "relative_key_query", please refer to Method 4 in Improve Transformer Models
with Better Relative Position Embeddings (Huang et al.).   projection_dim (int, optional, defaults to 0) —
Dimension of the projection for the context and question encoders. If it is set to zero (default), then no
projection is done.    DPRConfig is the configuration class to store the configuration


DPRConfig

of a DPRModel. This is the configuration class to store the configuration of a DPRContextEncoder, DPRQuestionEncoder, or a
DPRReader. It is used to instantiate the components of the DPR model according to the specified arguments,
defining the model component architectures. Instantiating a configuration with the defaults will yield a similar
configuration to that of the DPRContextEncoder
facebook/dpr-ctx_encoder-single-nq-base
architecture. This class is a subclass of BertConfig. Please check the superclass for the documentation of all kwargs.  Example:   Copied >>> from transformers import DPRConfig, DPRContextEncoder
>>> # Initializing a DPR facebook/dpr-ctx_encoder-single-nq-base style configuration
>>> configuration = DPRConfig()
>>> # Initializing a model (with random weights) from the facebook/dpr-ctx_encoder-single-nq-base style configuration
>>> model = DPRContextEncoder(configuration)
>>> # Accessing the model configuration
>>> configuration = model.confi


DPRContextEncoderTokenizer

class transformers.DPRContextEncoderTokenizer  < source > ( vocab_file do_lower_case = True do_basic_tokenize = True never_split = None unk_token = '[UNK]' sep_token = '[SEP]' pad_token = '[PAD]' cls_token = '[CLS]' mask_token = '[MASK]' tokenize_chinese_chars = True strip_accents = None **kwargs  )    Construct a DPRContextEncoder tokenizer. DPRContextEncoderTokenizer is identical to BertTokenizer and runs end-to-end tokenization: punctuation
splitting and wordpiece. Refer to superclass BertTokenizer for usage examples and documentation concerning parameters.


DPRContextEncoderTokenizerFast

class transformers.DPRContextEncoderTokenizerFast  < source > ( vocab_file = None tokenizer_file = None do_lower_case = True unk_token = '[UNK]' sep_token = '[SEP]' pad_token = '[PAD]' cls_token = '[CLS]' mask_token = '[MASK]' tokenize_chinese_chars = True strip_accents = None **kwargs  )    Construct a “fast” DPRContextEncoder tokenizer (backed by HuggingFace’s tokenizers library). DPRContextEncoderTokenizerFast is identical to BertTokenizerFast and runs end-to-end tokenization:
punctuation splitting and wordpiece. Refer to superclass BertTokenizerFast for usage examples and documentation concerning parameters.


DPRQuestionEncoderTokenizer

class transformers.DPRQuestionEncoderTokenizer  < source > ( vocab_file do_lower_case = True do_basic_tokenize = True never_split = None unk_token = '[UNK]' sep_token = '[SEP]' pad_token = '[PAD]' cls_token = '[CLS]' mask_token = '[MASK]' tokenize_chinese_chars = True strip_accents = None **kwargs  )    Constructs a DPRQuestionEncoder tokenizer. DPRQuestionEncoderTokenizer is identical to BertTokenizer and runs end-to-end tokenization: punctuation
splitting and wordpiece. Refer to superclass BertTokenizer for usage examples and documentation concerning parameters.


DPRQuestionEncoderTokenizerFast

class transformers.DPRQuestionEncoderTokenizerFast  < source > ( vocab_file = None tokenizer_file = None do_lower_case = True unk_token = '[UNK]' sep_token = '[SEP]' pad_token = '[PAD]' cls_token = '[CLS]' mask_token = '[MASK]' tokenize_chinese_chars = True strip_accents = None **kwargs  )    Constructs a “fast” DPRQuestionEncoder tokenizer (backed by HuggingFace’s tokenizers library). DPRQuestionEncoderTokenizerFast is identical to BertTokenizerFast and runs end-to-end tokenization:
punctuation splitting and wordpiece. Refer to superclass BertTokenizerFast for usage examples and documentation concerning parameters.


DPRReaderTokenizer

class transformers.DPRReaderTokenizer  < source > ( vocab_file do_lower_case = True do_basic_tokenize = True never_split = None unk_token = '[UNK]' sep_token = '[SEP]' pad_token = '[PAD]' cls_token = '[CLS]' mask_token = '[MASK]' tokenize_chinese_chars = True strip_accents = None **kwargs  ) → Dict[str, List[List[int]]]  Parameters   questions (str or List[str]) —
The questions to be encoded. You can specify one question for many passages. In this case, the question
will be duplicated like [questions] * n_passages. Otherwise you have to specify as many questions as in
titles or texts.   titles (str or List[str]) —
The passages titles to be encoded. This can be a string or a list of strings if there are several passages.   texts (str or List[str]) —
The passages texts to be encoded. This can be a string or a list of strings if there are several passages.   padding (bool, str or PaddingStrateg


DPRReaderTokenizer

y, optional, defaults to False) —
Activates and controls padding. Accepts the following values:
True or 'longest': Pad to the longest sequence in the batch (or no padding if only a single sequence
if provided).
'max_length': Pad to a maximum length specified with the argument max_length or to the maximum
acceptable input length for the model if that argument is not provided.
False or 'do_not_pad' (default): No padding (i.e., can output a batch with sequences of different
lengths).
   truncation (bool, str or TruncationStrategy, optional, defaults to False) —
Activates and controls truncation. Accepts the following values:
True or 'longest_first': Truncate to a maximum length specified with the argument max_length or to
the maximum acceptable input length for the model if that argument is not provided. This will truncate
token by token, removing a token from the longest sequence in the pair i


DPRReaderTokenizer

f a pair of sequences (or a batch
of pairs) is provided.
'only_first': Truncate to a maximum length specified with the argument max_length or to the maximum
acceptable input length for the model if that argument is not provided. This will only truncate the first
sequence of a pair if a pair of sequences (or a batch of pairs) is provided.
'only_second': Truncate to a maximum length specified with the argument max_length or to the maximum
acceptable input length for the model if that argument is not provided. This will only truncate the
second sequence of a pair if a pair of sequences (or a batch of pairs) is provided.
False or 'do_not_truncate' (default): No truncation (i.e., can output batch with sequence lengths
greater than the model maximum admissible input size).
   max_length (int, optional) —
Controls the maximum length to use by one of the truncation/padding parameters.
If left unset


DPRReaderTokenizer

or set to None, this will use the predefined model maximum length if a maximum length
is required by one of the truncation/padding parameters. If the model has no specific maximum input
length (like XLNet) truncation/padding to a maximum length will be deactivated.   return_tensors (str or TensorType, optional) —
If set, will return tensors instead of list of python integers. Acceptable values are:
'tf': Return TensorFlow tf.constant objects.
'pt': Return PyTorch torch.Tensor objects.
'np': Return Numpy np.ndarray objects.
   return_attention_mask (bool, optional) —
Whether or not to return the attention mask. If not set, will return the attention mask according to the
specific tokenizer’s default, defined by the return_outputs attribute.
What are attention masks?   Returns
Dict[str, List[List[int]]]

A dictionary with the following keys:
input_ids: List of token ids to be fed to a model.


DPRReaderTokenizer

attention_mask: List of indices specifying which tokens should be attended to by the model.
  Construct a DPRReader tokenizer. DPRReaderTokenizer is almost identical to BertTokenizer and runs end-to-end tokenization: punctuation
splitting and wordpiece. The difference is that is has three inputs strings: question, titles and texts that are
combined to be fed to the DPRReader model. Refer to superclass BertTokenizer for usage examples and documentation concerning parameters. Return a dictionary with the token ids of the input strings and other information to give to .decode_best_spans.
It converts the strings of a question and different passages (title and text) in a sequence of IDs (integers),
using the tokenizer and vocabulary. The resulting input_ids is a matrix of size (n_passages, sequence_length)  with the format:   Copied [CLS] <question token ids> [SEP] <titles ids> [SEP] <texts ids>


DPRReaderTokenizerFast

class transformers.DPRReaderTokenizerFast  < source > ( vocab_file = None tokenizer_file = None do_lower_case = True unk_token = '[UNK]' sep_token = '[SEP]' pad_token = '[PAD]' cls_token = '[CLS]' mask_token = '[MASK]' tokenize_chinese_chars = True strip_accents = None **kwargs  ) → Dict[str, List[List[int]]]  Parameters   questions (str or List[str]) —
The questions to be encoded. You can specify one question for many passages. In this case, the question
will be duplicated like [questions] * n_passages. Otherwise you have to specify as many questions as in
titles or texts.   titles (str or List[str]) —
The passages titles to be encoded. This can be a string or a list of strings if there are several passages.   texts (str or List[str]) —
The passages texts to be encoded. This can be a string or a list of strings if there are several passages.   padding (bool, str or PaddingStrategy, optional, defaul


DPRReaderTokenizerFast

ts to False) —
Activates and controls padding. Accepts the following values:
True or 'longest': Pad to the longest sequence in the batch (or no padding if only a single sequence
if provided).
'max_length': Pad to a maximum length specified with the argument max_length or to the maximum
acceptable input length for the model if that argument is not provided.
False or 'do_not_pad' (default): No padding (i.e., can output a batch with sequences of different
lengths).
   truncation (bool, str or TruncationStrategy, optional, defaults to False) —
Activates and controls truncation. Accepts the following values:
True or 'longest_first': Truncate to a maximum length specified with the argument max_length or to
the maximum acceptable input length for the model if that argument is not provided. This will truncate
token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a


DPRReaderTokenizerFast

 batch
of pairs) is provided.
'only_first': Truncate to a maximum length specified with the argument max_length or to the maximum
acceptable input length for the model if that argument is not provided. This will only truncate the first
sequence of a pair if a pair of sequences (or a batch of pairs) is provided.
'only_second': Truncate to a maximum length specified with the argument max_length or to the maximum
acceptable input length for the model if that argument is not provided. This will only truncate the
second sequence of a pair if a pair of sequences (or a batch of pairs) is provided.
False or 'do_not_truncate' (default): No truncation (i.e., can output batch with sequence lengths
greater than the model maximum admissible input size).
   max_length (int, optional) —
Controls the maximum length to use by one of the truncation/padding parameters.
If left unset or set to None, this will use the p


DPRReaderTokenizerFast

redefined model maximum length if a maximum length
is required by one of the truncation/padding parameters. If the model has no specific maximum input
length (like XLNet) truncation/padding to a maximum length will be deactivated.   return_tensors (str or TensorType, optional) —
If set, will return tensors instead of list of python integers. Acceptable values are:
'tf': Return TensorFlow tf.constant objects.
'pt': Return PyTorch torch.Tensor objects.
'np': Return Numpy np.ndarray objects.
   return_attention_mask (bool, optional) —
Whether or not to return the attention mask. If not set, will return the attention mask according to the
specific tokenizer’s default, defined by the return_outputs attribute.
What are attention masks?   Returns
Dict[str, List[List[int]]]

A dictionary with the following keys:
input_ids: List of token ids to be fed to a model.
attention_mask: List of indices specifying


DPRReaderTokenizerFast

 which tokens should be attended to by the model.
  Constructs a “fast” DPRReader tokenizer (backed by HuggingFace’s tokenizers library). DPRReaderTokenizerFast is almost identical to BertTokenizerFast and runs end-to-end tokenization:
punctuation splitting and wordpiece. The difference is that is has three inputs strings: question, titles and texts
that are combined to be fed to the DPRReader model. Refer to superclass BertTokenizerFast for usage examples and documentation concerning parameters. Return a dictionary with the token ids of the input strings and other information to give to .decode_best_spans.
It converts the strings of a question and different passages (title and text) in a sequence of IDs (integers),
using the tokenizer and vocabulary. The resulting input_ids is a matrix of size (n_passages, sequence_length)
with the format: [CLS] <question token ids> [SEP] <titles ids> [SEP] <texts


DPR specific outputs

class transformers.models.dpr.modeling_dpr.DPRContextEncoderOutput  < source > ( pooler_output: FloatTensor hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None attentions: typing.Optional[typing.Tuple[torch.FloatTensor]] = None  )   Parameters   pooler_output (torch.FloatTensor of shape (batch_size, embeddings_size)) —
The DPR encoder outputs the pooler_output that corresponds to the context representation. Last layer
hidden-state of the first token of the sequence (classification token) further processed by a Linear layer.
This output is to be used to embed contexts for nearest neighbors queries with questions embeddings.   hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer)


DPR specific outputs

of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.   attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) —
Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.    Class for outputs of DPRQuestionEncoder.
 class transformers.models.dpr.modeling_dpr.DPRQuestionEncoderOutput  < source > ( pooler_output: FloatTensor hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None attentions: typing.Optional[typing.Tuple[torch.FloatTensor]] = None  )   Parameters   pooler_output (torch.FloatTensor of shape (batch_size, embeddings_size)) —


DPR specific outputs

The DPR encoder outputs the pooler_output that corresponds to the question representation. Last layer
hidden-state of the first token of the sequence (classification token) further processed by a Linear layer.
This output is to be used to embed questions for nearest neighbors queries with context embeddings.   hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.   attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) —
Tuple of torch.FloatTensor (one for each layer) of shape (


DPR specific outputs

batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.    Class for outputs of DPRQuestionEncoder.
 class transformers.DPRReaderOutput  < source > ( start_logits: FloatTensor end_logits: FloatTensor = None relevance_logits: FloatTensor = None hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None attentions: typing.Optional[typing.Tuple[torch.FloatTensor]] = None  )   Parameters   start_logits (torch.FloatTensor of shape (n_passages, sequence_length)) —
Logits of the start index of the span for each passage.   end_logits (torch.FloatTensor of shape (n_passages, sequence_length)) —
Logits of the end index of the span for each passage.   relevance_logits (torch.FloatTensor of shape (n_passages, )) —
Outputs of the QA classifier of the DPRReader that corresp


DPR specific outputs

onds to the scores of each passage to answer the
question, compared to all the other passages.   hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.   attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) —
Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.    Class for outputs of DPRQuestionEncoder.


DPRContextEncoder

class transformers.DPRContextEncoder  < source > ( config: DPRConfig  )   Parameters   config (DPRConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare DPRContextEncoder transformer outputting pooler outputs as context representations. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Ten


DPRContextEncoder

sor] = None token_type_ids: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.dpr.modeling_dpr.DPRContextEncoderOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. To match pretraining, DPR input sequence should be
formatted with [CLS] and [SEP] tokens as follows:
(a) For sequence pairs (for a pair title+text for example):   Returns
transformers.models.dpr.modeling_dpr.DPRContextEncoderOutput or tuple(torch.FloatTensor)

A transformers.models.dpr.modeling_dpr.DPRContextEncoderOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the confi


DPRContextEncoder

guration (DPRConfig) and inputs.
pooler_output (torch.FloatTensor of shape (batch_size, embeddings_size)) — The DPR encoder outputs the pooler_output that corresponds to the context representation. Last layer
hidden-state of the first token of the sequence (classification token) further processed by a Linear layer.
This output is to be used to embed contexts for nearest neighbors queries with questions embeddings.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.Floa


DPRContextEncoder

tTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The DPRContextEncoder forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
>>> tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
>>> model = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
>>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors="pt")["input_ids"]
>>> embeddings = model(input_ids).pooler_outp


DPRQuestionEncoder

class transformers.DPRQuestionEncoder  < source > ( config: DPRConfig  )   Parameters   config (DPRConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare DPRQuestionEncoder transformer outputting pooler outputs as question representations. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tenso


DPRQuestionEncoder

r] = None token_type_ids: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.dpr.modeling_dpr.DPRQuestionEncoderOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. To match pretraining, DPR input sequence should be
formatted with [CLS] and [SEP] tokens as follows:
(a) For sequence pairs (for a pair title+text for example):   Returns
transformers.models.dpr.modeling_dpr.DPRQuestionEncoderOutput or tuple(torch.FloatTensor)

A transformers.models.dpr.modeling_dpr.DPRQuestionEncoderOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configura


DPRQuestionEncoder

tion (DPRConfig) and inputs.
pooler_output (torch.FloatTensor of shape (batch_size, embeddings_size)) — The DPR encoder outputs the pooler_output that corresponds to the question representation. Last layer
hidden-state of the first token of the sequence (classification token) further processed by a Linear layer.
This output is to be used to embed questions for nearest neighbors queries with context embeddings.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (


DPRQuestionEncoder

one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The DPRQuestionEncoder forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import DPRQuestionEncoder, DPRQuestionEncoderTokenizer
>>> tokenizer = DPRQuestionEncoderTokenizer.from_pretrained("facebook/dpr-question_encoder-single-nq-base")
>>> model = DPRQuestionEncoder.from_pretrained("facebook/dpr-question_encoder-single-nq-base")
>>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors="pt")["input_ids"]
>>> embeddings = model(input_ids).pooler_out


DPRReader

class transformers.DPRReader  < source > ( config: DPRConfig  )   Parameters   config (DPRConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare DPRReader transformer outputting span predictions. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing


DPRReader

.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.dpr.modeling_dpr.DPRReaderOutput or tuple(torch.FloatTensor)  Parameters   input_ids (Tuple[torch.LongTensor] of shapes (n_passages, sequence_length)) —
Indices of input sequence tokens in the vocabulary. It has to be a sequence triplet with 1) the question
and 2) the passages titles and 3) the passages texts To match pretraining, DPR input_ids sequence should
be formatted with [CLS] and [SEP] with the format:
[CLS] <question token ids> [SEP] <titles ids> [SEP] <texts ids>
DPR is a model with absolute position embeddings so it’s usually advised to pad the inputs on the right
rather than the left.
Indices can be obtained using DPRReaderTokenizer.


DPRReader

See this class documentation for more details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (n_passages, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   inputs_embeds (torch.FloatTensor of shape (n_passages, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Wh


DPRReader

ether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.models.dpr.modeling_dpr.DPRReaderOutput or tuple(torch.FloatTensor)

A transformers.models.dpr.modeling_dpr.DPRReaderOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (DPRConfig) and inputs.
start_logits (torch.FloatTensor of shape (n_passages, sequence_length)) — Logits of the start index of the span for each passage.
end_logits (torch.FloatTensor of shape (n_passages, sequence_length)) — Logits of the end index of the span for each passage.
relevance_logits (torch.FloatTensor of shape (n_passages, )) — Outputs of the QA classifier of the DPRRe


DPRReader

ader that corresponds to the scores of each passage to answer the
question, compared to all the other passages.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The DPRReader forward method, ove


DPRReader

rrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import DPRReader, DPRReaderTokenizer
>>> tokenizer = DPRReaderTokenizer.from_pretrained("facebook/dpr-reader-single-nq-base")
>>> model = DPRReader.from_pretrained("facebook/dpr-reader-single-nq-base")
>>> encoded_inputs = tokenizer(
...     questions=["What is love ?"],
...     titles=["Haddaway"],
...     texts=["'What Is Love' is a song recorded by the artist Haddaway"],
...     return_tensors="pt",
... )
>>> outputs = model(**encoded_inputs)
>>> start_logits = outputs.start_logits
>>> end_logits = outputs.end_logits
>>> relevance_logits = outputs.relevance_log


TFDPRContextEncoder

class transformers.TFDPRContextEncoder  < source > ( *args **kwargs  )   Parameters   config (DPRConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare DPRContextEncoder transformer outputting pooler outputs as context representations. This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a Tensorflow tf.keras.Model
subclass. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to
general usage and behavior. TensorF


TFDPRContextEncoder

low models and layers in transformers accept two formats as input: having all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a l


TFDPRContextEncoder

ist of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_mask]) or model([input_ids, attention_mask, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: tf.Tensor | None = None token_type_ids: tf.Tensor | None = None inputs_embeds: tf.Tensor | None = None output_attentions: bool | None = None output_hidden_states: bool | None = None return_dict: bool | None = None training: bool = False  ) → transformers.mo


TFDPRContextEncoder

dels.dpr.modeling_tf_dpr.TFDPRContextEncoderOutput or tuple(tf.Tensor)  Parameters   input_ids (Numpy array or tf.Tensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. To match pretraining, DPR input sequence should be
formatted with [CLS] and [SEP] tokens as follows:
(a) For sequence pairs (for a pair title+text for example):   Returns
transformers.models.dpr.modeling_tf_dpr.TFDPRContextEncoderOutput or tuple(tf.Tensor)

A transformers.models.dpr.modeling_tf_dpr.TFDPRContextEncoderOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (DPRConfig) and inputs.
pooler_output (tf.Tensor of shape (batch_size, embeddings_size)) — The DPR encoder outputs the pooler_output that corresponds to the context repres


TFDPRContextEncoder

entation. Last layer
hidden-state of the first token of the sequence (classification token) further processed by a Linear layer.
This output is to be used to embed contexts for nearest neighbors queries with questions embeddings.
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after


TFDPRContextEncoder

 the attention softmax, used to compute the weighted average in the self-attention
heads.
  The TFDPRContextEncoder forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import TFDPRContextEncoder, DPRContextEncoderTokenizer
>>> tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
>>> model = TFDPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base", from_pt=True)
>>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors="tf")["input_ids"]
>>> embeddings = model(input_ids).pooler_output


TFDPRQuestionEncoder

class transformers.TFDPRQuestionEncoder  < source > ( *args **kwargs  )   Parameters   config (DPRConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare DPRQuestionEncoder transformer outputting pooler outputs as question representations. This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a Tensorflow tf.keras.Model
subclass. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to
general usage and behavior. TensorF


TFDPRQuestionEncoder

low models and layers in transformers accept two formats as input: having all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a list


TFDPRQuestionEncoder

 of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_mask]) or model([input_ids, attention_mask, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: tf.Tensor | None = None token_type_ids: tf.Tensor | None = None inputs_embeds: tf.Tensor | None = None output_attentions: bool | None = None output_hidden_states: bool | None = None return_dict: bool | None = None training: bool = False  ) → transformers.models.d


TFDPRQuestionEncoder

pr.modeling_tf_dpr.TFDPRQuestionEncoderOutput or tuple(tf.Tensor)  Parameters   input_ids (Numpy array or tf.Tensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. To match pretraining, DPR input sequence should be
formatted with [CLS] and [SEP] tokens as follows:
(a) For sequence pairs (for a pair title+text for example):   Returns
transformers.models.dpr.modeling_tf_dpr.TFDPRQuestionEncoderOutput or tuple(tf.Tensor)

A transformers.models.dpr.modeling_tf_dpr.TFDPRQuestionEncoderOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (DPRConfig) and inputs.
pooler_output (tf.Tensor of shape (batch_size, embeddings_size)) — The DPR encoder outputs the pooler_output that corresponds to the question representat


TFDPRQuestionEncoder

ion. Last layer
hidden-state of the first token of the sequence (classification token) further processed by a Linear layer.
This output is to be used to embed questions for nearest neighbors queries with context embeddings.
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the atte


TFDPRQuestionEncoder

ntion softmax, used to compute the weighted average in the self-attention
heads.
  The TFDPRQuestionEncoder forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import TFDPRQuestionEncoder, DPRQuestionEncoderTokenizer
>>> tokenizer = DPRQuestionEncoderTokenizer.from_pretrained("facebook/dpr-question_encoder-single-nq-base")
>>> model = TFDPRQuestionEncoder.from_pretrained("facebook/dpr-question_encoder-single-nq-base", from_pt=True)
>>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors="tf")["input_ids"]
>>> embeddings = model(input_ids).pooler_out


TFDPRReader

class transformers.TFDPRReader  < source > ( *args **kwargs  )   Parameters   config (DPRConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare DPRReader transformer outputting span predictions. This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a Tensorflow tf.keras.Model
subclass. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to
general usage and behavior. TensorFlow models and layers in transformers accept two formats as input: having all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dic


TFDPRReader

t in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_mask]) or model([input_ids, attention_mask, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_ids": in


TFDPRReader

put_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: tf.Tensor | None = None inputs_embeds: tf.Tensor | None = None output_attentions: bool | None = None output_hidden_states: bool | None = None return_dict: bool | None = None training: bool = False  ) → transformers.models.dpr.modeling_tf_dpr.TFDPRReaderOutput or tuple(tf.Tensor)  Parameters   input_ids (Numpy array or tf.Tensor of shapes (n_passages, sequence_length)) —
Indices of input sequence tokens in the vocabulary. It has to be a sequence triplet with 1) the question
and 2) the passages titles and 3) the passages texts To match pretraining, DPR input_ids sequence should
be formatted with [CLS] and [SEP] with the format:
[CLS] <question token ids> [SEP] <titles ids> [SEP] <texts i


TFDPRReader

ds>
DPR is a model with absolute position embeddings so it’s usually advised to pad the inputs on the right
rather than the left.
Indices can be obtained using DPRReaderTokenizer. See this class documentation for more details.   attention_mask (Numpy array or tf.Tensor of shape (n_passages, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   inputs_embeds (Numpy array or tf.Tensor of shape (n_passages, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under return


TFDPRReader

ed tensors for
more detail. This argument can be used only in eager mode, in graph mode the value in the config will be
used instead.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in
eager mode, in graph mode the value will always be set to True.   training (bool, optional, defaults to False) —
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).   Returns
transformers.models.dpr.modeling_tf_dpr.TFDPRReaderOutput or tuple(tf.Tensor)

A transformers.models.dpr.modeling_tf_dpr.TFDPRReaderOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (DPRConfig) and inputs.
start_logits (tf.Tensor of shape (n_passages, sequence_length)) — Logits of the start index of the span for each passage.
end_logits (tf.Tensor of shape (


TFDPRReader

n_passages, sequence_length)) — Logits of the end index of the span for each passage.
relevance_logits (tf.Tensor of shape (n_passages, )) — Outputs of the QA classifier of the DPRReader that corresponds to the scores of each passage to answer the
question, compared to all the other passages.
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted


TFDPRReader

 average in the self-attention
heads.
  The TFDPRReader forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import TFDPRReader, DPRReaderTokenizer
>>> tokenizer = DPRReaderTokenizer.from_pretrained("facebook/dpr-reader-single-nq-base")
>>> model = TFDPRReader.from_pretrained("facebook/dpr-reader-single-nq-base", from_pt=True)
>>> encoded_inputs = tokenizer(
...     questions=["What is love ?"],
...     titles=["Haddaway"],
...     texts=["'What Is Love' is a song recorded by the artist Haddaway"],
...     return_tensors="tf",
... )
>>> outputs = model(encoded_inputs)
>>> start_logits = outputs.start_logits
>>> end_logits = outputs.end_logits
>>> relevance_logits = outputs.relevance_logit