TapasTokenizer

ication). It is the first token of the sequence when built with special tokens.   mask_token (str, optional, defaults to "[MASK]") —
The token used for masking values. This is the token used when training this model with masked language
modeling. This is the token which the model will try to predict.   empty_token (str, optional, defaults to "[EMPTY]") —
The token used for empty cell values in a table. Empty cell values include "", “n/a”, “nan” and ”?“.   tokenize_chinese_chars (bool, optional, defaults to True) —
Whether or not to tokenize Chinese characters. This should likely be deactivated for Japanese (see this
issue).   strip_accents (bool, optional) —
Whether or not to strip all accents. If this option is not specified, then it will be determined by the
value for lowercase (as in the original BERT).   cell_trim_length (int, optional, defaults to -1) —
If > 0: Trim cells so that the length is <= this value. Also


TapasTokenizer

 disables further cell trimming, should thus be
used with truncation set to True.   max_column_id (int, optional) —
Max column id to extract.   max_row_id (int, optional) —
Max row id to extract.   strip_column_names (bool, optional, defaults to False) —
Whether to add empty strings instead of column names.   update_answer_coordinates (bool, optional, defaults to False) —
Whether to recompute the answer coordinates from the answer text.   min_question_length (int, optional) —
Minimum length of each question in terms of tokens (will be skipped otherwise).   max_question_length (int, optional) —
Maximum length of each question in terms of tokens (will be skipped otherwise).    Construct a TAPAS tokenizer. Based on WordPiece. Flattens a table and one or more related sentences to be used by
TAPAS models. This tokenizer inherits from PreTrainedTokenizer which contains most of the main methods. Users should refer to
this su


TapasTokenizer

perclass for more information regarding those methods. TapasTokenizer creates several token type ids to
encode tabular structure. To be more precise, it adds 7 token type ids, in the following order: segment_ids,
column_ids, row_ids, prev_labels, column_ranks, inv_column_ranks and numeric_relations: segment_ids: indicate whether a token belongs to the question (0) or the table (1). 0 for special tokens and
padding. column_ids: indicate to which column of the table a token belongs (starting from 1). Is 0 for all question
tokens, special tokens and padding. row_ids: indicate to which row of the table a token belongs (starting from 1). Is 0 for all question tokens,
special tokens and padding. Tokens of column headers are also 0. prev_labels: indicate whether a token was (part of) an answer to the previous question (1) or not (0). Useful in
a conversational setup (such as SQA). column_ranks: indicate the rank of a table t


TapasTokenizer

oken relative to a column, if applicable. For example, if you have a
column “number of movies” with values 87, 53 and 69, then the column ranks of these tokens are 3, 1 and 2
respectively. 0 for all question tokens, special tokens and padding. inv_column_ranks: indicate the inverse rank of a table token relative to a column, if applicable. For example, if
you have a column “number of movies” with values 87, 53 and 69, then the inverse column ranks of these tokens are
1, 3 and 2 respectively. 0 for all question tokens, special tokens and padding. numeric_relations: indicate numeric relations between the question and the tokens of the table. 0 for all
question tokens, special tokens and padding. TapasTokenizer runs end-to-end tokenization on a table and associated sentences: punctuation splitting and
wordpiece.  __call__  < source > ( table: pd.DataFrame queries: typing.Union[str, typing.List[str], typing.List[int], typ


TapasTokenizer

ing.List[typing.List[str]], typing.List[typing.List[int]], NoneType] = None answer_coordinates: typing.Union[typing.List[typing.Tuple], typing.List[typing.List[typing.Tuple]], NoneType] = None answer_text: typing.Union[typing.List[str], typing.List[typing.List[str]], NoneType] = None add_special_tokens: bool = True padding: typing.Union[bool, str, transformers.utils.generic.PaddingStrategy] = False truncation: typing.Union[bool, str, transformers.models.tapas.tokenization_tapas.TapasTruncationStrategy] = False max_length: typing.Optional[int] = None pad_to_multiple_of: typing.Optional[int] = None return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None return_token_type_ids: typing.Optional[bool] = None return_attention_mask: typing.Optional[bool] = None return_overflowing_tokens: bool = False return_special_tokens_mask: bool = False return_offsets_mapping: bool = False return_length:


TapasTokenizer

bool = False verbose: bool = True **kwargs  )   Parameters   table (pd.DataFrame) —
Table containing tabular data. Note that all cell values must be text. Use .astype(str) on a Pandas
dataframe to convert it to string.   queries (str or List[str]) —
Question or batch of questions related to a table to be encoded. Note that in case of a batch, all
questions must refer to the same table.   answer_coordinates (List[Tuple] or List[List[Tuple]], optional) —
Answer coordinates of each table-question pair in the batch. In case only a single table-question pair
is provided, then the answer_coordinates must be a single list of one or more tuples. Each tuple must
be a (row_index, column_index) pair. The first data row (not the column header row) has index 0. The
first column has index 0. In case a batch of table-question pairs is provided, then the
answer_coordinates must be a list of lists of tuples (each list corresponding to


TapasTokenizer

 a single
table-question pair).   answer_text (List[str] or List[List[str]], optional) —
Answer text of each table-question pair in the batch. In case only a single table-question pair is
provided, then the answer_text must be a single list of one or more strings. Each string must be the
answer text of a corresponding answer coordinate. In case a batch of table-question pairs is provided,
then the answer_coordinates must be a list of lists of strings (each list corresponding to a single
table-question pair).   add_special_tokens (bool, optional, defaults to True) —
Whether or not to encode the sequences with the special tokens relative to their model.   padding (bool, str or PaddingStrategy, optional, defaults to False) —
Activates and controls padding. Accepts the following values:
True or 'longest': Pad to the longest sequence in the batch (or no padding if only a single
sequence if provided).
'max_length': Pad to a


TapasTokenizer

 maximum length specified with the argument max_length or to the maximum
acceptable input length for the model if that argument is not provided.
False or 'do_not_pad' (default): No padding (i.e., can output a batch with sequences of different
lengths).
   truncation (bool, str or TapasTruncationStrategy, optional, defaults to False) —
Activates and controls truncation. Accepts the following values:
True or 'drop_rows_to_fit': Truncate to a maximum length specified with the argument max_length
or to the maximum acceptable input length for the model if that argument is not provided. This will
truncate row by row, removing rows from the table.
False or 'do_not_truncate' (default): No truncation (i.e., can output batch with sequence lengths
greater than the model maximum admissible input size).
   max_length (int, optional) —
Controls the maximum length to use by one of the truncation/padding parameters.
If left unset or


TapasTokenizer

set to None, this will use the predefined model maximum length if a maximum length
is required by one of the truncation/padding parameters. If the model has no specific maximum input
length (like XLNet) truncation/padding to a maximum length will be deactivated.   is_split_into_words (bool, optional, defaults to False) —
Whether or not the input is already pre-tokenized (e.g., split into words). If set to True, the
tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace)
which it will tokenize. This is useful for NER or token classification.   pad_to_multiple_of (int, optional) —
If set will pad the sequence to a multiple of the provided value. This is especially useful to enable
the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta).   return_tensors (str or TensorType, optional) —
If set, will return tensors instead of list of python integers. A


TapasTokenizer

cceptable values are:
'tf': Return TensorFlow tf.constant objects.
'pt': Return PyTorch torch.Tensor objects.
'np': Return Numpy np.ndarray objects.
    Main method to tokenize and prepare for the model one or several sequence(s) related to a table.  convert_logits_to_predictions  < source > ( data logits logits_agg = None cell_classification_threshold = 0.5  ) → tuple comprising various elements depending on the inputs  Parameters   data (dict) —
Dictionary mapping features to actual values. Should be created using TapasTokenizer.   logits (torch.Tensor or tf.Tensor of shape (batch_size, sequence_length)) —
Tensor containing the logits at the token level.   logits_agg (torch.Tensor or tf.Tensor of shape (batch_size, num_aggregation_labels), optional) —
Tensor containing the aggregation logits.   cell_classification_threshold (float, optional, defaults to 0.5) —
Threshold to be used for cell selection. All table cells


TapasTokenizer

 for which their probability is larger than
this threshold will be selected.   Returns
tuple comprising various elements depending on the inputs

predicted_answer_coordinates (List[List[[tuple]] of length batch_size): Predicted answer coordinates
as a list of lists of tuples. Each element in the list contains the predicted answer coordinates of a
single example in the batch, as a list of tuples. Each tuple is a cell, i.e. (row index, column index).
predicted_aggregation_indices (List[int]of length batch_size, optional, returned when
logits_aggregation is provided): Predicted aggregation operator indices of the aggregation head.
  Converts logits of TapasForQuestionAnswering to actual predicted answer coordinates and optional
aggregation indices. The original implementation, on which this function is based, can be found
here.  save_vocabulary  < source > ( save_directory: str filename_prefix: typing.Optional[str] =


TapasModel

class transformers.TapasModel  < source > ( config add_pooling_layer = True  )   Parameters   config (TapasConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare Tapas Model transformer outputting raw hidden-states without any specific head on top.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its models (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior. This class is a sma


TapasModel

ll change compared to BertModel, taking into account the additional token type ids. The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of
cross-attention is added between the self-attention layers, following the architecture described in Attention is
all you need by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit,
Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None encoder_hidden_states: typing.Optional[torch.FloatTensor] = None encoder_attentio


TapasModel

n_mask: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.BaseModelOutputWithPooling or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Indices can be obtained using AutoTokenizer. See
PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_si


TapasModel

ze, sequence_length, 7), optional) —
Token indices that encode tabular structure. Indices can be obtained using AutoTokenizer. See this
class for more info.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. If
reset_position_index_per_cell of TapasConfig is set to True, relative position embeddings will be
used. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]: - 1
indicates the head is not masked, - 0 indicates the head is masked.   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size


TapasModel

), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.BaseModelOutputWithPooling or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutputWithPooling or a tuple of
torch.FloatTensor (


TapasModel

if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (TapasConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing
through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns
the classification token after processing through a linear layer and a tanh activation function. The linear
layer weights are trained from the next sentence prediction (classification) objective during pretraining.
hidden_states (tuple(torch.FloatTensor), optional, returned when output


TapasModel

_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The TapasModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function,


TapasModel

one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoTokenizer, TapasModel
>>> import pandas as pd
>>> tokenizer = AutoTokenizer.from_pretrained("google/tapas-base")
>>> model = TapasModel.from_pretrained("google/tapas-base")
>>> data = {
...     "Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"],
...     "Age": ["56", "45", "59"],
...     "Number of movies": ["87", "53", "69"],
... }
>>> table = pd.DataFrame.from_dict(data)
>>> queries = ["How many movies has George Clooney played in?", "How old is Brad Pitt?"]
>>> inputs = tokenizer(table=table, queries=queries, padding="max_length", return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_state


TapasForMaskedLM

class transformers.TapasForMaskedLM  < source > ( config  )   Parameters   config (TapasConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Tapas Model with a language modeling head on top.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its models (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: t


TapasForMaskedLM

yping.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None encoder_hidden_states: typing.Optional[torch.FloatTensor] = None encoder_attention_mask: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.LongTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None **kwargs  ) → transformers.modeling_outputs.MaskedLMOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Indices can be obtained using AutoTokenizer. See
PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size,


TapasForMaskedLM

sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length, 7), optional) —
Token indices that encode tabular structure. Indices can be obtained using AutoTokenizer. See this
class for more info.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. If
reset_position_index_per_cell of TapasConfig is set to True, relative position embeddings will be
used. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attentio


TapasForMaskedLM

n modules. Mask values selected in [0, 1]: - 1
indicates the head is not masked, - 0 indicates the head is masked.   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size, sequence


TapasForMaskedLM

_length), optional) —
Labels for computing the masked language modeling loss. Indices should be in [-100, 0, ..., config.vocab_size] (see input_ids docstring) Tokens with indices set to -100 are ignored (masked), the
loss is only computed for the tokens with labels in [0, ..., config.vocab_size]   Returns
transformers.modeling_outputs.MaskedLMOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.MaskedLMOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (TapasConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Masked language modeling (MLM) loss.
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
hidden_states (tuple(torch.


TapasForMaskedLM

FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The TapasForMaskedLM forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call


TapasForMaskedLM

the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoTokenizer, TapasForMaskedLM
>>> import pandas as pd
>>> tokenizer = AutoTokenizer.from_pretrained("google/tapas-base")
>>> model = TapasForMaskedLM.from_pretrained("google/tapas-base")
>>> data = {
...     "Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"],
...     "Age": ["56", "45", "59"],
...     "Number of movies": ["87", "53", "69"],
... }
>>> table = pd.DataFrame.from_dict(data)
>>> inputs = tokenizer(
...     table=table, queries="How many [MASK] has George [MASK] played in?", return_tensors="pt"
... )
>>> labels = tokenizer(
...     table=table, queries="How many movies has George Clooney played in?", return_tensors="pt"
... )["input_ids"]
>>> outputs = model(**inputs, labels=labels)
>>> logits = outputs.l


TapasForSequenceClassification

class transformers.TapasForSequenceClassification  < source > ( config  )   Parameters   config (TapasConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Tapas Model with a sequence classification head on top (a linear layer on top of the pooled output), e.g. for table
entailment tasks, such as TabFact (Chen et al., 2020). This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its models (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional


TapasForSequenceClassification

[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.LongTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.SequenceClassifierOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Indices can be obtained using AutoTokenizer. See
PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attentio


TapasForSequenceClassification

n on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length, 7), optional) —
Token indices that encode tabular structure. Indices can be obtained using AutoTokenizer. See this
class for more info.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. If
reset_position_index_per_cell of TapasConfig is set to True, relative position embeddings will be
used. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]: - 1
indicates the head is not masked, - 0 indicates th


TapasForSequenceClassification

e head is masked.   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If co


TapasForSequenceClassification

nfig.num_labels == 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-Entropy). Note: this is called
“classification_class_index” in the original implementation.   Returns
transformers.modeling_outputs.SequenceClassifierOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.SequenceClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (TapasConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.outpu


TapasForSequenceClassification

t_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The TapasForSequenceClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing


TapasForSequenceClassification

steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoTokenizer, TapasForSequenceClassification
>>> import torch
>>> import pandas as pd
>>> tokenizer = AutoTokenizer.from_pretrained("google/tapas-base-finetuned-tabfact")
>>> model = TapasForSequenceClassification.from_pretrained("google/tapas-base-finetuned-tabfact")
>>> data = {
...     "Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"],
...     "Age": ["56", "45", "59"],
...     "Number of movies": ["87", "53", "69"],
... }
>>> table = pd.DataFrame.from_dict(data)
>>> queries = [
...     "There is only one actor who is 45 years old",
...     "There are 3 actors which played in more than 60 movies",
... ]
>>> inputs = tokenizer(table=table, queries=queries, padding="max_length", return_tensors="pt")
>>> labels = torch.tensor([1, 0])  # 1 means entailed, 0 means refuted
>>> outputs = model(**inputs, labels=labels)
>>> loss = outputs.loss
>>> logits = outputs.lo


TapasForQuestionAnswering

class transformers.TapasForQuestionAnswering  < source > ( config: TapasConfig  )   Parameters   config (TapasConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Tapas Model with a cell selection head and optional aggregation head on top for question-answering tasks on tables
(linear layers on top of the hidden-states output to compute logits and optional logits_aggregation), e.g. for
SQA, WTQ or WikiSQL-supervised tasks. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its models (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch docu


TapasForQuestionAnswering

mentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None table_mask: typing.Optional[torch.LongTensor] = None labels: typing.Optional[torch.LongTensor] = None aggregation_labels: typing.Optional[torch.LongTensor] = None float_answer: typing.Optional[torch.FloatTensor] = None numeric_values: typing.Optional[torch.FloatTensor] = None numeric_values_scale: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.tapas.modeling_tapas.TableQuestionAnsweringOu


TapasForQuestionAnswering

tput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Indices can be obtained using AutoTokenizer. See
PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length, 7), optional) —
Token indices that encode tabular structure. Indices can be obtained using AutoTokenizer. See this
class for more info.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position em


TapasForQuestionAnswering

beddings. If
reset_position_index_per_cell of TapasConfig is set to True, relative position embeddings will be
used. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]: - 1
indicates the head is not masked, - 0 indicates the head is masked.   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for m


TapasForQuestionAnswering

ore detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   table_mask (torch.LongTensor of shape (batch_size, seq_length), optional) —
Mask for the table. Indicates which tokens belong to the table (1). Question tokens, table headers and
padding are 0.   labels (torch.LongTensor of shape (batch_size, seq_length), optional) —
Labels per token for computing the hierarchical cell selection loss. This encodes the positions of the
answer appearing in the table. Can be obtained using AutoTokenizer.
1 for tokens that are part of the answer,
0 for tokens that are not part of the answer.
   aggregation_labels (torch.LongTensor of shape (batch_size, ), optional) —
Aggregation function index for every example in the batch for computing the aggregation loss. Ind


TapasForQuestionAnswering

ices
should be in [0, ..., config.num_aggregation_labels - 1]. Only required in case of strong supervision for
aggregation (WikiSQL-supervised).   float_answer (torch.FloatTensor of shape (batch_size, ), optional) —
Float answer for every example in the batch. Set to float(‘nan’) for cell selection questions. Only
required in case of weak supervision (WTQ) to calculate the aggregate mask and regression loss.   numeric_values (torch.FloatTensor of shape (batch_size, seq_length), optional) —
Numeric values of every token, NaN for tokens which are not numeric values. Can be obtained using
AutoTokenizer. Only required in case of weak supervision for aggregation (WTQ) to calculate the
regression loss.   numeric_values_scale (torch.FloatTensor of shape (batch_size, seq_length), optional) —
Scale of the numeric values of every token. Can be obtained using AutoTokenizer. Only required in case
of weak supervision for aggregation (WTQ) to calculate the regression los


TapasForQuestionAnswering

s.   Returns
transformers.models.tapas.modeling_tapas.TableQuestionAnsweringOutput or tuple(torch.FloatTensor)

A transformers.models.tapas.modeling_tapas.TableQuestionAnsweringOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (TapasConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels (and possibly answer, aggregation_labels, numeric_values and numeric_values_scale are provided)) — Total loss as the sum of the hierarchical cell selection log-likelihood loss and (optionally) the
semi-supervised regression loss and (optionally) supervised loss for aggregations.
logits (torch.FloatTensor of shape (batch_size, sequence_length)) — Prediction scores of the cell selection head, for every token.
logits_aggregation (torch.FloatTensor, optional, of shape (batch_size, num_aggregation_labels)) — Prediction scores of t


TapasForQuestionAnswering

he aggregation head, for every aggregation operator.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size). Hidden-states of the model at the output of each layer
plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Attentions weights after the attention softmax, used to compute the weighted average in
the self-attention heads.
  The TapasForQuestionAnswering forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this functi


TapasForQuestionAnswering

on, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoTokenizer, TapasForQuestionAnswering
>>> import pandas as pd
>>> tokenizer = AutoTokenizer.from_pretrained("google/tapas-base-finetuned-wtq")
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base-finetuned-wtq")
>>> data = {
...     "Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"],
...     "Age": ["56", "45", "59"],
...     "Number of movies": ["87", "53", "69"],
... }
>>> table = pd.DataFrame.from_dict(data)
>>> queries = ["How many movies has George Clooney played in?", "How old is Brad Pitt?"]
>>> inputs = tokenizer(table=table, queries=queries, padding="max_length", return_tensors="pt")
>>> outputs = model(**inputs)
>>> logits = outputs.logits
>>> logits_aggregation = outputs.logits_aggregat


TFTapasModel

class transformers.TFTapasModel  < source > ( *args **kwargs  )   Parameters   config (TapasConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare Tapas Model transformer outputting raw hidden-states without any specific head on top. This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior. TensorFlow models and layers in transformers accept two formats as input: having all i


TFTapasModel

nputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_mask]) or model([input_i


TFTapasModel

ds, attention_mask, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None training: Optional[bool] = False  ) → transformers.modeling_tf_outputs.TFBaseModelOutputWithPooling or tupl


TFTapasModel

e(tf.Tensor)  Parameters   input_ids (np.ndarray, tf.Tensor, List[tf.Tensor] `Dict[str, tf.Tensor] or Dict[str, np.ndarray] and each example must have the shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and
PreTrainedTokenizer.encode() for details.
What are input IDs?   attention_mask (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (np.ndarray or tf.Tensor of shape (batch_size, sequence_length, 7), optional) —
Token indices that encode tabular structure. Indices can be obtained using AutoTokenizer. See this
class for more info.
What are token type IDs?   position_ids (np.ndar


TFTapasModel

ray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. If
reset_position_index_per_cell of TapasConfig is set to True, relative position embeddings will be
used. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (np.ndarray or tf.Tensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (np.ndarray or tf.Tensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s i


TFTapasModel

nternal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the
config will be used instead.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail. This argument can be used only in eager mode, in graph mode the value in the config will be
used instead.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in
eager mode, in graph mode the value will always be set to True.   training (bool, optional, defaults to `False“) —
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between t


TFTapasModel

raining and evaluation).   Returns
transformers.modeling_tf_outputs.TFBaseModelOutputWithPooling or tuple(tf.Tensor)

A transformers.modeling_tf_outputs.TFBaseModelOutputWithPooling or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (TapasConfig) and inputs.
last_hidden_state (tf.Tensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (tf.Tensor of shape (batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) further processed by a
Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence
prediction (classification) objective during pretraining.
This output is usually not a good summary of the semantic content of the input, you


TFTapasModel

’re often better with
averaging or pooling the sequence of hidden-states for the whole input sequence.
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The TFTapasModel forward method, overrides the __call__ special method. Although the recipe for forwa


TFTapasModel

rd pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoTokenizer, TapasModel
>>> import pandas as pd
>>> tokenizer = AutoTokenizer.from_pretrained("google/tapas-base")
>>> model = TapasModel.from_pretrained("google/tapas-base")
>>> data = {
...     "Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"],
...     "Age": ["56", "45", "59"],
...     "Number of movies": ["87", "53", "69"],
... }
>>> table = pd.DataFrame.from_dict(data)
>>> queries = ["How many movies has George Clooney played in?", "How old is Brad Pitt?"]
>>> inputs = tokenizer(table=table, queries=queries, padding="max_length", return_tensors="tf")
>>> outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_st


TFTapasForMaskedLM

class transformers.TFTapasForMaskedLM  < source > ( *args **kwargs  )   Parameters   config (TapasConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Tapas Model with a language modeling head on top. This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior. TensorFlow models and layers in transformers accept two formats as input: having all inputs as keyword arguments (like PyTorch models),


TFTapasForMaskedLM

 or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_mask]) or model([input_ids, attention_mask, token_type_ids]) a dictionary with one


TFTapasForMaskedLM

 or several input Tensors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None labels: np.ndarray | tf.Tensor | None = None training: Optional[bool] = False  ) → transformers.modeling_tf_outputs.TFMaskedLMOutput or tuple(tf.Tensor)  Parameters   input_i


TFTapasForMaskedLM

ds (np.ndarray, tf.Tensor, List[tf.Tensor] `Dict[str, tf.Tensor] or Dict[str, np.ndarray] and each example must have the shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and
PreTrainedTokenizer.encode() for details.
What are input IDs?   attention_mask (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (np.ndarray or tf.Tensor of shape (batch_size, sequence_length, 7), optional) —
Token indices that encode tabular structure. Indices can be obtained using AutoTokenizer. See this
class for more info.
What are token type IDs?   position_ids (np.ndarray or tf.Tensor of shape (batch_size, sequ


TFTapasForMaskedLM

ence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. If
reset_position_index_per_cell of TapasConfig is set to True, relative position embeddings will be
used. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (np.ndarray or tf.Tensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (np.ndarray or tf.Tensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions


TFTapasForMaskedLM

 (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the
config will be used instead.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail. This argument can be used only in eager mode, in graph mode the value in the config will be
used instead.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in
eager mode, in graph mode the value will always be set to True.   training (bool, optional, defaults to `False“) —
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).   labels (tf.Tensor or np.ndarray of


TFTapasForMaskedLM

 shape (batch_size, sequence_length), optional) —
Labels for computing the masked language modeling loss. Indices should be in [-100, 0, ..., config.vocab_size] (see input_ids docstring) Tokens with indices set to -100 are ignored (masked), the
loss is only computed for the tokens with labels in [0, ..., config.vocab_size]   Returns
transformers.modeling_tf_outputs.TFMaskedLMOutput or tuple(tf.Tensor)

A transformers.modeling_tf_outputs.TFMaskedLMOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (TapasConfig) and inputs.
loss (tf.Tensor of shape (n,), optional, where n is the number of non-masked labels, returned when labels is provided) — Masked language modeling (MLM) loss.
logits (tf.Tensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each v


TFTapasForMaskedLM

ocabulary token before SoftMax).
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The TFTapasForMaskedLM forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Mod


TFTapasForMaskedLM

ule
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoTokenizer, TapasForMaskedLM
>>> import pandas as pd
>>> tokenizer = AutoTokenizer.from_pretrained("google/tapas-base")
>>> model = TapasForMaskedLM.from_pretrained("google/tapas-base")
>>> data = {
...     "Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"],
...     "Age": ["56", "45", "59"],
...     "Number of movies": ["87", "53", "69"],
... }
>>> table = pd.DataFrame.from_dict(data)
>>> inputs = tokenizer(
...     table=table, queries="How many [MASK] has George [MASK] played in?", return_tensors="tf"
... )
>>> labels = tokenizer(
...     table=table, queries="How many movies has George Clooney played in?", return_tensors="tf"
... )["input_ids"]
>>> outputs = model(**inputs, labels=labels)
>>> logits = output


TFTapasForSequenceClassification

class transformers.TFTapasForSequenceClassification  < source > ( *args **kwargs  )   Parameters   config (TapasConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Tapas Model with a sequence classification head on top (a linear layer on top of the pooled output), e.g. for table
entailment tasks, such as TabFact (Chen et al., 2020). This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior. TensorFlow models and layers in transformers ac


TFTapasForSequenceClassification

cept two formats as input: having all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_mask]) or model([input_ids, attention_mas


TFTapasForSequenceClassification

k, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None labels: np.ndarray | tf.Tensor | None = None training: Optional[bool] = False  ) → transformers.modeling_tf_outputs.TFSequenceClassifierOutput or tuple(tf.Tensor)  Parameters   inp


TFTapasForSequenceClassification

ut_ids (np.ndarray, tf.Tensor, List[tf.Tensor] `Dict[str, tf.Tensor] or Dict[str, np.ndarray] and each example must have the shape (batch_size, num_choices, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and
PreTrainedTokenizer.encode() for details.
What are input IDs?   attention_mask (np.ndarray or tf.Tensor of shape (batch_size, num_choices, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (np.ndarray or tf.Tensor of shape (batch_size, num_choices, sequence_length, 7), optional) —
Token indices that encode tabular structure. Indices can be obtained using AutoTokenizer. See this
class for more info.
What are token type IDs?   position_ids (np.ndarray or tf.Tensor of shape (batch_size, num_choi


TFTapasForSequenceClassification

ces, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. If
reset_position_index_per_cell of TapasConfig is set to True, relative position embeddings will be
used. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (np.ndarray or tf.Tensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (np.ndarray or tf.Tensor of shape (batch_size, num_choices, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Wheth


TFTapasForSequenceClassification

er or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the
config will be used instead.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail. This argument can be used only in eager mode, in graph mode the value in the config will be
used instead.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in
eager mode, in graph mode the value will always be set to True.   training (bool, optional, defaults to `False“) —
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the sequence classificat


TFTapasForSequenceClassification

ion/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-Entropy). Note: this is called
“classification_class_index” in the original implementation.   Returns
transformers.modeling_tf_outputs.TFSequenceClassifierOutput or tuple(tf.Tensor)

A transformers.modeling_tf_outputs.TFSequenceClassifierOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (TapasConfig) and inputs.
loss (tf.Tensor of shape (batch_size, ), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (tf.Tensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(tf.Tensor), optional, returned when


TFTapasForSequenceClassification

 output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The TFTapasForSequenceClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the


TFTapasForSequenceClassification

 latter silently ignores them.  Examples:   Copied >>> from transformers import AutoTokenizer, TapasForSequenceClassification
>>> import tensorflow as tf
>>> import pandas as pd
>>> tokenizer = AutoTokenizer.from_pretrained("google/tapas-base-finetuned-tabfact")
>>> model = TapasForSequenceClassification.from_pretrained("google/tapas-base-finetuned-tabfact")
>>> data = {
...     "Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"],
...     "Age": ["56", "45", "59"],
...     "Number of movies": ["87", "53", "69"],
... }
>>> table = pd.DataFrame.from_dict(data)
>>> queries = [
...     "There is only one actor who is 45 years old",
...     "There are 3 actors which played in more than 60 movies",
... ]
>>> inputs = tokenizer(table=table, queries=queries, padding="max_length", return_tensors="tf")
>>> labels = tf.convert_to_tensor([1, 0])  # 1 means entailed, 0 means refuted
>>> outputs = model(**inputs, labels=labels)
>>> loss = outputs.loss
>>> logits = outputs.l


TFTapasForQuestionAnswering

class transformers.TFTapasForQuestionAnswering  < source > ( *args **kwargs  )   Parameters   config (TapasConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Tapas Model with a cell selection head and optional aggregation head on top for question-answering tasks on tables
(linear layers on top of the hidden-states output to compute logits and optional logits_aggregation), e.g. for
SQA, WTQ or WikiSQL-supervised tasks. This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 docu


TFTapasForQuestionAnswering

mentation for all matter related to general usage and
behavior. TensorFlow models and layers in transformers accept two formats as input: having all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with input_ids only and nothing else: model(input_ids) a list of varying length with o


TFTapasForQuestionAnswering

ne or several input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_mask]) or model([input_ids, attention_mask, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None table_mask: np.ndarray | tf.Tensor | None = None aggregation_labels: np.ndarray | tf.Tensor | None = None float_answer: np.ndarray | tf.Tensor


TFTapasForQuestionAnswering

| None = None numeric_values: np.ndarray | tf.Tensor | None = None numeric_values_scale: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None labels: np.ndarray | tf.Tensor | None = None training: Optional[bool] = False  ) → transformers.models.tapas.modeling_tf_tapas.TFTableQuestionAnsweringOutput or tuple(tf.Tensor)  Parameters   input_ids (np.ndarray, tf.Tensor, List[tf.Tensor] `Dict[str, tf.Tensor] or Dict[str, np.ndarray] and each example must have the shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and
PreTrainedTokenizer.encode() for details.
What are input IDs?   attention_mask (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask valu


TFTapasForQuestionAnswering

es selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (np.ndarray or tf.Tensor of shape (batch_size, sequence_length, 7), optional) —
Token indices that encode tabular structure. Indices can be obtained using AutoTokenizer. See this
class for more info.
What are token type IDs?   position_ids (np.ndarray or tf.Tensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. If
reset_position_index_per_cell of TapasConfig is set to True, relative position embeddings will be
used. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (np.ndarray or tf.Tensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the


TFTapasForQuestionAnswering

 head is masked.
   inputs_embeds (np.ndarray or tf.Tensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the
config will be used instead.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail. This argument can be used only in eager mode, in graph mode the value in the config will be
used instead.   return_dict (bool, optional) —
Whether or not


TFTapasForQuestionAnswering

to return a ModelOutput instead of a plain tuple. This argument can be used in
eager mode, in graph mode the value will always be set to True.   training (bool, optional, defaults to `False“) —
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).   table_mask (tf.Tensor of shape (batch_size, seq_length), optional) —
Mask for the table. Indicates which tokens belong to the table (1). Question tokens, table headers and
padding are 0.   labels (tf.Tensor of shape (batch_size, seq_length), optional) —
Labels per token for computing the hierarchical cell selection loss. This encodes the positions of the
answer appearing in the table. Can be obtained using AutoTokenizer.
1 for tokens that are part of the answer,
0 for tokens that are not part of the answer.
   aggregation_labels (tf.Tensor of shape (batch_size, ), optional) —
Aggregation function index for every exam


TFTapasForQuestionAnswering

ple in the batch for computing the aggregation loss. Indices
should be in [0, ..., config.num_aggregation_labels - 1]. Only required in case of strong supervision for
aggregation (WikiSQL-supervised).   float_answer (tf.Tensor of shape (batch_size, ), optional) —
Float answer for every example in the batch. Set to float(‘nan’) for cell selection questions. Only
required in case of weak supervision (WTQ) to calculate the aggregate mask and regression loss.   numeric_values (tf.Tensor of shape (batch_size, seq_length), optional) —
Numeric values of every token, NaN for tokens which are not numeric values. Can be obtained using
AutoTokenizer. Only required in case of weak supervision for aggregation (WTQ) to calculate the
regression loss.   numeric_values_scale (tf.Tensor of shape (batch_size, seq_length), optional) —
Scale of the numeric values of every token. Can be obtained using AutoTokenizer. Only required in case
of weak supervision for aggregati


TFTapasForQuestionAnswering

on (WTQ) to calculate the regression loss.   Returns
transformers.models.tapas.modeling_tf_tapas.TFTableQuestionAnsweringOutput or tuple(tf.Tensor)

A transformers.models.tapas.modeling_tf_tapas.TFTableQuestionAnsweringOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (TapasConfig) and inputs.
loss (tf.Tensor of shape (1,), optional, returned when labels (and possibly answer, aggregation_labels, numeric_values and numeric_values_scale are provided)) — Total loss as the sum of the hierarchical cell selection log-likelihood loss and (optionally) the
semi-supervised regression loss and (optionally) supervised loss for aggregations.
logits (tf.Tensor of shape (batch_size, sequence_length)) — Prediction scores of the cell selection head, for every token.
logits_aggregation (tf.Tensor, optional, of shape (batch_size, num_aggregation_labels)) — Pred


TFTapasForQuestionAnswering

iction scores of the aggregation head, for every aggregation operator.
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size). Hidden-states of the model at the output of each layer plus
the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Attentions weights after the attention softmax, used to compute the weighted average in
the self-attention heads.
  The TFTapasForQuestionAnswering forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function,


TFTapasForQuestionAnswering

one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoTokenizer, TapasForQuestionAnswering
>>> import pandas as pd
>>> tokenizer = AutoTokenizer.from_pretrained("google/tapas-base-finetuned-wtq")
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base-finetuned-wtq")
>>> data = {
...     "Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"],
...     "Age": ["56", "45", "59"],
...     "Number of movies": ["87", "53", "69"],
... }
>>> table = pd.DataFrame.from_dict(data)
>>> queries = ["How many movies has George Clooney played in?", "How old is Brad Pitt?"]
>>> inputs = tokenizer(table=table, queries=queries, padding="max_length", return_tensors="tf")
>>> outputs = model(**inputs)
>>> logits = outputs.logits
>>> logits_aggregation = outputs.logits_aggr


Overview

DialoGPT was proposed in DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation by Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao,
Jianfeng Gao, Jingjing Liu, Bill Dolan. It’s a GPT2 Model trained on 147M conversation-like exchanges extracted from
Reddit.
The abstract from the paper is the following:
We present a large, tunable neural conversational response generation model, DialoGPT (dialogue generative pre-trained
transformer). Trained on 147M conversation-like exchanges extracted from Reddit comment chains over a period spanning
from 2005 through 2017, DialoGPT extends the Hugging Face PyTorch transformer to attain a performance close to human
both


Overview

in terms of automatic and human evaluation in single-turn dialogue settings. We show that conversational systems
that leverage DialoGPT generate more relevant, contentful and context-consistent responses than strong baseline
systems. The pre-trained model and training pipeline are publicly released to facilitate research into neural response
generation and the development of more intelligent open-domain dialogue systems.
Tips:
DialoGPT is a model with absolute position embeddings so it’s usually advised to pad the inputs on the right rather
than the left. DialoGPT was trained with a causal language modeling (CLM) objective on conversational data and is therefore powerful
at response generation in open-domain dialo


Overview

gue systems. DialoGPT enables the user to create a chat bot in just 10 lines of code as shown on DialoGPT’s model card.
Training:
In order to train or fine-tune DialoGPT, one can use causal language modeling training. To cite the official paper: We
follow the OpenAI GPT-2 to model a multiturn dialogue session as a long text and frame the generation task as language
modeling. We first concatenate all dialog turns within a dialogue session into a long text x_1,…, x_N (N is the
sequence length), ended by the end-of-text token. For more information please confer to the original paper.
DialoGPT’s architecture is based on the GPT2 model, so one can refer to GPT2’s documentation page.
The original code can be found here.


Overview

The MusicGen model was proposed in the paper Simple and Controllable Music Generation
by Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi and Alexandre Défossez.
MusicGen is a single stage auto-regressive Transformer model capable of generating high-quality music samples conditioned
on text descriptions or audio prompts. The text descriptions are passed through a frozen text encoder model to obtain a
sequence of hidden-state representations. MusicGen is then trained to predict discrete audio tokens, or audio codes,
conditioned on these hidden-states. These audio tokens are then decoded using an audio compression model, such as EnCodec,
to recover the audio waveform.
Through an ef


Overview

ficient token interleaving pattern, MusicGen does not require a self-supervised semantic representation of
the text/audio prompts, thus eliminating the need to cascade multiple models to predict a set of codebooks (e.g.
hierarchically or upsampling). Instead, it is able to generate all the codebooks in a single forward pass.
The abstract from the paper is the following:
We tackle the task of conditional music generation. We introduce MusicGen, a single Language Model (LM) that operates
over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised
of a single-stage transformer LM together with efficient token interleaving patterns, which eliminates the need f


Overview

or
cascading several models, e.g., hierarchically or upsampling. Following this approach, we demonstrate how MusicGen
can generate high-quality samples, while being conditioned on textual description or melodic features, allowing better
controls over the generated output. We conduct extensive empirical evaluation, considering both automatic and human
studies, showing the proposed approach is superior to the evaluated baselines on a standard text-to-music benchmark.
Through ablation studies, we shed light over the importance of each of the components comprising MusicGen.
This model was contributed by sanchit-gandhi. The original code can be found
here. The pre-trained checkpoints can be found on the
Hugging Face Hub.


Generation

MusicGen is compatible with two generation modes: greedy and sampling. In practice, sampling leads to significantly
better results than greedy, thus we encourage sampling mode to be used where possible. Sampling is enabled by default,
and can be explicitly specified by setting do_sample=True in the call to MusicgenForConditionalGeneration.generate(),
or by overriding the model’s generation config (see below).
Generation is limited by the sinusoidal positional embeddings to 30 second inputs. Meaning, MusicGen cannot generate more
than 30 seconds of audio (1503 tokens), and input audio passed by Audio-Prompted Generation contributes to this limit so,
given an input of 20 seconds of audio, MusicGen cannot generate more than 10 seconds of additional audio.


Unconditional Generation

The inputs for unconditional (or ‘null’) generation can be obtained through the method
MusicgenForConditionalGeneration.get_unconditional_inputs():
  Copied >>> from transformers import MusicgenForConditionalGeneration
>>> model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
>>> unconditional_inputs = model.get_unconditional_inputs(num_samples=1)
>>> audio_values = model.generate(**unconditional_inputs, do_sample=True, max_new_tokens=256)
The audio outputs are a three-dimensional Torch tensor of shape (


Unconditional Generation

batch_size, num_channels, sequence_length). To listen
to the generated audio samples, you can either play them in an ipynb notebook:
  Copied from IPython.display import Audio
sampling_rate = model.config.audio_encoder.sampling_rate
Audio(audio_values[0].numpy(), rate=sampling_rate)
Or save them as a .wav file using a third-party library, e.g. scipy:
  Copied >>> import scipy
>>> sampling_rate = model.config.audio_encoder.sampling_rate
>>> scipy.io.wavfile.write("musicgen_out.wav", rate=sampling_rate, data=audio_values[0, 0].numpy())


Text-Conditional Generation

The model can generate an audio sample conditioned on a text prompt through use of the MusicgenProcessor to pre-process
the inputs:
  Copied >>> from transformers import AutoProcessor, MusicgenForConditionalGeneration
>>> processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
>>> model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
>>> inputs = processor(
...     text=["80s pop track with bassy drums and synth", "90s rock song with loud guitars and heavy drums"],
...     padding=True,
...     return_tensors="pt",
... )
>>> audio_values


Text-Conditional Generation

= model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)
The guidance_scale is used in classifier free guidance (CFG), setting the weighting between the conditional logits
(which are predicted from the text prompts) and the unconditional logits (which are predicted from an unconditional or
‘null’ prompt). Higher guidance scale encourages the model to generate samples that are more closely linked to the input
prompt, usually at the expense of poorer audio quality. CFG is enabled by setting guidance_scale > 1. For best results,
use guidance_scale=3 (default).


Audio-Prompted Generation

The same MusicgenProcessor can be used to pre-process an audio prompt that is used for audio continuation. In the
following example, we load an audio file using the 🤗 Datasets library, which can be pip installed through the command
below:
  Copied pip install --upgrade pip
pip install datasets[audio]
  Copied >>> from transformers import AutoProcessor, MusicgenForConditionalGeneration
>>> from datasets import load_dataset
>>> processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
>>> model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
>>> dataset = load_dataset("sanchit-gandhi/gtzan", split="train", streaming=True)
>>> sample = next(iter(dataset))["audio"]
>>> # take the first half of the audio sample
>>> sample["array"] = sample["array"][: len(sample


Audio-Prompted Generation

["array"]) // 2]
>>> inputs = processor(
...     audio=sample["array"],
...     sampling_rate=sample["sampling_rate"],
...     text=["80s blues track with groovy saxophone"],
...     padding=True,
...     return_tensors="pt",
... )
>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)
For batched audio-prompted generation, the generated audio_values can be post-processed to remove padding by using the
MusicgenProcessor class:
  Copied >>> from transformers import AutoProcessor, MusicgenForConditionalGeneration
>>> from datasets import load_dataset
>>> processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
>>> model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
>>> dataset = load_dataset("sanchit-gandhi/gtzan


Audio-Prompted Generation

", split="train", streaming=True)
>>> sample = next(iter(dataset))["audio"]
>>> # take the first quarter of the audio sample
>>> sample_1 = sample["array"][: len(sample["array"]) // 4]
>>> # take the first half of the audio sample
>>> sample_2 = sample["array"][: len(sample["array"]) // 2]
>>> inputs = processor(
...     audio=[sample_1, sample_2],
...     sampling_rate=sample["sampling_rate"],
...     text=["80s blues track with groovy saxophone", "90s rock song with loud guitars and heavy drums"],
...     padding=True,
...     return_tensors="pt",
... )
>>> audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)
>>> # post-process to remove padding from the batched audio
>>> audio_values = processor.batch_decode(audio_values, padding_mask=inputs.padding_mas


Generation Configuration

The default parameters that control the generation process, such as sampling, guidance scale and number of generated
tokens, can be found in the model’s generation config, and updated as desired:
  Copied >>> from transformers import MusicgenForConditionalGeneration
>>> model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
>>> # inspect the default generation config
>>> model.generation_config
>>> # increase the guidance scale to 4.0
>>> model.generation_config.guidance_scale = 4.0
>>> # decrease the max length to 256 tokens
>>> model.generation_config.max_length = 256
Note that any arguments passed to the generate method will supersede those in the generation config, so setting
do_sample=False in the call to generate will supersede the setting of model.generation_config.do_sample in the
generation config.


Model Structure

The MusicGen model can be de-composed into three distinct stages:
Text encoder: maps the text inputs to a sequence of hidden-state representations. The pre-trained MusicGen models use a frozen text encoder from either T5 or Flan-T5 MusicGen decoder: a language model (LM) that auto-regressively generates audio tokens (or codes) conditional on the encoder hidden-state representations Audio encoder/decoder: used to encode an audio prompt to use as prompt tokens, and recover the audio waveform from the audio tokens predicted by the decoder
Thus, the MusicGen model can either be used as a standalone decoder model, corresponding to the class MusicgenForCausalLM,
or as a composite model that includes the text encoder and audio encode


Model Structure

r/decoder, corresponding to the class
MusicgenForConditionalGeneration. If only the decoder needs to be loaded from the pre-trained checkpoint, it can be loaded by first
specifying the correct config, or be accessed through the .decoder attribute of the composite model:
  Copied >>> from transformers import AutoConfig, MusicgenForCausalLM, MusicgenForConditionalGeneration
>>> # Option 1: get decoder config and pass to `.from_pretrained`
>>> decoder_config = AutoConfig.from_pretrained("facebook/musicgen-small").decoder
>>> decoder = MusicgenForCausalLM.from_pretrained("facebook/musicgen-small", **decoder_config)
>>> # Option 2: load the entire composite model, but only return the decoder
>>> decoder = MusicgenForConditionalGene


Model Structure

ration.from_pretrained("facebook/musicgen-small").decoder
Since the text encoder and audio encoder/decoder models are frozen during training, the MusicGen decoder MusicgenForCausalLM
can be trained standalone on a dataset of encoder hidden-states and audio codes. For inference, the trained decoder can
be combined with the frozen text encoder and audio encoder/decoders to recover the composite MusicgenForConditionalGeneration
model.
Tips:
MusicGen is trained on the 32kHz checkpoint of Encodec. You should ensure you use a compatible version of the Encodec model. Sampling mode tends to deliver better results than greedy - you can toggle sampling with the variable do_sample in the call to MusicgenForConditionalGeneration.generate(


MusicgenDecoderConfig

class transformers.MusicgenDecoderConfig  < source > ( vocab_size = 2048 max_position_embeddings = 2048 num_hidden_layers = 24 ffn_dim = 4096 num_attention_heads = 16 layerdrop = 0.0 use_cache = True activation_function = 'gelu' hidden_size = 1024 dropout = 0.1 attention_dropout = 0.0 activation_dropout = 0.0 initializer_factor = 0.02 scale_embedding = False num_codebooks = 4 pad_token_id = 2048 bos_token_id = 2048 eos_token_id = None tie_word_embeddings = False **kwargs  )   Parameters   vocab_size (int, optional, defaults to 2048) —
Vocabulary size of the MusicgenDecoder model. Defines the number of different tokens that can be
represented by the inputs_ids passed when calling MusicgenDecoder.   hidden_size (int, optional, defaults to 1024) —
Dimensionality of the layers and the pooler layer.   num_hidden_


MusicgenDecoderConfig

layers (int, optional, defaults to 24) —
Number of decoder layers.   num_attention_heads (int, optional, defaults to 16) —
Number of attention heads for each attention layer in the Transformer block.   ffn_dim (int, optional, defaults to 4096) —
Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer block.   activation_function (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the decoder and pooler. If string, "gelu",
"relu", "silu" and "gelu_new" are supported.   dropout (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the embeddings, text_encoder, and pooler.   attention_dropout (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.   a


MusicgenDecoderConfig

ctivation_dropout (float, optional, defaults to 0.0) —
The dropout ratio for activations inside the fully connected layer.   max_position_embeddings (int, optional, defaults to 2048) —
The maximum sequence length that this model might ever be used with. Typically, set this to something large
just in case (e.g., 512 or 1024 or 2048).   initializer_factor (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   layerdrop (float, optional, defaults to 0.0) —
The LayerDrop probability for the decoder. See the [LayerDrop paper](see https://arxiv.org/abs/1909.11556)
for more details.   scale_embedding (bool, optional, defaults to False) —
Scale embeddings by diving by sqrt(hidden_size).   use_cache (bool, optional, defaults to True) —


MusicgenDecoderConfig

Whether the model should return the last key/values attentions (not used by all models)   num_codebooks (int, optional, defaults to 4) —
The number of parallel codebooks forwarded to the model.   tie_word_embeddings(bool, optional, defaults to False) —
Whether input and output word embeddings should be tied.    This is the configuration class to store the configuration of an MusicgenDecoder. It is used to instantiate a
MusicGen decoder according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the MusicGen
facebook/musicgen-small architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information


MusicgenConfig

class transformers.MusicgenConfig  < source > ( **kwargs  )   Parameters   kwargs (optional) —
Dictionary of keyword arguments. Notably:
text_encoder (PretrainedConfig, optional) — An instance of a configuration object that
defines the text encoder config.
audio_encoder (PretrainedConfig, optional) — An instance of a configuration object that
defines the audio encoder config.
decoder (PretrainedConfig, optional) — An instance of a configuration object that defines
the decoder config.
    This is the configuration class to store the configuration of a MusicgenModel. It is used to instantiate a
MusicGen model according to the specified arguments, defining the text encoder, audio encoder and MusicGen decoder
configs. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documen


MusicgenConfig

tation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import (
...     MusicgenConfig,
...     MusicgenDecoderConfig,
...     T5Config,
...     EncodecConfig,
...     MusicgenForConditionalGeneration,
... )
>>> # Initializing text encoder, audio encoder, and decoder model configurations
>>> text_encoder_config = T5Config()
>>> audio_encoder_config = EncodecConfig()
>>> decoder_config = MusicgenDecoderConfig()
>>> configuration = MusicgenConfig.from_sub_models_config(
...     text_encoder_config, audio_encoder_config, decoder_config
... )
>>> # Initializing a MusicgenForConditionalGeneration (with random weights) from the facebook/musicgen-small style configuration
>>> model = MusicgenForConditionalGeneration(configuration)
>>> # Accessing the model configuration
>>> configuration = model.c


MusicgenConfig

onfig
>>> config_text_encoder = model.config.text_encoder
>>> config_audio_encoder = model.config.audio_encoder
>>> config_decoder = model.config.decoder
>>> # Saving the model, including its configuration
>>> model.save_pretrained("musicgen-model")
>>> # loading model and config from pretrained folder
>>> musicgen_config = MusicgenConfig.from_pretrained("musicgen-model")
>>> model = MusicgenForConditionalGeneration.from_pretrained("musicgen-model", config=musicgen_config)  from_sub_models_config  < source > ( text_encoder_config: PretrainedConfig audio_encoder_config: PretrainedConfig decoder_config: MusicgenDecoderConfig **kwargs  ) → MusicgenConfig  Returns
MusicgenConfig

An instance of a configuration object
  Instantiate a MusicgenConfig (or a derived class) from text encoder, audio encoder and decoder
configurations.


MusicgenProcessor

class transformers.MusicgenProcessor  < source > ( feature_extractor tokenizer  )   Parameters   feature_extractor (EncodecFeatureExtractor) —
An instance of EncodecFeatureExtractor. The feature extractor is a required input.   tokenizer (T5Tokenizer) —
An instance of T5Tokenizer. The tokenizer is a required input.    Constructs a MusicGen processor which wraps an EnCodec feature extractor and a T5 tokenizer into a single processor
class. MusicgenProcessor offers all the functionalities of EncodecFeatureExtractor and TTokenizer. See
__call__() and dec


MusicgenProcessor

ode() for more information.  batch_decode  < source > ( *args **kwargs  )    This method is used to decode either batches of audio outputs from the MusicGen model, or batches of token ids
from the tokenizer. In the case of decoding token ids, this method forwards all its arguments to T5Tokenizer’s
batch_decode(). Please refer to the docstring of this method for more information.  decode  < source > ( *args **kwargs  )    This method forwards all its arguments to T5Tokenizer’s decode(). Please refer to the
docstring of this method for more information.


MusicgenModel

class transformers.MusicgenModel  < source > ( config: MusicgenDecoderConfig  )   Parameters   config (MusicgenConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare Musicgen decoder model outputting raw hidden-states without any specific head on top. The Musicgen model was proposed in Simple and Controllable Music Generation by
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, Alexandre Défossez. It is an
encoder decoder transformer trained on the task of conditional music generation This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all it


MusicgenModel

s model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: LongTensor = None attention_mask: typing.Optional[torch.Tensor] = None encoder_hidden_states: typing.Optional[torch.FloatTensor] = None encoder_attention_mask: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.Tensor] = None cross_attn_head_mask: typing.Optional[torch.Tensor] = None past_key_values: typing.Optional[typing.Tuple[typing.Tuple[torch.FloatTensor]]] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_state