YosoConfig

Dimension of the encoder layers and the pooler layer.   num_hidden_layers (int, optional, defaults to 12) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 12) —
Number of attention heads for each attention layer in the Transformer encoder.   intermediate_size (int, optional, defaults to 3072) —
Dimension of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.   hidden_act (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "selu" and "gelu_new" are supported.   hidden_dropout_prob (float, optional, defaults to 0.1) —
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.   att


YosoConfig

ention_probs_dropout_prob (float, optional, defaults to 0.1) —
The dropout ratio for the attention probabilities.   max_position_embeddings (int, optional, defaults to 512) —
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).   type_vocab_size (int, optional, defaults to 2) —
The vocabulary size of the token_type_ids passed when calling YosoModel.   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   layer_norm_eps (float, optional, defaults to 1e-12) —
The epsilon used by the layer normalization layers.   position_embedding_type (str, optional, defaults to "absolute") —
Type of position embedding. C


YosoConfig

hoose one of "absolute", "relative_key", "relative_key_query".   use_expectation (bool, optional, defaults to True) —
Whether or not to use YOSO Expectation. Overrides any effect of num_hash.   hash_code_len (int, optional, defaults to 9) —
The length of hashes generated by the hash functions.   num_hash (int, optional, defaults to 64) —
Number of hash functions used in YosoSelfAttention.   conv_window (int, optional) —
Kernel size of depth-wise convolution.   use_fast_hash (bool, optional, defaults to False) —
Whether or not to use custom cuda kernels which perform fast random projection via hadamard transform.   lsh_backward (bool, optional, defaults to True) —
Whether or not to perform backpropagation using Locality Sensitive Hashing.    This is the configuration class to store the configuratio


YosoConfig

n of a YosoModel. It is used to instantiate an YOSO
model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
defaults will yield a similar configuration to that of the YOSO
uw-madison/yoso-4096 architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import YosoConfig, YosoModel
>>> # Initializing a YOSO uw-madison/yoso-4096 style configuration
>>> configuration = YosoConfig()
>>> # Initializing a model (with random weights) from the uw-madison/yoso-4096 style configuration
>>> model = YosoModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.conf


YosoModel

class transformers.YosoModel  < source > ( config  )   Parameters   config (YosoConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare YOSO Model transformer outputting raw hidden-states without any specific head on top.
This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None output_attentions: typing.O


YosoModel

ptional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.BaseModelOutputWithCrossAttentions or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corr


YosoModel

esponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output


YosoModel

_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.BaseModelOutputWithCrossAttentions or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutputWithCrossAttentions or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (YosoConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
hidden_states (tuple(torch


YosoModel

.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True and config.add_cross_attention=True is passed or when config.output_attentions=True) — Tuple


YosoModel

of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
  The YosoModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, YosoModel
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("uw-madison/yoso-4096")
>>> model = YosoModel.from_pretrained("uw-madison/yoso-4096")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_stat


YosoForMaskedLM

class transformers.YosoForMaskedLM  < source > ( config  )   Parameters   config (YosoConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    YOSO Model with a language modeling head on top.
This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = N


YosoForMaskedLM

one labels: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.MaskedLMOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length)


YosoForMaskedLM

, optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to direc


YosoForMaskedLM

tly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Labels for computing the masked language modeling loss. Indices should be in [-100, 0, ..., config.vocab_size] (see input_ids docstring) Tokens with indices set to -100 are ignored (masked), the
los


YosoForMaskedLM

s is only computed for the tokens with labels in [0, ..., config.vocab_size].   Returns
transformers.modeling_outputs.MaskedLMOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.MaskedLMOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (YosoConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Masked language modeling (MLM) loss.
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the out


YosoForMaskedLM

put of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The YosoForMaskedLM forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post


YosoForMaskedLM

processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, YosoForMaskedLM
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("uw-madison/yoso-4096")
>>> model = YosoForMaskedLM.from_pretrained("uw-madison/yoso-4096")
>>> inputs = tokenizer("The capital of France is [MASK].", return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> # retrieve index of [MASK]
>>> mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]
>>> predicted_token_id = logits[0, mask_token_index].argmax(axis=-1)
>>> labels = tokenizer("The capital of France is Paris.", return_tensors="pt")["input_ids"]
>>> # mask labels of non-[MASK] tokens
>>> labels = torch.where(inputs.input_ids == tokenizer.mask_token_id, labels, -100)
>>> outputs = model(**inputs, labels=lab


YosoForSequenceClassification

class transformers.YosoForSequenceClassification  < source > ( config  )   Parameters   config (YosoConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    YOSO Model transformer with a sequence classification/regression head on top (a linear layer on top of
the pooled output) e.g. for GLUE tasks.
This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optiona


YosoForSequenceClassification

l[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.SequenceClassifierOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongT


YosoForSequenceClassification

ensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose t


YosoForSequenceClassification

o directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss i


YosoForSequenceClassification

s computed (Cross-Entropy).   Returns
transformers.modeling_outputs.SequenceClassifierOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.SequenceClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (YosoConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the outpu


YosoForSequenceClassification

t of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The YosoForSequenceClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example of single-label classification:   Co


YosoForSequenceClassification

pied >>> import torch
>>> from transformers import AutoTokenizer, YosoForSequenceClassification
>>> tokenizer = AutoTokenizer.from_pretrained("uw-madison/yoso-4096")
>>> model = YosoForSequenceClassification.from_pretrained("uw-madison/yoso-4096")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_class_id = logits.argmax().item()
>>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`
>>> num_labels = len(model.config.id2label)
>>> model = YosoForSequenceClassification.from_pretrained("uw-madison/yoso-4096", num_labels=num_labels)
>>> labels = torch.tensor([1])
>>> loss = model(**inputs, labels=labels).loss  Example of multi-label classification:   Copied >>> import torch
>>> from transformers import AutoTokenizer, YosoForSequenceClassification
>>> tokenizer = A


YosoForSequenceClassification

utoTokenizer.from_pretrained("uw-madison/yoso-4096")
>>> model = YosoForSequenceClassification.from_pretrained("uw-madison/yoso-4096", problem_type="multi_label_classification")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_class_ids = torch.arange(0, logits.shape[-1])[torch.sigmoid(logits).squeeze(dim=0) > 0.5]
>>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`
>>> num_labels = len(model.config.id2label)
>>> model = YosoForSequenceClassification.from_pretrained(
...     "uw-madison/yoso-4096", num_labels=num_labels, problem_type="multi_label_classification"
... )
>>> labels = torch.sum(
...     torch.nn.functional.one_hot(predicted_class_ids[None, :].clone(), num_classes=num_labels), dim=1
... ).to(torch.float)
>>> loss = model(**inputs, labels=labels)


YosoForMultipleChoice

class transformers.YosoForMultipleChoice  < source > ( config  )   Parameters   config (YosoConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    YOSO Model with a multiple choice classification head on top (a linear layer on top of
the pooled output and a softmax) e.g. for RocStories/SWAG tasks.
This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mas


YosoForMultipleChoice

k: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.MultipleChoiceModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, num_choices, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, num_choices, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are mas


YosoForMultipleChoice

ked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, num_choices, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, num_choices, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size


YosoForMultipleChoice

, num_choices, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the multiple choice classification loss. Indices should be in [0, ..., num_choices-1] where


YosoForMultipleChoice

num_choices is the size of the second dimension of the input tensors. (See
input_ids above)   Returns
transformers.modeling_outputs.MultipleChoiceModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.MultipleChoiceModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (YosoConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification loss.
logits (torch.FloatTensor of shape (batch_size, num_choices)) — num_choices is the second dimension of the input tensors. (see input_ids above).
Classification scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the outpu


YosoForMultipleChoice

t of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The YosoForMultipleChoice forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps


YosoForMultipleChoice

while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, YosoForMultipleChoice
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("uw-madison/yoso-4096")
>>> model = YosoForMultipleChoice.from_pretrained("uw-madison/yoso-4096")
>>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
>>> choice0 = "It is eaten with a fork and a knife."
>>> choice1 = "It is eaten while held in the hand."
>>> labels = torch.tensor(0).unsqueeze(0)  # choice0 is correct (according to Wikipedia ;)), batch size 1
>>> encoding = tokenizer([prompt, prompt], [choice0, choice1], return_tensors="pt", padding=True)
>>> outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()}, labels=labels)  # batch size is 1
>>> # the linear classifier still needs to be trained
>>> loss = outputs.loss
>>> logits = outputs.logit


YosoForTokenClassification

class transformers.YosoForTokenClassification  < source > ( config  )   Parameters   config (YosoConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    YOSO Model with a token classification head on top (a linear layer on top of
the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks.
This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[


YosoForTokenClassification

torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.TokenClassifierOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,


YosoForTokenClassification

0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTe


YosoForTokenClassification

nsor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Labels for computing the token classification loss. Indices


YosoForTokenClassification

 should be in [0, ..., config.num_labels - 1].   Returns
transformers.modeling_outputs.TokenClassifierOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.TokenClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (YosoConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided)  — Classification loss.
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.num_labels)) — Classification scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of ea


YosoForTokenClassification

ch layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The YosoForTokenClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   C


YosoForTokenClassification

opied >>> from transformers import AutoTokenizer, YosoForTokenClassification
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("uw-madison/yoso-4096")
>>> model = YosoForTokenClassification.from_pretrained("uw-madison/yoso-4096")
>>> inputs = tokenizer(
...     "HuggingFace is a company based in Paris and New York", add_special_tokens=False, return_tensors="pt"
... )
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_token_class_ids = logits.argmax(-1)
>>> # Note that tokens are classified rather then input words which means that
>>> # there might be more predicted token classes than words.
>>> # Multiple token classes might account for the same word
>>> predicted_tokens_classes = [model.config.id2label[t.item()] for t in predicted_token_class_ids[0]]
>>> labels = predicted_token_class_ids
>>> loss = model(**inputs, labels=labels).


YosoForQuestionAnswering

class transformers.YosoForQuestionAnswering  < source > ( config  )   Parameters   config (YosoConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    YOSO Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear
layers on top of the hidden-states output to compute span start logits and span end logits).
This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.T


YosoForQuestionAnswering

ensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None start_positions: typing.Optional[torch.Tensor] = None end_positions: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.QuestionAnsweringModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of


YosoForQuestionAnswering

shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want mor


YosoForQuestionAnswering

e control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   start_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the start of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
are not taken into account for computing the loss.   end_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the end of


YosoForQuestionAnswering

the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
are not taken into account for computing the loss.   Returns
transformers.modeling_outputs.QuestionAnsweringModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.QuestionAnsweringModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (YosoConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Total span extraction loss is the sum of a Cross-Entropy for the start and end positions.
start_logits (torch.FloatTensor of shape (batch_size, sequence_length)) — Span-start scores (before SoftMax).
end_logits (torch.FloatTensor of shape (batch_size, sequence_length)) — Span-end scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor)


YosoForQuestionAnswering

, optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The YosoForQuestionAnswering forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this


YosoForQuestionAnswering

since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, YosoForQuestionAnswering
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("uw-madison/yoso-4096")
>>> model = YosoForQuestionAnswering.from_pretrained("uw-madison/yoso-4096")
>>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
>>> inputs = tokenizer(question, text, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> answer_start_index = outputs.start_logits.argmax()
>>> answer_end_index = outputs.end_logits.argmax()
>>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
>>> # target is "nice puppet"
>>> target_start_index = torch.tensor([14])
>>> target_end_index = torch.tensor([15])
>>> outputs = model(**inputs, start_positions=target_start_index, end_positions=target_end_index)
>>> loss = outputs.lo


Overview

SAM (Segment Anything Model) was proposed in Segment Anything by Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alex Berg, Wan-Yen Lo, Piotr Dollar, Ross Girshick.
The model can be used to predict segmentation masks of any object of interest given an input image.
The abstract from the paper is the following:
We introduce the Segment Anything (SA) project: a new task, model, and dataset for image segmentation. Using our efficient model in a data collection loop, we built the largest segmentation dataset to date (by far), with over 1 billion masks on 11M licensed and privacy respecting images. The model is designed and trained to be promptable, so it can transfer zero-shot to new image distributions and tasks. We evaluate its capabilities on numerous tasks and find that its zero-shot performance is impressive — often competitive with or even superior to prior fully supervised results. We are re


Overview

leasing the Segment Anything Model (SAM) and corresponding dataset (SA-1B) of 1B masks and 11M images at https://segment-anything.com to foster research into foundation models for computer vision.
Tips:
The model predicts binary masks that states the presence or not of the object of interest given an image. The model predicts much better results if input 2D points and/or input bounding boxes are provided You can prompt multiple points for the same image, and predict a single mask. Fine-tuning the model is not supported yet According to the paper, textual input should be also supported. However, at this time of writing this seems to be not supported according to the official repository.
This model was contributed by ybelkada and ArthurZ.
The original code can be found here.
Below is an example on how to run mask generation given an image and a 2D point:
  Copied import torch
from PIL import Image
import requests
from transformers import SamModel, SamProcessor
devic


Overview

e = "cuda" if torch.cuda.is_available() else "cpu"
model = SamModel.from_pretrained("facebook/sam-vit-huge").to(device)
processor = SamProcessor.from_pretrained("facebook/sam-vit-huge")
img_url = "https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png"
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert("RGB")
input_points = [[[450, 600]]]  # 2D location of a window in the image
inputs = processor(raw_image, input_points=input_points, return_tensors="pt").to(device)
outputs = model(**inputs)
masks = processor.image_processor.post_process_masks(
    outputs.pred_masks.cpu(), inputs["original_sizes"].cpu(), inputs["reshaped_input_sizes"].cpu()
)
scores = outputs.iou_scores
Resources:
Demo notebook for using the model. Demo notebook for using the automatic mask generation pipeline. Demo notebook for inference with MedSAM, a fine-tuned version of SAM on the medical domain. Demo notebook for fine-tuning the model on custom data.


SamConfig

class transformers.SamConfig  < source > ( vision_config = None prompt_encoder_config = None mask_decoder_config = None initializer_range = 0.02 **kwargs  )   Parameters   vision_config (Union[dict, SamVisionConfig], optional) —
Dictionary of configuration options used to initialize SamVisionConfig.   prompt_encoder_config (Union[dict, SamPromptEncoderConfig], optional) —
Dictionary of configuration options used to initialize SamPromptEncoderConfig.   mask_decoder_config (Union[dict, SamMaskDecoderConfig], optional) —
Dictionary of configuration options used to initialize SamMaskDecoderConfig.   kwargs (optional) —
Dictionary of keyword arguments.    SamConfig is the configuration class to s


SamConfig

tore the configuration of a SamModel. It is used to instantiate a
SAM model according to the specified arguments, defining the vision model, prompt-encoder model and mask decoder
configs. Instantiating a configuration with the defaults will yield a similar configuration to that of the
SAM-ViT-H facebook/sam-vit-huge architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import (
...     SamVisionConfig,
...     SamPromptEncoderConfig,
...     SamMaskDecoderConfig,
...     SamModel,
... )
>>> # Initializing a SamConfig with `"face


SamConfig

book/sam-vit-huge"` style configuration
>>> configuration = SamConfig()
>>> # Initializing a SamModel (with random weights) from the `"facebook/sam-vit-huge"` style configuration
>>> model = SamModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config
>>> # We can also initialize a SamConfig from a SamVisionConfig, SamPromptEncoderConfig, and SamMaskDecoderConfig
>>> # Initializing SAM vision, SAM Q-Former and language model configurations
>>> vision_config = SamVisionConfig()
>>> prompt_encoder_config = SamPromptEncoderConfig()
>>> mask_decoder_config = SamMaskDecoderConfig()
>>> config = SamConfig(vision_config, prompt_encoder_config, mask_decoder_config


SamVisionConfig

class transformers.SamVisionConfig  < source > ( hidden_size = 768 output_channels = 256 num_hidden_layers = 12 num_attention_heads = 12 num_channels = 3 image_size = 1024 patch_size = 16 hidden_act = 'gelu' layer_norm_eps = 1e-06 attention_dropout = 0.0 initializer_range = 1e-10 qkv_bias = True mlp_ratio = 4.0 use_abs_pos = True use_rel_pos = True window_size = 14 global_attn_indexes = [2, 5, 8, 11] num_pos_feats = 128 mlp_dim = None **kwargs  )   Parameters   hidden_size (int, optional, defaults to 768) —
Dimensionality of the encoder layers and the pooler layer.   output_channels (int, optional, defaults to 256) —
Dimensionality of the output channels in the Patch Encoder.   num_hidden_layers (int, optional, defaults to 12) —
Number of hidden


SamVisionConfig

layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 12) —
Number of attention heads for each attention layer in the Transformer encoder.   num_channels (int, optional, defaults to 3) —
Number of channels in the input image.   image_size (int, optional, defaults to 1024) —
Expected resolution. Target size of the resized input image.   patch_size (int, optional, defaults to 16) —
Size of the patches to be extracted from the input image.   hidden_act (str, optional, defaults to "gelu") —
The non-linear activation function (function or string)   layer_norm_eps (float, optional, defaults to 1e-06) —
The epsilon used by the layer normalization layers.   attention_dropout (float, optional, defaults to 0.0) —
The dropout


SamVisionConfig

ratio for the attention probabilities.   initializer_range (float, optional, defaults to 1e-10) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   qkv_bias (bool, optional, defaults to True) —
Whether to add a bias to query, key, value projections.   mlp_ratio (float, optional, defaults to 4.0) —
Ratio of mlp hidden dim to embedding dim.   use_abs_pos (bool, optional, defaults to True) —
Whether to use absolute position embedding.   use_rel_pos (bool, optional, defaults to True) —
Whether to use relative position embedding.   window_size (int, optional, defaults to 14) —
Window size for relative position.   global_attn_indexes (List[int], optional, defaults to [2, 5, 8, 11]) —
The indexes of the g


SamVisionConfig

lobal attention layers.   num_pos_feats (int, optional, defaults to 128) —
The dimensionality of the position embedding.   mlp_dim (int, optional) —
The dimensionality of the MLP layer in the Transformer encoder. If None, defaults to mlp_ratio * hidden_size.    This is the configuration class to store the configuration of a SamVisionModel. It is used to instantiate a SAM
vision encoder according to the specified arguments, defining the model architecture. Instantiating a configuration
defaults will yield a similar configuration to that of the SAM ViT-h
facebook/sam-vit-huge architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more informati


SamMaskDecoderConfig

class transformers.SamMaskDecoderConfig  < source > ( hidden_size = 256 hidden_act = 'relu' mlp_dim = 2048 num_hidden_layers = 2 num_attention_heads = 8 attention_downsample_rate = 2 num_multimask_outputs = 3 iou_head_depth = 3 iou_head_hidden_dim = 256 layer_norm_eps = 1e-06 **kwargs  )   Parameters   hidden_size (int, optional, defaults to 256) —
Dimensionality of the hidden states.   hidden_act (str, optional, defaults to "relu") —
The non-linear activation function used inside the SamMaskDecoder module.   mlp_dim (int, optional, defaults to 2048) —
Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.   num_hidden_layers (int, optional, defaults to 2) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 8) —
Number of attention heads for each attention layer in the Transformer encoder.   attention_downsample_rate (int, optional, defaults to 2) —
The downsampling rate of the atte


SamMaskDecoderConfig

ntion layer.   num_multimask_outputs (int, optional, defaults to 3) —
The number of outputs from the SamMaskDecoder module. In the Segment Anything paper, this is set to 3.   iou_head_depth (int, optional, defaults to 3) —
The number of layers in the IoU head module.   iou_head_hidden_dim (int, optional, defaults to 256) —
The dimensionality of the hidden states in the IoU head module.   layer_norm_eps (float, optional, defaults to 1e-06) —
The epsilon used by the layer normalization layers.    This is the configuration class to store the configuration of a SamMaskDecoder. It is used to instantiate a SAM
mask decoder to the specified arguments, defining the model architecture. Instantiating a configuration defaults
will yield a similar configuration to that of the SAM-vit-h
facebook/sam-vit-huge architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information


SamPromptEncoderConfig

class transformers.SamPromptEncoderConfig  < source > ( hidden_size = 256 image_size = 1024 patch_size = 16 mask_input_channels = 16 num_point_embeddings = 4 hidden_act = 'gelu' layer_norm_eps = 1e-06 **kwargs  )   Parameters   hidden_size (int, optional, defaults to 256) —
Dimensionality of the hidden states.   image_size (int, optional, defaults to 1024) —
The expected output resolution of the image.   patch_size (int, optional, defaults to 16) —
The size (resolution) of each patch.   mask_input_channels (int, optional, defaults to 16) —
The number of channels to be fed to the MaskDecoder module.   num_point_embeddings (int, optional,


SamPromptEncoderConfig

 defaults to 4) —
The number of point embeddings to be used.   hidden_act (str, optional, defaults to "gelu") —
The non-linear activation function in the encoder and pooler.    This is the configuration class to store the configuration of a SamPromptEncoder. The SamPromptEncoder
module is used to encode the input 2D points and bounding boxes. Instantiating a configuration defaults will yield
a similar configuration to that of the SAM-vit-h
facebook/sam-vit-huge architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.


SamProcessor

class transformers.SamProcessor  < source > ( image_processor  )   Parameters   image_processor (SamImageProcessor) —
An instance of SamImageProcessor. The image processor is a required input.    Constructs a SAM processor which wraps a SAM image processor and an 2D points & Bounding boxes processor into a
single processor. SamProcessor offers all the functionalities of SamImageProcessor. See the docstring of
call() for more information.


SamImageProcessor

class transformers.SamImageProcessor  < source > ( do_resize: bool = True size: typing.Dict[str, int] = None resample: Resampling = <Resampling.BILINEAR: 2> do_rescale: bool = True rescale_factor: typing.Union[int, float] = 0.00392156862745098 do_normalize: bool = True image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float], NoneType] = None do_pad: bool = True pad_size: int = None do_convert_rgb: bool = True **kwargs  )   Parameters   do_resize (bool, optional, defaults to True) —
Whether to resize the image’s (height, width) dimensions to the specified size. Can be overridden by the
do_resize parameter in the preprocess method.   size (dict, optional, defaults to {"longest_edge" -- 1024}):
Size of the output image after resizing. Resizes the longest edge of the image to match
size["longest_edge"] while maintaining the aspect ratio. Can be overridden by the size parameter in the
preprocess method.   resam


SamImageProcessor

ple (PILImageResampling, optional, defaults to Resampling.BILINEAR) —
Resampling filter to use if resizing the image. Can be overridden by the resample parameter in the
preprocess method.   do_rescale (bool, optional, defaults to True) —
Wwhether to rescale the image by the specified scale rescale_factor. Can be overridden by the
do_rescale parameter in the preprocess method.   rescale_factor (int or float, optional, defaults to 1/255) —
Scale factor to use if rescaling the image. Only has an effect if do_rescale is set to True. Can be
overridden by the rescale_factor parameter in the preprocess method.   do_normalize (bool, optional, defaults to True) —
Whether to normalize the image. Can be overridden by the do_normalize parameter in the preprocess
method. Can be overridden by the do_normalize parameter in the preprocess method.   image_mean (float or List[float], optional, defaults to IMAGENET_DEFAULT_MEAN) —
Mean to use if normalizing the image. This is a float or


SamImageProcessor

 list of floats the length of the number of
channels in the image. Can be overridden by the image_mean parameter in the preprocess method. Can be
overridden by the image_mean parameter in the preprocess method.   image_std (float or List[float], optional, defaults to IMAGENET_DEFAULT_STD) —
Standard deviation to use if normalizing the image. This is a float or list of floats the length of the
number of channels in the image. Can be overridden by the image_std parameter in the preprocess method.
Can be overridden by the image_std parameter in the preprocess method.   do_pad (bool, optional, defaults to True) —
Whether to pad the image to the specified pad_size. Can be overridden by the do_pad parameter in the
preprocess method.   pad_size (dict, optional, defaults to {"height" -- 1024, "width": 1024}):
Size of the output image after padding. Can be overridden by the pad_size parameter in the preprocess
method.   do_convert_rgb (bool, optional, defaults to True) —
Wheth


SamImageProcessor

er to convert the image to RGB.    Constructs a SAM image processor.  filter_masks  < source > ( masks iou_scores original_size cropped_box_image pred_iou_thresh = 0.88 stability_score_thresh = 0.95 mask_threshold = 0 stability_score_offset = 1 return_tensors = 'pt'  )   Parameters   masks (Union[torch.Tensor, tf.Tensor]) —
Input masks.   iou_scores (Union[torch.Tensor, tf.Tensor]) —
List of IoU scores.   original_size (Tuple[int,int]) —
Size of the orginal image.   cropped_box_image (np.array) —
The cropped image.   pred_iou_thresh (float, optional, defaults to 0.88) —
The threshold for the iou scores.   stability_score_thresh (float, optional, defaults to 0.95) —
The threshold for the stability score.   mask_threshold (float, optional, defaults to 0) —
The threshold for the predicted masks.   stability_score_offset (float, optional, defaults to 1) —
The offset for the stability score used in the _compute_stability_score method.   return_tensors (str, optional, defau


SamImageProcessor

lts to pt) —
If pt, returns torch.Tensor. If tf, returns tf.Tensor.    Filters the predicted masks by selecting only the ones that meets several criteria. The first criterion being
that the iou scores needs to be greater than pred_iou_thresh. The second criterion is that the stability
score needs to be greater than stability_score_thresh. The method also converts the predicted masks to
bounding boxes and pad the predicted masks if necessary.  generate_crop_boxes  < source > ( image target_size crop_n_layers: int = 0 overlap_ratio: float = 0.3413333333333333 points_per_crop: typing.Optional[int] = 32 crop_n_points_downscale_factor: typing.Optional[typing.List[int]] = 1 device: typing.Optional[ForwardRef('torch.device')] = None input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None return_tensors: str = 'pt'  )   Parameters   image (np.array) —
Input original image   target_size (int) —
Target size of the resized image   crop_n_


SamImageProcessor

layers (int, optional, defaults to 0) —
If >0, mask prediction will be run again on crops of the image. Sets the number of layers to run, where
each layer has 2**i_layer number of image crops.   overlap_ratio (float, optional, defaults to 512/1500) —
Sets the degree to which crops overlap. In the first crop layer, crops will overlap by this fraction of
the image length. Later layers with more crops scale down this overlap.   points_per_crop (int, optional, defaults to 32) —
Number of points to sample from each crop.   crop_n_points_downscale_factor (List[int], optional, defaults to 1) —
The number of points-per-side sampled in layer n is scaled down by crop_n_points_downscale_factor**n.   device (torch.device, optional, defaults to None) —
Device to use for the computation. If None, cpu will be used.   input_data_format (str or ChannelDimension, optional) —
The channel dimension format of the input image. If not provided, it will be inferred.   return_tensors (str, op


SamImageProcessor

tional, defaults to pt) —
If pt, returns torch.Tensor. If tf, returns tf.Tensor.    Generates a list of crop boxes of different sizes. Each layer has (2i)2 boxes for the ith layer.  pad_image  < source > ( image: ndarray pad_size: typing.Dict[str, int] data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None **kwargs  )   Parameters   image (np.ndarray) —
Image to pad.   pad_size (Dict[str, int]) —
Size of the output image after padding.   data_format (str or ChannelDimension, optional) —
The data format of the image. Can be either “channels_first” or “channels_last”. If None, the
data_format of the image will be used.   input_data_format (str or ChannelDimension, optional) —
The channel dimension format of the input image. If not provided, it will be inferred.    Pad an image to (pad_size["height"], pad_size["width"]) with zeros to the ri


SamImageProcessor

ght and bottom.  post_process_for_mask_generation  < source > ( all_masks all_scores all_boxes crops_nms_thresh return_tensors = 'pt'  )   Parameters   all_masks (Union[List[torch.Tensor], List[tf.Tensor]]) —
List of all predicted segmentation masks   all_scores (Union[List[torch.Tensor], List[tf.Tensor]]) —
List of all predicted iou scores   all_boxes (Union[List[torch.Tensor], List[tf.Tensor]]) —
List of all bounding boxes of the predicted masks   crops_nms_thresh (float) —
Threshold for NMS (Non Maximum Suppression) algorithm.   return_tensors (str, optional, defaults to pt) —
If pt, returns torch.Tensor. If tf, returns tf.Tensor.    Post processes mask that are generated by calling the Non Maximum Suppression algorithm on the predicted masks.  post_process_masks  < source > ( masks original_sizes reshaped_input_sizes mask_threshold = 0.0 binarize = True pad_size = None return_tensors = 'pt'  ) → (Union[torch.Tensor, tf.Tensor])  Parameters   masks (Union[List[torc


SamImageProcessor

h.Tensor], List[np.ndarray], List[tf.Tensor]]) —
Batched masks from the mask_decoder in (batch_size, num_channels, height, width) format.   original_sizes (Union[torch.Tensor, tf.Tensor, List[Tuple[int,int]]]) —
The original sizes of each image before it was resized to the model’s expected input shape, in (height,
width) format.   reshaped_input_sizes (Union[torch.Tensor, tf.Tensor, List[Tuple[int,int]]]) —
The size of each image as it is fed to the model, in (height, width) format. Used to remove padding.   mask_threshold (float, optional, defaults to 0.0) —
The threshold to use for binarizing the masks.   binarize (bool, optional, defaults to True) —
Whether to binarize the masks.   pad_size (int, optional, defaults to self.pad_size) —
The target size the images were padded to before being passed to the model. If None, the target size is
assumed to be the processor’s pad_size.   return_tensors (str, optional, defaults to "pt") —
If "pt", return PyTorch tensors. If "


SamImageProcessor

tf", return TensorFlow tensors.   Returns
(Union[torch.Tensor, tf.Tensor])

Batched masks in batch_size, num_channels, height, width) format, where
(height, width) is given by original_size.
  Remove padding and upscale masks to the original image size.  preprocess  < source > ( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]] do_resize: typing.Optional[bool] = None size: typing.Union[typing.Dict[str, int], NoneType] = None resample: typing.Optional[ForwardRef('PILImageResampling')] = None do_rescale: typing.Optional[bool] = None rescale_factor: typing.Union[int, float, NoneType] = None do_normalize: typing.Optional[bool] = None image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float], NoneType] = None do_pad: typing.Optional[bool] = None pad_size:


SamImageProcessor

 typing.Union[typing.Dict[str, int], NoneType] = None do_convert_rgb: bool = None return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None data_format: ChannelDimension = <ChannelDimension.FIRST: 'channels_first'> input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None **kwargs  )   Parameters   images (ImageInput) —
Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If
passing in images with pixel values between 0 and 1, set do_rescale=False.   do_resize (bool, optional, defaults to self.do_resize) —
Whether to resize the image.   size (Dict[str, int], optional, defaults to self.size) —
Controls the size of the image after resize. The longest edge of the image is resized to
size["longest_edge"] whilst preserving the aspect ratio.   resample (PILImageResampling, optional, defaults to self.resample) —
PILImageResampling filter to use when resizing th


SamImageProcessor

e image e.g. PILImageResampling.BILINEAR.   do_rescale (bool, optional, defaults to self.do_rescale) —
Whether to rescale the image pixel values by rescaling factor.   rescale_factor (int or float, optional, defaults to self.rescale_factor) —
Rescale factor to apply to the image pixel values.   do_normalize (bool, optional, defaults to self.do_normalize) —
Whether to normalize the image.   image_mean (float or List[float], optional, defaults to self.image_mean) —
Image mean to normalize the image by if do_normalize is set to True.   image_std (float or List[float], optional, defaults to self.image_std) —
Image standard deviation to normalize the image by if do_normalize is set to True.   do_pad (bool, optional, defaults to self.do_pad) —
Whether to pad the image.   pad_size (Dict[str, int], optional, defaults to self.pad_size) —
Controls the size of the padding applied to the image. The image is padded to pad_size["height"] and
pad_size["width"] if do_pad is set to Tr


SamImageProcessor

ue.   do_convert_rgb (bool, optional, defaults to self.do_convert_rgb) —
Whether to convert the image to RGB.   return_tensors (str or TensorType, optional) —
The type of tensors to return. Can be one of:
Unset: Return a list of np.ndarray.
TensorType.TENSORFLOW or 'tf': Return a batch of type tf.Tensor.
TensorType.PYTORCH or 'pt': Return a batch of type torch.Tensor.
TensorType.NUMPY or 'np': Return a batch of type np.ndarray.
TensorType.JAX or 'jax': Return a batch of type jax.numpy.ndarray.
   data_format (ChannelDimension or str, optional, defaults to ChannelDimension.FIRST) —
The channel dimension format for the output image. Can be one of:
"channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
"channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
Unset: Use the channel dimension format of the input image.
   input_data_format (ChannelDimension or str, optional) —
The channel dimension format


SamImageProcessor

for the input image. If unset, the channel dimension format is inferred
from the input image. Can be one of:
"channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
"channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
"none" or ChannelDimension.NONE: image in (height, width) format.
    Preprocess an image or batch of images.  resize  < source > ( image: ndarray size: typing.Dict[str, int] resample: Resampling = <Resampling.BICUBIC: 3> data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None **kwargs  ) → np.ndarray  Parameters   image (np.ndarray) —
Image to resize.   size (Dict[str, int]) —
Dictionary in the format {"longest_edge": int} specifying the size of the output image. The longest
edge of the image will be resized to the specified size, while the other edge wi


SamImageProcessor

ll be resized to
maintain the aspect ratio.
resample —
PILImageResampling filter to use when resizing the image e.g. PILImageResampling.BILINEAR.   data_format (ChannelDimension or str, optional) —
The channel dimension format for the output image. If unset, the channel dimension format of the input
image is used. Can be one of:
"channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
"channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
   input_data_format (ChannelDimension or str, optional) —
The channel dimension format for the input image. If unset, the channel dimension format is inferred
from the input image. Can be one of:
"channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
"channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
   Returns
np.ndarray

The resized image.
  Resize an image to (size["height"], size["width


SamModel

class transformers.SamModel  < source > ( config  )   Parameters   config (SamConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Segment Anything Model (SAM) for generating segmentation masks, given an input image and  optional 2D location and bounding boxes.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( pixel_values: t


SamModel

yping.Optional[torch.FloatTensor] = None input_points: typing.Optional[torch.FloatTensor] = None input_labels: typing.Optional[torch.LongTensor] = None input_boxes: typing.Optional[torch.FloatTensor] = None input_masks: typing.Optional[torch.LongTensor] = None image_embeddings: typing.Optional[torch.FloatTensor] = None multimask_output: bool = True attention_similarity: typing.Optional[torch.FloatTensor] = None target_embedding: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None **kwargs  )   Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using SamProcessor. See SamProcessor.__call__() for
details.   input_points (torch.FloatTensor of shape (batch_size, num_points, 2)) —


SamModel

Input 2D spatial points, this is used by the prompt encoder to encode the prompt. Generally yields to much
better results. The points can be obtained by passing a list of list of list to the processor that will
create corresponding torch tensors of dimension 4. The first dimension is the image batch size, the
second dimension is the point batch size (i.e. how many segmentation masks do we want the model to predict
per input point), the third dimension is the number of points per segmentation mask (it is possible to pass
multiple points for a single mask), and the last dimension is the x (vertical) and y (horizontal)
coordinates of the point. If a different number of points is passed either for each image, or for each
mask, the processor will create “PAD” points that will correspond to the (0, 0) coordinate, and the
computation of the embedding will be skipped for these points using the labels.


SamModel

  input_labels (torch.LongTensor of shape (batch_size, point_batch_size, num_points)) —
Input labels for the points, this is used by the prompt encoder to encode the prompt. According to the
official implementation, there are 3 types of labels
1: the point is a point that contains the object of interest
0: the point is a point that does not contain the object of interest
-1: the point corresponds to the background
We added the label:
-10: the point is a padding point, thus should be ignored by the prompt encoder
The padding labels should be automatically done by the processor.   input_boxes (torch.FloatTensor of shape (batch_size, num_boxes, 4)) —
Input boxes for the points, this is used by the prompt encoder to encode the prompt. Generally yields to
much better generated masks. The boxes can be obtained by passing a list of list of list to the processor,
that will generate a torch tensor, with


SamModel

 each dimension corresponding respectively to the image batch
size, the number of boxes per image and the coordinates of the top left and botton right point of the box.
In the order (x1, y1, x2, y2):
x1: the x coordinate of the top left point of the input box
y1: the y coordinate of the top left point of the input box
x2: the x coordinate of the bottom right point of the input box
y2: the y coordinate of the bottom right point of the input box
   input_masks (torch.FloatTensor of shape (batch_size, image_size, image_size)) —
SAM model also accepts segmentation masks as input. The mask will be embedded by the prompt encoder to
generate a corresponding embedding, that will be fed later on to the mask decoder. These masks needs to be
manually fed by the user, and they need to be of shape (batch_size, image_size, image_size).   image_embeddings (torch.FloatTensor of shape (batch_size, output_channe


SamModel

ls, window_size, window_size)) —
Image embeddings, this is used by the mask decder to generate masks and iou scores. For more memory
efficient computation, users can first retrieve the image embeddings using the get_image_embeddings
method, and then feed them to the forward method instead of feeding the pixel_values.   multimask_output (bool, optional) —
In the original implementation and paper, the model always outputs 3 masks per image (or per point / per
bounding box if relevant). However, it is possible to just output a single mask, that corresponds to the
“best” mask, by specifying multimask_output=False.   attention_similarity (torch.FloatTensor, optional) —
Attention similarity tensor, to be provided to the mask decoder for target-guided attention in case the
model is used for personalization as introduced in PerSAM.   target_embedding (torch.FloatTensor, optional) —
Embedding of the tar


SamModel

get concept, to be provided to the mask decoder for target-semantic prompting in case
the model is used for personalization as introduced in PerSAM.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.
Example —    The SamModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.


TFSamModel

class transformers.TFSamModel  < source > ( *args **kwargs  )   Parameters   config (SamConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Segment Anything Model (SAM) for generating segmentation masks, given an input image and  optional 2D location and bounding boxes.
This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a TensorFlow tf.keras.Model
subclass. Use it as a regular TensorFlow Model and refer to the TensorFlow documentation for all matter related to
general usage and behavior.  call  < source > ( pixel_values: TFModelInputType


TFSamModel

 | None = None input_points: tf.Tensor | None = None input_labels: tf.Tensor | None = None input_boxes: tf.Tensor | None = None input_masks: tf.Tensor | None = None image_embeddings: tf.Tensor | None = None multimask_output: bool = True output_attentions: bool | None = None output_hidden_states: bool | None = None return_dict: bool | None = None training: bool = False **kwargs  )   Parameters   pixel_values (tf.Tensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using SamProcessor. See SamProcessor.__call__() for
details.   input_points (tf.Tensor of shape (batch_size, num_points, 2)) —
Input 2D spatial points, this is used by the prompt encoder to encode the prompt. Generally yields to much
better results. The points can be obtained by passing a list of list of list to the processor that will
create corresponding tf tensors of dimension 4. The first dimension is the image


TFSamModel

batch size, the second
dimension is the point batch size (i.e. how many segmentation masks do we want the model to predict per
input point), the third dimension is the number of points per segmentation mask (it is possible to pass
multiple points for a single mask), and the last dimension is the x (vertical) and y (horizontal)
coordinates of the point. If a different number of points is passed either for each image, or for each
mask, the processor will create “PAD” points that will correspond to the (0, 0) coordinate, and the
computation of the embedding will be skipped for these points using the labels.   input_labels (tf.Tensor of shape (batch_size, point_batch_size, num_points)) —
Input labels for the points, this is used by the prompt encoder to encode the prompt. According to the
official implementation, there are 3 types of labels
1: the point is a point that contains the object of interest
0: the point is a point that


TFSamModel

 does not contain the object of interest
-1: the point corresponds to the background
We added the label:
-10: the point is a padding point, thus should be ignored by the prompt encoder
The padding labels should be automatically done by the processor.   input_boxes (tf.Tensor of shape (batch_size, num_boxes, 4)) —
Input boxes for the points, this is used by the prompt encoder to encode the prompt. Generally yields to
much better generated masks. The boxes can be obtained by passing a list of list of list to the processor,
that will generate a tf tensor, with each dimension corresponding respectively to the image batch size,
the number of boxes per image and the coordinates of the top left and botton right point of the box. In the
order (x1, y1, x2, y2):
x1: the x coordinate of the top left point of the input box
y1: the y coordinate of the top left point of the input box
x2: the x coordinate of the bottom right point of the i


TFSamModel

nput box
y2: the y coordinate of the bottom right point of the input box
   input_masks (tf.Tensor of shape (batch_size, image_size, image_size)) —
SAM model also accepts segmentation masks as input. The mask will be embedded by the prompt encoder to
generate a corresponding embedding, that will be fed later on to the mask decoder. These masks needs to be
manually fed by the user, and they need to be of shape (batch_size, image_size, image_size).   image_embeddings (tf.Tensor of shape (batch_size, output_channels, window_size, window_size)) —
Image embeddings, this is used by the mask decder to generate masks and iou scores. For more memory
efficient computation, users can first retrieve the image embeddings using the get_image_embeddings
method, and then feed them to the call method instead of feeding the pixel_values.   multimask_output (bool, optional) —
In the original implementation and paper, the model always outputs 3


TFSamModel

 masks per image (or per point / per
bounding box if relevant). However, it is possible to just output a single mask, that corresponds to the
“best” mask, by specifying multimask_output=False.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.    The TFSamModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores


Overview

The BertGeneration model is a BERT model that can be leveraged for sequence-to-sequence tasks using
EncoderDecoderModel as proposed in Leveraging Pre-trained Checkpoints for Sequence Generation
Tasks by Sascha Rothe, Shashi Narayan, Aliaksei Severyn.
The abstract from the paper is the following:
Unsupervised pretraining of large neural models has recently revolutionized Natural Language Processing. By
warm-starting from the publicly released checkpoints, NLP practitioners have pushed the state-of-the-art on multiple
benchmarks while saving significant amounts of compute time. So far the focus has been mainly on the Natural Language
Understanding tasks. In this paper, we demonstrate the efficacy of pre-trained checkpoints for Sequence Generation. We
developed a Transformer-based sequence-to-sequence model


Overview

 that is compatible with publicly available pre-trained BERT,
GPT-2 and RoBERTa checkpoints and conducted an extensive empirical study on the utility of initializing our model, both
encoder and decoder, with these checkpoints. Our models result in new state-of-the-art results on Machine Translation,
Text Summarization, Sentence Splitting, and Sentence Fusion.
Usage:
The model can be used in combination with the EncoderDecoderModel to leverage two pretrained
BERT checkpoints for subsequent fine-tuning.
  Copied >>> # leverage checkpoints for Bert2Bert model...
>>> # use BERT's cls token as BOS token and sep token as EOS token
>>> encoder = BertGenerationEncoder.from_pretrained("bert-large-uncased", bos_token_id=101, eos_token_id=102)
>>> # add cross attention layers and use BERT's cls token as BOS token a


Overview

nd sep token as EOS token
>>> decoder = BertGenerationDecoder.from_pretrained(
...     "bert-large-uncased", add_cross_attention=True, is_decoder=True, bos_token_id=101, eos_token_id=102
... )
>>> bert2bert = EncoderDecoderModel(encoder=encoder, decoder=decoder)
>>> # create tokenizer...
>>> tokenizer = BertTokenizer.from_pretrained("bert-large-uncased")
>>> input_ids = tokenizer(
...     "This is a long article to summarize", add_special_tokens=False, return_tensors="pt"
... ).input_ids
>>> labels = tokenizer("This is a short summary", return_tensors="pt").input_ids
>>> # train...
>>> loss = bert2bert(input_ids=input_ids, decoder_input_ids=labels, labels=labels).loss
>>> loss.backward()
Pretrained EncoderDecoderModel are also directly available in the model hub, e.g.,
  Copied >>> # instantiate sentence


Overview

 fusion model
>>> sentence_fuser = EncoderDecoderModel.from_pretrained("google/roberta2roberta_L-24_discofuse")
>>> tokenizer = AutoTokenizer.from_pretrained("google/roberta2roberta_L-24_discofuse")
>>> input_ids = tokenizer(
...     "This is the first sentence. This is the second sentence.", add_special_tokens=False, return_tensors="pt"
... ).input_ids
>>> outputs = sentence_fuser.generate(input_ids)
>>> print(tokenizer.decode(outputs[0]))
Tips:
BertGenerationEncoder and BertGenerationDecoder should be used in
combination with EncoderDecoder. For summarization, sentence splitting, sentence fusion and translation, no special tokens are required for the input.
Therefore, no EOS token should be added to the end of the input.
This model was contributed by patrickvonplaten. The original code can be
found her


BertGenerationConfig

class transformers.BertGenerationConfig  < source > ( vocab_size = 50358 hidden_size = 1024 num_hidden_layers = 24 num_attention_heads = 16 intermediate_size = 4096 hidden_act = 'gelu' hidden_dropout_prob = 0.1 attention_probs_dropout_prob = 0.1 max_position_embeddings = 512 initializer_range = 0.02 layer_norm_eps = 1e-12 pad_token_id = 0 bos_token_id = 2 eos_token_id = 1 position_embedding_type = 'absolute' use_cache = True **kwargs  )   Parameters   vocab_size (int, optional, defaults to 50358) —
Vocabulary size of the BERT model. Defines the number of different tokens that can be represented by the
inputs_ids passed when calling BertGeneration.   hidden_size (int, optional, defaults to 1024) —
Dimensionality of the encoder layers and the pooler layer.   num_hidden_layers (int, optional, defaults to 24) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 16) —
Number of attention heads for each a


BertGenerationConfig

ttention layer in the Transformer encoder.   intermediate_size (int, optional, defaults to 3072) —
Dimensionality of the “intermediate” (often called feed-forward) layer in the Transformer encoder.   hidden_act (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "silu" and "gelu_new" are supported.   hidden_dropout_prob (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.   attention_probs_dropout_prob (float, optional, defaults to 0.1) —
The dropout ratio for the attention probabilities.   max_position_embeddings (int, optional, defaults to 512) —
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).   initializer_range (float, optional, defaults to 0.02) —
The standard d


BertGenerationConfig

eviation of the truncated_normal_initializer for initializing all weight matrices.   layer_norm_eps (float, optional, defaults to 1e-12) —
The epsilon used by the layer normalization layers.   position_embedding_type (str, optional, defaults to "absolute") —
Type of position embedding. Choose one of "absolute", "relative_key", "relative_key_query". For
positional embeddings use "absolute". For more information on "relative_key", please refer to
Self-Attention with Relative Position Representations (Shaw et al.).
For more information on "relative_key_query", please refer to Method 4 in Improve Transformer Models
with Better Relative Position Embeddings (Huang et al.).   is_decoder (bool, optional, defaults to False) —
Whether the model is used as a decoder or not. If False, the model is used as an encoder.   use_cache (bool, optional, defaults to True) —
Whether or not the model should return the last key/values attentions (not used by all models). O


BertGenerationConfig

nly
relevant if config.is_decoder=True.    This is the configuration class to store the configuration of a BertGenerationPreTrainedModel. It is used to
instantiate a BertGeneration model according to the specified arguments, defining the model architecture.
Instantiating a configuration with the defaults will yield a similar configuration to that of the BertGeneration
google/bert_for_seq_generation_L-24_bbc_encoder
architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Examples:   Copied >>> from transformers import BertGenerationConfig, BertGenerationEncoder
>>> # Initializing a BertGeneration config
>>> configuration = BertGenerationConfig()
>>> # Initializing a model (with random weights) from the config
>>> model = BertGenerationEncoder(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config


BertGenerationTokenizer

class transformers.BertGenerationTokenizer  < source > ( vocab_file bos_token = '<s>' eos_token = '</s>' unk_token = '<unk>' pad_token = '<pad>' sep_token = '<::::>' sp_model_kwargs: typing.Union[typing.Dict[str, typing.Any], NoneType] = None **kwargs  )   Parameters   vocab_file (str) —
SentencePiece file (generally has a .spm extension) that
contains the vocabulary necessary to instantiate a tokenizer.   eos_token (str, optional, defaults to "</s>") —
The end of sequence token.   bos_token (str, optional, defaults to "<s>") —
The begin of sequence token.   unk_token (str, optional, defaults to "<unk>") —
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.   pad_token (str, optional, defaults to "<pad>") —
The token used for padding, for example when batching sequences of different lengths.   sp_model_kwargs (dict, optional) —
Will be passed to the Sent


BertGenerationTokenizer

encePieceProcessor.__init__() method. The Python wrapper for
SentencePiece can be used, among other things,
to set:
enable_sampling: Enable subword regularization.
nbest_size: Sampling parameters for unigram. Invalid for BPE-Dropout.
nbest_size = {0,1}: No sampling is performed.
nbest_size > 1: samples from the nbest_size results.
nbest_size < 0: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
using forward-filtering-and-backward-sampling algorithm.
alpha: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
BPE-dropout.
    Construct a BertGeneration tokenizer. Based on SentencePiece. This tokenizer inherits from PreTrainedTokenizer which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.  save_vocabulary  < source > ( save_directory: str filename_prefix: typing.Optional[str] = None  )


BertGenerationEncoder

class transformers.BertGenerationEncoder  < source > ( config  )   Parameters   config (BertGenerationConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare BertGeneration model transformer outputting raw hidden-states without any specific head on top. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior. The model can behave as an encoder (with o


BertGenerationEncoder

nly self-attention) as well as a decoder, in which case a layer of
cross-attention is added between the self-attention layers, following the architecture described in Attention is
all you need by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit,
Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. This model should be used when leveraging Bert or Roberta checkpoints for the EncoderDecoderModel class as
described in Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
by Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. To behave as an decoder the model needs to be initialized with the is_decoder argument of the configuration set
to True. To be used in a Seq2Seq model, the model needs to initialized with both is_decoder argument and
add_cross_attention set to True; an encoder_hidden_states is then expected as an input to the forward pass.  forward  < source > (


BertGenerationEncoder

input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None encoder_hidden_states: typing.Optional[torch.Tensor] = None encoder_attention_mask: typing.Optional[torch.Tensor] = None past_key_values: typing.Optional[typing.Tuple[typing.Tuple[torch.FloatTensor]]] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.BaseModelOutputWithPastAndCrossAttentions or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained usi


BertGenerationEncoder

ng AutoTokenizer. See PreTrainedTokenizer.call() and
PreTrainedTokenizer.encode() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (


BertGenerationEncoder

torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   encoder_hidden_states  (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Sequence of hidden-states at the output


BertGenerationEncoder

of the last layer of the encoder. Used in the cross-attention if
the model is configured as a decoder.   encoder_attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in
the cross-attention if the model is configured as a decoder. Mask values selected in [0, 1]: 1 for
tokens that are NOT MASKED, 0 for MASKED tokens.   past_key_values (tuple(tuple(torch.FloatTensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)) —
Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this mod


BertGenerationEncoder

el) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   Returns
transformers.modeling_outputs.BaseModelOutputWithPastAndCrossAttentions or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutputWithPastAndCrossAttentions or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (BertGenerationConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
If past_key_values is used only the last hidden-state of the sequences of shape (batch_size, 1, hidden_size) i


BertGenerationEncoder

s output.
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and optionally if
config.is_encoder_decoder=True 2 additional tensors of shape (batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if
config.is_encoder_decoder=True in the cross-attention blocks) that can be used (see past_key_values
input) to speed up sequential decoding.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, i