Blip2Model

class transformers.Blip2Model  < source > ( config: Blip2Config  )   Parameters   config (Blip2Config) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    BLIP-2 Model for generating text and image features. The model consists of a vision encoder, Querying Transformer
(Q-Former) and a language model. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( pixel_values: FloatTensor input_ids: FloatTensor attention_mas


Blip2Model

k: typing.Optional[torch.LongTensor] = None decoder_input_ids: typing.Optional[torch.LongTensor] = None decoder_attention_mask: typing.Optional[torch.LongTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None labels: typing.Optional[torch.LongTensor] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.blip_2.modeling_blip_2.Blip2ForConditionalGenerationModelOutput or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using Blip2Processor. See Blip2Processor.__call__() for
details.   input_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of input sequence tokens in the vocabulary of the language model. Input tokens can optionally be
provided to serve as text prompt, which the language model can continue.
Indices can be obtained using Blip2Processo


Blip2Model

r. See Blip2Processor.__call__() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   decoder_input_ids (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Indices of decoder input sequence tokens in the vocabulary of the language model. Only relevant in case an
encoder-decoder language model (like T5) is used.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details. What are decoder input IDs?   decoder_attention_mask (torch.BoolTensor of shape (batch_size, target_sequence_length), optional) —
Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also
be used by default.
Only relevant in case an


Blip2Model

encoder-decoder language model (like T5) is used.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.models.blip_2.modeling_blip_2.Blip2ForConditionalGenerationModelOutput or tuple(torch.FloatTensor)

A transformers.models.blip_2.modeling_blip_2.Blip2ForConditionalGenerationModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.blip_2.configuration_blip_2.Blip2VisionConfig'>) and inputs.
loss (torch.FloatTensor, optional, returned when labels


Blip2Model

 is provided, torch.FloatTensor of shape (1,)) — Language modeling loss from the language model.
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head of the language model.
vision_outputs (BaseModelOutputWithPooling) — Outputs of the vision encoder.
qformer_outputs (BaseModelOutputWithPoolingAndCrossAttentions) — Outputs of the Q-Former (Querying Transformer).
language_model_outputs (CausalLMOutputWithPast or Seq2SeqLMOutput) — Outputs of the language model.
  The Blip2Model forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from PIL import Image
>>> import requests
>>> from transformers import Blip2Processor, Blip


Blip2Model

2Model
>>> import torch
>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
>>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16)
>>> model.to(device)
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> prompt = "Question: how many cats are there? Answer:"
>>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(device, torch.float16)
>>> outputs = model(**inputs)  get_text_features  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None decoder_input_ids: typing.Optional[torch.Tensor] = None decoder_attention_mask: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None retur


Blip2Model

n_dict: typing.Optional[bool] = None  ) → text_outputs (CausalLMOutputWithPast, or tuple(torch.FloatTensor) if return_dict=False)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it. Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details. What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?
   decoder_input_ids (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Indices of decoder input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedToken


Blip2Model

izer.call() for details.
What are decoder input IDs?
T5 uses the pad_token_id as the starting token for decoder_input_ids generation. If past_key_values
is used, optionally only the last decoder_input_ids have to be input (see past_key_values).
To know more on how to prepare decoder_input_ids for pretraining take a look at T5
Training.   decoder_attention_mask (torch.BoolTensor of shape (batch_size, target_sequence_length), optional) —
Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also
be used by default.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.


Blip2Model

  Returns
text_outputs (CausalLMOutputWithPast, or tuple(torch.FloatTensor) if return_dict=False)

The language model outputs. If return_dict=True, the output is a CausalLMOutputWithPast that
contains the language model logits, the past key values and the hidden states if
output_hidden_states=True.
  The Blip2Model forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> import torch
>>> from transformers import AutoTokenizer, Blip2Model
>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16)
>>> model.to(device)
>>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/blip2-opt-2.7b")
>>> inputs = t


Blip2Model

okenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt").to(device)
>>> text_features = model.get_text_features(**inputs)  get_image_features  < source > ( pixel_values: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → vision_outputs (BaseModelOutputWithPooling or tuple of torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using Blip2Processor. See Blip2Processor.__call__() for
details.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more d


Blip2Model

etail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
vision_outputs (BaseModelOutputWithPooling or tuple of torch.FloatTensor)

The vision model outputs. If return_dict=True, the output is a BaseModelOutputWithPooling that
contains the image features, the pooled image features and the hidden states if
output_hidden_states=True.
  The Blip2Model forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> import torch
>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, Blip2Model
>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b",


Blip2Model

 torch_dtype=torch.float16)
>>> model.to(device)
>>> processor = AutoProcessor.from_pretrained("Salesforce/blip2-opt-2.7b")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt").to(device, torch.float16)
>>> image_outputs = model.get_image_features(**inputs)  get_qformer_features  < source > ( pixel_values: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → vision_outputs (BaseModelOutputWithPooling or tuple of torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using Blip2Processor. See Blip2Processor.__call__() for
details.   input_ids (torch.LongTensor of shape (batch_size, sequence_length), optional


Blip2Model

) —
Indices of input sequence tokens in the vocabulary of the language model. Input tokens can optionally be
provided to serve as text prompt, which the language model can continue.
Indices can be obtained using Blip2Processor. See Blip2Processor.__call__() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   decoder_input_ids (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Indices of decoder input sequence tokens in the vocabulary of the language model. Only relevant in case an
encoder-decoder language model (like T5) is used.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details. What are decoder input IDs?   decoder_attention_mask (


Blip2Model

torch.BoolTensor of shape (batch_size, target_sequence_length), optional) —
Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also
be used by default.
Only relevant in case an encoder-decoder language model (like T5) is used.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
vision_outputs (BaseModelOutputWithPooling or tuple of torch.FloatTensor)

The vision model outputs. If return_dict=True, the output is a BaseModelOutputWithPooling that
contains the image features, the pooled image features and the hidden states if
output_hidden_states=True.


Blip2Model

  The Blip2Model forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> import torch
>>> from PIL import Image
>>> import requests
>>> from transformers import Blip2Processor, Blip2Model
>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
>>> model = Blip2Model.from_pretrained("Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16)
>>> model.to(device)
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt").to(device, torch.float16)
>>> qformer_outputs = model.get_qformer_features(**inputs)


class transformers.Blip2ForConditionalGeneration  < source > ( config: Blip2Config  )   Parameters   config (Blip2Config) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    BLIP-2 Model for generating text given an image and an optional text prompt. The model consists of a vision
encoder, Querying Transformer (Q-Former) and a language model. One can optionally pass input_ids to the model, which serve as a text prompt, to make the language model continue
the prompt. Otherwise, the language model starts generating text from the [BOS] (beginning-of-sequence) token. Note that Flan-T5 checkpoints cannot be cast to float16. They are pre-trained using bfloat16. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its


 model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( pixel_values: FloatTensor input_ids: FloatTensor attention_mask: typing.Optional[torch.LongTensor] = None decoder_input_ids: typing.Optional[torch.LongTensor] = None decoder_attention_mask: typing.Optional[torch.LongTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None labels: typing.Optional[torch.LongTensor] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.blip_2.modeling_blip_2.Blip2ForConditionalGenerationModelOutput or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using Blip2Pro


cessor. See Blip2Processor.__call__() for
details.   input_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of input sequence tokens in the vocabulary of the language model. Input tokens can optionally be
provided to serve as text prompt, which the language model can continue.
Indices can be obtained using Blip2Processor. See Blip2Processor.__call__() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   decoder_input_ids (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Indices of decoder input sequence tokens in the vocabulary of the language model. Only relevant in case an
encoder-decoder language model (like T5) is used.
Indices can be obtained using AutoTokenizer. See Pr


eTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details. What are decoder input IDs?   decoder_attention_mask (torch.BoolTensor of shape (batch_size, target_sequence_length), optional) —
Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also
be used by default.
Only relevant in case an encoder-decoder language model (like T5) is used.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.models.blip_2.modeling_blip_2.Blip2ForConditionalGenerationModelOutput or tuple(torch.FloatTensor)

A transformers.models.blip_2.modeling_blip


_2.Blip2ForConditionalGenerationModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.blip_2.configuration_blip_2.Blip2VisionConfig'>) and inputs.
loss (torch.FloatTensor, optional, returned when labels is provided, torch.FloatTensor of shape (1,)) — Language modeling loss from the language model.
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head of the language model.
vision_outputs (BaseModelOutputWithPooling) — Outputs of the vision encoder.
qformer_outputs (BaseModelOutputWithPoolingAndCrossAttentions) — Outputs of the Q-Former (Querying Transformer).
language_model_outputs (CausalLMOutputWithPast or Seq2SeqLMOutput) — Outputs of the language model.
  The Blip2ForConditionalGeneration forward method, overrides the __call__ special method. Although


the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them. Examples:  Image captioning (without providing a text prompt):   Copied >>> from PIL import Image
>>> import requests
>>> from transformers import Blip2Processor, Blip2ForConditionalGeneration
>>> import torch
>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
>>> model = Blip2ForConditionalGeneration.from_pretrained(
...     "Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16
... )
>>> model.to(device)
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt").to(device, torch.float16)
>>> generated_ids = model.generate(**inp


uts)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
>>> print(generated_text)
two cats laying on a couch  Visual question answering (prompt = question):   Copied >>> from PIL import Image
>>> import requests
>>> from transformers import Blip2Processor, Blip2ForConditionalGeneration
>>> import torch
>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
>>> model = Blip2ForConditionalGeneration.from_pretrained(
...     "Salesforce/blip2-opt-2.7b", load_in_8bit=True, device_map={"": 0}, torch_dtype=torch.float16
... )  # doctest: +IGNORE_RESULT
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> prompt = "Question: how many cats are there? Answer:"
>>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(device="cuda", dtype=torch.float16)
>>> generated_ids


= model.generate(**inputs)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
>>> print(generated_text)
two Note that int8 inference is also supported through bitsandbytes.
This greatly reduces the amount of memory used by the model while maintaining the same performance.    Copied >>> from PIL import Image
>>> import requests
>>> from transformers import Blip2Processor, Blip2ForConditionalGeneration
>>> import torch
>>> processor = Blip2Processor.from_pretrained("Salesforce/blip2-flan-t5-xl")
>>> model = Blip2ForConditionalGeneration.from_pretrained(
...     "Salesforce/blip2-flan-t5-xl", load_in_8bit=True, device_map={"": 0}, torch_dtype=torch.bfloat16
... )  # doctest: +IGNORE_RESULT
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> prompt = "Question: how many cats are there? Answer:"
>>> inputs = processor(images=image, text=prompt, return_tensors="


pt").to(device="cuda", dtype=torch.bfloat16)
>>> generated_ids = model.generate(**inputs)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
>>> print(generated_text)
two  generate  < source > ( pixel_values: FloatTensor input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.LongTensor] = None **generate_kwargs  ) → captions (list)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Input images to be processed.   input_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
The sequence used as a prompt for the generation.   attention_mask (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices   Returns
captions (list)

A list of strings of length batch_size * num_captions.
  Overrides generate function to be able to use the model as a conditional generato


Overview

The SwitchTransformers model was proposed in Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity by William Fedus, Barret Zoph, Noam Shazeer.
The Switch Transformer model uses a sparse T5 encoder-decoder architecture, where the MLP are replaced by a Mixture of Experts (MoE). A routing mechanism (top 1 in this case) associates each token to one of the expert, where each expert is a dense MLP. While switch transformers have a lot more weights than their equivalent dense models, the sparsity allows better scaling and better finetuning performance at scale.
During a forward pass, only a fraction of the weights are used. The routing mechanism allows the model to select relevant weights on the fly which increases the model capacity without increasing the number of o


Overview

perations.
The abstract from the paper is the following:
In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example. The result is a sparsely-activated model — with outrageous numbers of parameters — but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs and training instability — we address these with the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques help wrangle the instabilities and we show large sparse models may be trained, for the first time


Overview

, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the “Colossal Clean Crawled Corpus” and achieve a 4x speedup over the T5-XXL model.
Tips:
SwitchTransformers uses the T5Tokenizer, which can be loaded directly from each model’s repository. The released weights are pretrained on English Masked Language Modeling task, and should be finetuned.
This model was contributed by Younes Belkada and Arthur Zucker .
The original code can be found here.


SwitchTransformersConfig

class transformers.SwitchTransformersConfig  < source > ( vocab_size = 32128 d_model = 768 d_kv = 64 d_ff = 2048 expert_capacity = 64 num_layers = 12 num_sparse_encoder_layers = 3 num_decoder_layers = 12 num_sparse_decoder_layers = 3 num_heads = 12 num_experts = 8 router_bias = False router_jitter_noise = 0.01 router_dtype = 'float32' router_ignore_padding_tokens = False relative_attention_num_buckets = 32 relative_attention_max_distance = 128 dropout_rate = 0.1 layer_norm_epsilon = 1e-06 router_z_loss_coef = 0.001 router_aux_loss_coef = 0.001 initializer_factor = 1.0 dense_act_fn = 'relu' is_encoder_decoder = True add_router_probs = False use_cache = True pad_token_id = 0 eos_token_id = 1 **kwargs  )   Parameters   vocab_size (int, optional, defaults to 32128) —
Vocabulary size of the SwitchTransformers model. Defines the number of different tokens that can be
represented by the inputs_ids passed when calling SwitchTransformersModel.   d_mo


SwitchTransformersConfig

del (int, optional, defaults to 512) —
Size of the encoder layers and the pooler layer.   d_kv (int, optional, defaults to 64) —
Size of the key, query, value projections per attention head. d_kv has to be equal to d_model // num_heads.   d_ff (int, optional, defaults to 2048) —
Size of the intermediate feed forward layer in each SwitchTransformersBlock.   expert_capacity (int, optional, defaults to 64) —
Number of tokens that can be stored in each expert. If set to 1, the model will behave like a regular
Transformer.   num_layers (int, optional, defaults to 12) —
Number of dense hidden layers in the Transformer encoder layer.   num_sparse_encoder_layers (int, optional, defaults to 6) —
Number of sparse (MoE) dense hidden layers in the Transformer encoder layer.   num_decoder_layers (int, optional, defaults to 12) —
Number of hidden layers in the Transformer decoder. Will use the same value as num_layers if not set.   num_sparse_decoder_laye


SwitchTransformersConfig

rs (int, optional, defaults to 12) —
Number of sparse (MoE) dense hidden layers in the Transformer decoder layer.   num_heads (int, optional, defaults to 8) —
Number of attention heads for each attention layer in the Transformer encoder.   num_experts (int, optional, defaults to 8) —
Number of experts for each SwitchTransformer layer.   router_type (str, optional, defaults to "tokens_masked") —
Router type - choose between "tokens_masked", “tokens_scatter”and“experts_masked”`.   router_bias (bool, optional, defaults to True) —
Whether to add a bias to the router.   router_jitter_noise (float, optional, defaults to 0.1) —
Amount of noise to add to the router.   router_dtype (str, optional, default to "float32") —
The dtype used for the routers. It is preferable to keep the dtype to "float32" as specified in the
selective precision discussion in the paper.   router_ignore_padding_tokens (bool, optional, defaults to False) —
Whether to ignore p


SwitchTransformersConfig

adding tokens when routing.   relative_attention_num_buckets (int, optional, defaults to 32) —
The number of buckets to use for each attention layer.   relative_attention_max_distance (int, optional, defaults to 128) —
The maximum distance of the longer sequences for the bucket separation.   dropout_rate (float, optional, defaults to 0.1) —
The ratio for all dropout layers.   layer_norm_eps (float, optional, defaults to 1e-6) —
The epsilon used by the layer normalization layers.   router_z_loss_coef (float, optional, defaults to 0.001) —
The z loss factor for the total loss.   router_aux_loss_coef (float, optional, defaults to 0.001) —
The aux loss factor for the total loss.   initializer_factor (float, optional, defaults to 1) —
A factor for initializing all weight matrices (should be kept to 1, used internally for initialization
testing).   feed_forward_proj (string, optional, defaults to "relu") —
Type of feed forward layer to be used. Sh


SwitchTransformersConfig

ould be one of "relu" or "gated-gelu". SwitchTransformersv1.1
uses the "gated-gelu" feed forward projection. Original SwitchTransformers uses "relu".   add_router_probs (bool, optional, defaults to False) —
Whether to output router probabilities to compute router auxiliary loss.   use_cache (bool, optional, defaults to True) —
Whether or not the model should return the last key/values attentions (not used by all models).    This is the configuration class to store the configuration of a SwitchTransformersModel. It is used to
instantiate a SwitchTransformers model according to the specified arguments, defining the model architecture.
Instantiating a configuration with the defaults will yield a similar configuration to that of the
SwitchTransformers google/switch-base-8 architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more informatio


class transformers.SwitchTransformersTop1Router  < source > ( config: SwitchTransformersConfig  )    Router using tokens choose top-1 experts assignment. This router uses the same mechanism as in Switch Transformer (https://arxiv.org/abs/2101.03961) and V-MoE
(https://arxiv.org/abs/2106.05974): tokens choose their top experts. Items are sorted by router_probs and then
routed to their choice of expert until the expert’s expert_capacity is reached. There is no guarantee that each
token is processed by an expert, or that each expert receives at least one token.  _compute_router_probabilities  < source > ( hidden_states: Tensor  ) → router_probabilities (torch.Tensor)  Parameter


s   hidden_states (torch.Tensor) —
(batch_size, sequence_length, hidden_dim) from which router probabilities are computed.   Returns
router_probabilities (torch.Tensor)

Tensor of shape (batch_size, sequence_length, num_experts) corresponding to the probabilities for each
token and expert. Used for routing tokens to experts.
router_logits (torch.Tensor):
Logits tensor of shape (batch_size, sequence_length, num_experts) corresponding to raw router logits.
This is used later for computing router z-loss.
  Computes router probabilities from input hidden states.  forward  < source > ( hidden_states: Tensor  )   Parameters   hidden_states (torch.Tensor)  —
[num_groups, tokens_


per_group, hidden_dim] inputs to send to experts.    Generic forward function for every Router class. Each Router expects to have the same input hidden states
(hidden_states) corresponding to the hidden states for each token, the expert_capacity corresponding to the
number of tokens the Router will send to each expert, some Routers can send up to few tokens to each expert. Each Router works as the following: it expects the hidden states for each token, gets the router_probs and
router_logits from the router_weights. This will assign for each token, the raw probability to be assigned
to an expert. Then each Router class will have to define its own _compute_routing_instruction


SwitchTransformersSparseMLP

class transformers.SwitchTransformersSparseMLP  < source > ( config: SwitchTransformersConfig expert_class: Module = <class 'transformers.models.switch_transformers.modeling_switch_transformers.SwitchTransformersDenseActDense'>  )    Implementation of the Switch Transformers Sparse MLP module.  forward  < source > ( hidden_states  )    Hold on, this will be slightly tricky to understand In the correct order, a MoE layer does the following: 1- Gets the router_mask from the router. The shape of the mask is (batch_size, sequence_length, num_expert)
and corresponds to the argmax of the router_probs. The probabilities are needed in the computation of the
hidden states : they are broadcasted to the hidden states values (can be interpreted as a scaling factor). 2- Dispatch the tokens to its associated experts. We do a classic for loop over the experts and assign for each
expert the corresponding hidden states.


SwitchTransformersModel

class transformers.SwitchTransformersModel  < source > ( config: SwitchTransformersConfig  )   Parameters   config (SwitchTransformersConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare SWITCH_TRANSFORMERS Model transformer outputting raw hidden-states without any specific head on top. The SWITCH_TRANSFORMERS model was proposed in Switch Transformers: Scaling to Trillion Parameter Models with
Simple and Efficient Sparsity by William
Fedus, Barret
Zoph, and Noam
Shazeer. It’s an encoder-decoder T5-like model
with sparse Feed Forward that stands for Mixture of Experts (MoE) architecture. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the


SwitchTransformersModel

 input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None decoder_input_ids: typing.Optional[torch.LongTensor] = None decoder_attention_mask: typing.Optional[torch.BoolTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None decoder_head_mask: typing.Optional[torch.FloatTensor] = None cross_attn_head_mask: typing.Optional[torch.Tensor] = None encoder_outputs: typing.Optional[typing.Tuple[typing.Tuple[torch.FloatTensor]]] = None past_key_values: typing.Optional[typing.Tuple[typing.Tuple[torch.FloatTensor]]] = None inputs_embeds: typing.Optional[torch.Tensor] = None decoder_inputs_embeds: typing.Optional[torch.Tensor] = None use_cache: typing.Optional[bool] = None output_at


SwitchTransformersModel

tentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None output_router_logits: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.Seq2SeqMoEModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. SWITCH_TRANSFORMERS is a model with relative position
embeddings so you should be able to pad the inputs on both the right and the left.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for detail.
What are input IDs?
To know more on how to prepare input_ids for pretraining take a look a SWITCH_TRANSFORMERS
Training.   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are no


SwitchTransformersModel

t masked,
0 for tokens that are masked.
What are attention masks?   decoder_input_ids (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Indices of decoder input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are decoder input IDs?
SWITCH_TRANSFORMERS uses the pad_token_id as the starting token for decoder_input_ids generation. If
past_key_values is used, optionally only the last decoder_input_ids have to be input (see
past_key_values).
To know more on how to prepare decoder_input_ids for pretraining take a look at SWITCH_TRANSFORMERS
Training.   decoder_attention_mask (torch.BoolTensor of shape (batch_size, target_sequence_length), optional) —
Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also
be used by default.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, n


SwitchTransformersModel

um_heads), optional) —
Mask to nullify selected heads of the self-attention modules in the encoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   decoder_head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   cross_attn_head_mask (torch.Tensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in
[0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   encoder_outputs (tuple(tuple(torch.FloatTensor), optional) —
Tuple consists of (last_hidden_state, optional: hidden_states, optional: attentions)
last_hidden_state of shape (batch_size, sequence_length, hidden_size) is a sequenc


SwitchTransformersModel

e of hidden states at
the output of the last layer of the encoder. Used in the cross-attention of the decoder.   past_key_values (tuple(tuple(torch.FloatTensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)) —
Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s


SwitchTransformersModel

internal embedding lookup matrix.   decoder_inputs_embeds (torch.FloatTensor of shape (batch_size, target_sequence_length, hidden_size), optional) —
Optionally, instead of passing decoder_input_ids you can choose to directly pass an embedded
representation. If past_key_values is used, optionally only the last decoder_inputs_embeds have to be
input (see past_key_values). This is useful if you want more control over how to convert
decoder_input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
If decoder_input_ids and decoder_inputs_embeds are both unset, decoder_inputs_embeds takes the value
of inputs_embeds.   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden


SwitchTransformersModel

_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   output_router_logits (bool, optional) —
Whether or not to return the logits of all the routers. They are useful for computing the router loss, and
should not be returned during inference.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.Seq2SeqMoEModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.Seq2SeqMoEModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (SwitchTransformersConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the decoder of the model.
If past_key_values is used only th


SwitchTransformersModel

e last hidden-state of the sequences of shape (batch_size, 1, hidden_size) is output.
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (bat


SwitchTransformersModel

ch_size, sequence_length, hidden_size).
Hidden-states of the decoder at the output of each layer plus the optional initial embedding outputs.
decoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
decoder_router_logits (tuple(torch.FloatTensor), optional, returned when output_router_logits=True is passed or when config.add_router_probs=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, sequence_length, num_experts).
Router logits of the decoder model, useful to compute the auxiliary loss for Mixture of Experts models.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed o


SwitchTransformersModel

r when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the optional initial embedding outputs.
encoder_attentions (tuple(torch.F


SwitchTransformersModel

loatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
encoder_router_logits (tuple(torch.FloatTensor), optional, returned when output_router_logits=True is passed or when config.add_router_probs=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, sequence_length, num_experts).
Router logits of the encoder model, useful to compute the auxiliary loss and the z_loss for the sparse
modules.
  The SwitchTransformersModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the


SwitchTransformersModel

 pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, SwitchTransformersModel
>>> tokenizer = AutoTokenizer.from_pretrained("google/switch-base-8")
>>> model = SwitchTransformersModel.from_pretrained("google/switch-base-8")
>>> input_ids = tokenizer(
...     "Studies have been shown that owning a dog is good for you", return_tensors="pt"
... ).input_ids  # Batch size 1
>>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids  # Batch size 1
>>> # preprocess: Prepend decoder_input_ids with start token which is pad token for SwitchTransformersModel.
>>> # This is not needed for torch's SwitchTransformersForConditionalGeneration as it does this internally using labels arg.
>>> decoder_input_ids = model._shift_right(decoder_input_ids)
>>> # forward pass
>>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids)
>>> last_hidden_states = outputs.last_hid


class transformers.SwitchTransformersForConditionalGeneration  < source > ( config: SwitchTransformersConfig  )   Parameters   config (SwitchTransformersConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    SWITCH_TRANSFORMERS Model with a language modeling head on top. The SWITCH_TRANSFORMERS model was proposed in Switch Transformers: Scaling to Trillion Parameter Models with
Simple and Efficient Sparsity by William
Fedus, Barret
Zoph, and Noam
Shazeer. It’s an encoder-decoder T5-like model
with sparse Feed Forward that stands for Mixture of Experts (MoE) architecture. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or


saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None decoder_input_ids: typing.Optional[torch.LongTensor] = None decoder_attention_mask: typing.Optional[torch.BoolTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None decoder_head_mask: typing.Optional[torch.FloatTensor] = None cross_attn_head_mask: typing.Optional[torch.Tensor] = None encoder_outputs: typing.Optional[typing.Tuple[typing.Tuple[torch.Tensor]]] = None past_key_values: typing.Optional[typing.Tuple[typing.Tuple[torch.Tensor]]] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None decoder_inputs_embeds: typing.Optional[torch.Fl


oatTensor] = None labels: typing.Optional[torch.LongTensor] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None output_router_logits: typing.Optional[bool] = True return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.Seq2SeqMoEOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. SWITCH_TRANSFORMERS is a model with relative position
embeddings so you should be able to pad the inputs on both the right and the left.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for detail.
What are input IDs?
To know more on how to prepare input_ids for pretraining take a look a SWITCH_TRANSFORMERS
Training.   attention_mask (torch.FloatTensor of shape (b


atch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   decoder_input_ids (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Indices of decoder input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are decoder input IDs?
SWITCH_TRANSFORMERS uses the pad_token_id as the starting token for decoder_input_ids generation. If
past_key_values is used, optionally only the last decoder_input_ids have to be input (see
past_key_values).
To know more on how to prepare decoder_input_ids for pretraining take a look at SWITCH_TRANSFORMERS
Training.   decoder_attention_mask (torch.BoolTensor of shape (batch_size, target_sequence_length), o


ptional) —
Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also
be used by default.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules in the encoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   decoder_head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   cross_attn_head_mask (torch.Tensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in
[0, 1]:
1 indicates the head is not masked,
0 indicates


 the head is masked.
   encoder_outputs (tuple(tuple(torch.FloatTensor), optional) —
Tuple consists of (last_hidden_state, optional: hidden_states, optional: attentions)
last_hidden_state of shape (batch_size, sequence_length, hidden_size) is a sequence of hidden states at
the output of the last layer of the encoder. Used in the cross-attention of the decoder.   past_key_values (tuple(tuple(torch.FloatTensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)) —
Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).   inputs_embeds (torc


h.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   decoder_inputs_embeds (torch.FloatTensor of shape (batch_size, target_sequence_length, hidden_size), optional) —
Optionally, instead of passing decoder_input_ids you can choose to directly pass an embedded
representation. If past_key_values is used, optionally only the last decoder_inputs_embeds have to be
input (see past_key_values). This is useful if you want more control over how to convert
decoder_input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
If decoder_input_ids and decoder_inputs_embeds are both unset, decoder_inputs_embeds takes the value
of inpu


ts_embeds.   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   output_router_logits (bool, optional) —
Whether or not to return the logits of all the routers. They are useful for computing the router loss, and
should not be returned during inference.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the sequence classification/regression loss. Indices should be in [-100, 0, ...,


 config.vocab_size - 1]. All labels set to -100 are ignored (masked), the loss is only computed for
labels in [0, ..., config.vocab_size]   Returns
transformers.modeling_outputs.Seq2SeqMoEOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.Seq2SeqMoEOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (SwitchTransformersConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Language modeling loss.
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of lengt


h config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the decoder at the output of each layer plus the initial embedding outputs.
decoder_attentions (tuple(torch.FloatTensor), optional, returned when output_att


entions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
decoder_router_logits (tuple(torch.FloatTensor), optional, returned when output_router_logits=True is passed or when config.add_router_probs=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, sequence_length, num_experts).
Router logits of the decoder model, useful to compute the auxiliary loss for Mixture of Experts models.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights


of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the initial embedding outputs.
encoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor


 (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
encoder_router_logits (tuple(torch.FloatTensor), optional, returned when output_router_logits=True is passed or when config.add_router_probs=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, sequence_length, num_experts).
Router logits of the encoder model, useful to compute the auxiliary loss and z_loss for Mixture of Experts
models.
  The SwitchTransformersForConditionalGeneration forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores the


m.  Examples:   Copied >>> from transformers import AutoTokenizer, SwitchTransformersForConditionalGeneration
>>> tokenizer = AutoTokenizer.from_pretrained("google/switch-base-8")
>>> model = SwitchTransformersForConditionalGeneration.from_pretrained("google/switch-base-8")
>>> # training
>>> input_ids = tokenizer("The <extra_id_0> walks in <extra_id_1> park", return_tensors="pt").input_ids
>>> labels = tokenizer("<extra_id_0> cute dog <extra_id_1> the <extra_id_2>", return_tensors="pt").input_ids
>>> outputs = model(input_ids=input_ids, labels=labels)
>>> loss = outputs.loss
>>> logits = outputs.logits
>>> # inference
>>> input_ids = tokenizer(
...     "summarize: studies have shown that owning a dog is good for you", return_tensors="pt"
... ).input_ids  # Batch size 1
>>> outputs = model.generate(input_ids)
>>> # . To, let’s say you have a dog. To summarize:
>>> # Since the model has been trained on MLM, this will output gibb


SwitchTransformersEncoderModel

class transformers.SwitchTransformersEncoderModel  < source > ( config: SwitchTransformersConfig  )   Parameters   config (SwitchTransformersConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare SWITCH_TRANSFORMERS Model transformer outputting encoder’s raw hidden-states without any specific head on top. The SWITCH_TRANSFORMERS model was proposed in Switch Transformers: Scaling to Trillion Parameter Models with
Simple and Efficient Sparsity by William
Fedus, Barret
Zoph, and Noam
Shazeer. It’s an encoder-decoder T5-like model
with sparse Feed Forward that stands for Mixture of Experts (MoE) architecture. This model inherits from PreTrainedModel. Check the superclass documentation fo


SwitchTransformersEncoderModel

r the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None output_router_logits: typing.Optional[bool] = True return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.MoEModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batc


SwitchTransformersEncoderModel

h_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. SWITCH_TRANSFORMERS is a model with relative position
embeddings so you should be able to pad the inputs on both the right and the left.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for detail.
To know more on how to prepare input_ids for pretraining take a look a SWITCH_TRANSFORMERS
Training.   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1


SwitchTransformersEncoderModel

 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   output_router_logits (bool, optional) —
Whether or not to return the logits of all the routers. They are useful for computing the router loss, and
sh


SwitchTransformersEncoderModel

ould not be returned during inference.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.MoEModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.MoEModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (SwitchTransformersConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the outpu


SwitchTransformersEncoderModel

t of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
router_probs (tuple(torch.FloatTensor), optional, returned when output_router_probs=True and config.add_router_probs=True is passed or when config.output_router_probs=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, sequence_length, num_experts).
Raw router probabilities that are computed by MoE routers, these terms are used to com


SwitchTransformersEncoderModel

pute the auxiliary
loss and the z_loss for Mixture of Experts models.
  The SwitchTransformersEncoderModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, SwitchTransformersEncoderModel
>>> tokenizer = AutoTokenizer.from_pretrained("google/switch-base-8")
>>> model = SwitchTransformersEncoderModel.from_pretrained("google/switch-base-8")
>>> input_ids = tokenizer(
...     "Studies have been shown that owning a dog is good for you", return_tensors="pt"
... ).input_ids  # Batch size 1
>>> outputs = model(input_ids=input_ids)
>>> last_hidden_states = outputs.last_hidden_


Overview

The GroupViT model was proposed in GroupViT: Semantic Segmentation Emerges from Text Supervision by Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, Xiaolong Wang.
Inspired by CLIP, GroupViT is a vision-language model that can perform zero-shot semantic segmentation on any given vocabulary categories.
The abstract from the paper is the following:
Grouping and recognition are important components of visual scene understanding, e.g., for object detection and semantic segmentation. With end-to-end deep learning systems, grouping of image regions usually happens implicitly via top-down supervision from pixel-level recognition labels. Instead, in this paper, we propose to bring back the grouping mechanism into deep networks, which allows semantic segments to emerge automatically with only text supervision. We propose a hierarchical Grouping Vision Transformer (GroupViT), which goes beyond the regular grid structure representation and l


Overview

earns to group image regions into progressively larger arbitrary-shaped segments. We train GroupViT jointly with a text encoder on a large-scale image-text dataset via contrastive losses. With only text supervision and without any pixel-level annotations, GroupViT learns to group together semantic regions and successfully transfers to the task of semantic segmentation in a zero-shot manner, i.e., without any further fine-tuning. It achieves a zero-shot accuracy of 52.3% mIoU on the PASCAL VOC 2012 and 22.4% mIoU on PASCAL Context datasets, and performs competitively to state-of-the-art transfer-learning methods requiring greater levels of supervision.
Tips:
You may specify output_segmentation=True in the forward of GroupViTModel to get the segmentation logits of input texts.
This model was contributed by xvjiarui. The TensorFlow version was contributed by ariG23498 with the help of Yih-Dar SHIEH, Amy Roberts, and Joao Gante.
The original code can be found here.


Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with GroupViT.
The quickest way to get started with GroupViT is by checking the example notebooks (which showcase zero-shot segmentation inference). One can also check out the HuggingFace Spaces demo to play with GroupViT.


GroupViTConfig

class transformers.GroupViTConfig  < source > ( text_config = None vision_config = None projection_dim = 256 projection_intermediate_dim = 4096 logit_scale_init_value = 2.6592 **kwargs  )   Parameters   text_config (dict, optional) —
Dictionary of configuration options used to initialize GroupViTTextConfig.   vision_config (dict, optional) —
Dictionary of configuration options used to initialize GroupViTVisionConfig.   projection_dim (int, optional, defaults to 256) —
Dimentionality of text and vision projection layers.   projection_intermediate_dim (int, optional, defaults to 4096) —
Dimentionality of intermediate layer of text and vision projection layers.   logit_scale_init_value (float, optional, defaults to 2.6592) —
The inital value of the logit_scale parameter. Default is used as per the original GroupViT
implementation.   kwargs (optional) —
Dictionary of keywo


GroupViTConfig

rd arguments.    GroupViTConfig is the configuration class to store the configuration of a GroupViTModel. It is used to
instantiate a GroupViT model according to the specified arguments, defining the text model and vision model
configs. Instantiating a configuration with the defaults will yield a similar configuration to that of the GroupViT
nvidia/groupvit-gcc-yfcc architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  from_text_vision_configs  < source > ( text_config: GroupViTTextConfig vision_config: GroupViTVisionConfig **kwargs  ) → GroupViTConfig  Returns
GroupViTConfig

An instance of a configuration object
  Instantiate a GroupViTConfig (or a derived class) from groupvit text model configuration and groupvit
vision model configuration


GroupViTTextConfig

class transformers.GroupViTTextConfig  < source > ( vocab_size = 49408 hidden_size = 256 intermediate_size = 1024 num_hidden_layers = 12 num_attention_heads = 4 max_position_embeddings = 77 hidden_act = 'quick_gelu' layer_norm_eps = 1e-05 dropout = 0.0 attention_dropout = 0.0 initializer_range = 0.02 initializer_factor = 1.0 pad_token_id = 1 bos_token_id = 49406 eos_token_id = 49407 **kwargs  )   Parameters   vocab_size (int, optional, defaults to 49408) —
Vocabulary size of the GroupViT text model. Defines the number of different tokens that can be represented
by the inputs_ids passed when calling GroupViTModel.   hidden_size (int, optional, defaults to 256) —
Dimensionality of the encoder layers and the pooler layer.   intermediate_size (int, optional, defaul


GroupViTTextConfig

ts to 1024) —
Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.   num_hidden_layers (int, optional, defaults to 12) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 4) —
Number of attention heads for each attention layer in the Transformer encoder.   max_position_embeddings (int, optional, defaults to 77) —
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).   hidden_act (str or function, optional, defaults to "quick_gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "selu" and "gelu_new" `"quick_gelu" a


GroupViTTextConfig

re supported.   layer_norm_eps (float, optional, defaults to 1e-5) —
The epsilon used by the layer normalization layers.   attention_dropout (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.   dropout (float, optional, defaults to 0.0) —
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   initializer_factor (float, optional, defaults to 1.0) —
A factor for initializing all weight matrices (should be kept to 1, used internally for initialization
testing).    This is the configuration class to store the configuration of a GroupVi


GroupViTTextConfig

TTextModel. It is used to instantiate an
GroupViT model according to the specified arguments, defining the model architecture. Instantiating a configuration
with the defaults will yield a similar configuration to that of the GroupViT
nvidia/groupvit-gcc-yfcc architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import GroupViTTextConfig, GroupViTTextModel
>>> # Initializing a GroupViTTextModel with nvidia/groupvit-gcc-yfcc style configuration
>>> configuration = GroupViTTextConfig()
>>> model = GroupViTTextModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config


GroupViTVisionConfig

class transformers.GroupViTVisionConfig  < source > ( hidden_size = 384 intermediate_size = 1536 depths = [6, 3, 3] num_hidden_layers = 12 num_group_tokens = [64, 8, 0] num_output_groups = [64, 8, 8] num_attention_heads = 6 image_size = 224 patch_size = 16 num_channels = 3 hidden_act = 'gelu' layer_norm_eps = 1e-05 dropout = 0.0 attention_dropout = 0.0 initializer_range = 0.02 initializer_factor = 1.0 assign_eps = 1.0 assign_mlp_ratio = [0.5, 4] **kwargs  )   Parameters   hidden_size (int, optional, defaults to 384) —
Dimensionality of the encoder layers and the pooler layer.   intermediate_size (int, optional, defaults to 1536) —
Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.   depths (List[int], optional, defaults to [6, 3, 3])


GroupViTVisionConfig

 —
The number of layers in each encoder block.   num_group_tokens (List[int], optional, defaults to [64, 8, 0]) —
The number of group tokens for each stage.   num_output_groups (List[int], optional, defaults to [64, 8, 8]) —
The number of output groups for each stage, 0 means no group.   num_attention_heads (int, optional, defaults to 6) —
Number of attention heads for each attention layer in the Transformer encoder.   image_size (int, optional, defaults to 224) —
The size (resolution) of each image.   patch_size (int, optional, defaults to 16) —
The size (resolution) of each patch.   hidden_act (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "selu" and "gelu_new"


GroupViTVisionConfig

 `"quick_gelu" are supported.   layer_norm_eps (float, optional, defaults to 1e-5) —
The epsilon used by the layer normalization layers.   dropout (float, optional, defaults to 0.0) —
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.   attention_dropout (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   initializer_factor (float, optional, defaults to 1.0) —
A factor for initializing all weight matrices (should be kept to 1, used internally for initialization
testing).    This is the configuration class to store the configuration of a Gro


GroupViTVisionConfig

upViTVisionModel. It is used to instantiate
an GroupViT model according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the GroupViT
nvidia/groupvit-gcc-yfcc architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import GroupViTVisionConfig, GroupViTVisionModel
>>> # Initializing a GroupViTVisionModel with nvidia/groupvit-gcc-yfcc style configuration
>>> configuration = GroupViTVisionConfig()
>>> model = GroupViTVisionModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.con


GroupViTModel

class transformers.GroupViTModel  < source > ( config: GroupViTConfig  )   Parameters   config (GroupViTConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    This model is a PyTorch torch.nn.Module subclass. Use it
as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None pixel_values: typing.Optional[torch.FloatTensor] = None attention_mask: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.LongTensor] = None return_loss: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None output_segmentation


GroupViTModel

: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.groupvit.modeling_groupvit.GroupViTModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using CLIPTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embedd


GroupViTModel

ings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
CLIPImageProcessor.call() for details.   return_loss (bool, optional) —
Whether or not to return the contrastive loss.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.models.groupvit.modeling_groupvit.GroupViTModelOutput or tuple(torch.FloatTensor)

A transformers.models.groupvi


GroupViTModel

t.modeling_groupvit.GroupViTModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.groupvit.configuration_groupvit.GroupViTConfig'>) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when return_loss is True) — Contrastive loss for image-text similarity.
logits_per_image (torch.FloatTensor of shape (image_batch_size, text_batch_size)) — The scaled dot product scores between image_embeds and text_embeds. This represents the image-text
similarity scores.
logits_per_text (torch.FloatTensor of shape (text_batch_size, image_batch_size)) — The scaled dot product scores between text_embeds and image_embeds. This represents the text-image
similarity scores.
segmentation_logits (torch.FloatTensor of shape (batch_size, config.num_labels, logits_height, logits_width)) —


GroupViTModel

Classification scores for each pixel.
The logits returned do not necessarily have the same size as the pixel_values passed as inputs. This is
to avoid doing two interpolations and lose some quality when a user needs to resize the logits to the
original image size as post-processing. You should always check your logits shape and resize as needed.
text_embeds (torch.FloatTensor of shape (batch_size, output_dim) — The text embeddings obtained by applying the projection layer to the pooled output of
GroupViTTextModel.
image_embeds (torch.FloatTensor of shape (batch_size, output_dim) — The image embeddings obtained by applying the projection layer to the pooled output of
GroupViTVisionModel.
text_model_output (BaseModelOutputWithPooling) — The output of the GroupViTTextModel.
vision_model_output (BaseModelOutputWithPooling) — The output of the GroupViTVisionModel.
  The GroupViTModel forward method, overrides the __call__ sp


GroupViTModel

ecial method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, GroupViTModel
>>> model = GroupViTModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> processor = AutoProcessor.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(
...     text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True
... )
>>> outputs = model(**inputs)
>>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
>>> probs = logi


GroupViTModel

ts_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities  get_text_features  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → text_features (torch.FloatTensor of shape (batch_size, output_dim)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using CLIPTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing atten


GroupViTModel

tion on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
text_features (torch.FloatTensor of shape (batch_size, output_dim)

The text embeddings obtain


GroupViTModel

ed by
applying the projection layer to the pooled output of GroupViTTextModel.
  The GroupViTModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import CLIPTokenizer, GroupViTModel
>>> model = GroupViTModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> tokenizer = CLIPTokenizer.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
>>> text_features = model.get_text_features(**inputs)  get_image_features  < source > ( pixel_values: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hid


GroupViTModel

den_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → image_features (torch.FloatTensor of shape (batch_size, output_dim)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using
AutoImageProcessor. See CLIPImageProcessor.call() for details.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
image_features (torch.FloatTensor of shape (batch_size, output_dim)

Th


GroupViTModel

e image embeddings obtained by
applying the projection layer to the pooled output of GroupViTVisionModel.
  The GroupViTModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, GroupViTModel
>>> model = GroupViTModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> processor = AutoProcessor.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt")
>>> image_features = model.get_image_features


GroupViTTextModel

class transformers.GroupViTTextModel  < source > ( config: GroupViTTextConfig  )     forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.BaseModelOutputWithPooling or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using CLIPTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_le


GroupViTTextModel

ngth), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
tran


GroupViTTextModel

sformers.modeling_outputs.BaseModelOutputWithPooling or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutputWithPooling or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.groupvit.configuration_groupvit.GroupViTTextConfig'>) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing
through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns
the classification token after processing throug


GroupViTTextModel

h a linear layer and a tanh activation function. The linear
layer weights are trained from the next sentence prediction (classification) objective during pretraining.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the atten


GroupViTTextModel

tion softmax, used to compute the weighted average in the self-attention
heads.
  The GroupViTTextModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import CLIPTokenizer, GroupViTTextModel
>>> tokenizer = CLIPTokenizer.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> model = GroupViTTextModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_state = outputs.last_hidden_state
>>> pooled_output = outputs.pooler_output  # pooled (EOS token) st


GroupViTVisionModel

class transformers.GroupViTVisionModel  < source > ( config: GroupViTVisionConfig  )     forward  < source > ( pixel_values: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.BaseModelOutputWithPooling or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using
AutoImageProcessor. See CLIPImageProcessor.call() for details.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors f


GroupViTVisionModel

or
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.BaseModelOutputWithPooling or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutputWithPooling or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.groupvit.configuration_groupvit.GroupViTVisionConfig'>) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing
through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns
the


GroupViTVisionModel

classification token after processing through a linear layer and a tanh activation function. The linear
layer weights are trained from the next sentence prediction (classification) objective during pretraining.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted


GroupViTVisionModel

average in the self-attention
heads.
  The GroupViTVisionModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, GroupViTVisionModel
>>> processor = AutoProcessor.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> model = GroupViTVisionModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_state = outputs.last_hidden_state
>>> pooled_output = outputs.pooler_output  # pooled CLS stat


TFGroupViTModel

class transformers.TFGroupViTModel  < source > ( *args **kwargs  )   Parameters   config (GroupViTConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior. TF 2.0 models accepts two formats as inputs: having all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argu


TFGroupViTModel

ments. This second option is useful when using tf.keras.Model.fit method which currently requires having all the
tensors in the first argument of the model call function: model(inputs). If you choose this second option, there are three possibilities you can use to gather all the input Tensors in the
first positional argument : a single Tensor with input_ids only and nothing else: model(input_ids) a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([input_ids, attention_mask]) or model([input_ids, attention_mask, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"input_ids": input_ids, "token_type_ids": token_type_ids})  call  < source > ( input_ids: TFModelInputType | None = None pixel_values: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None position_ids: np.ndarra


TFGroupViTModel

y | tf.Tensor | None = None return_loss: Optional[bool] = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None output_segmentation: Optional[bool] = None return_dict: Optional[bool] = None training: bool = False  ) → transformers.models.groupvit.modeling_tf_groupvit.TFGroupViTModelOutput or tuple(tf.Tensor)  Parameters   input_ids (np.ndarray, tf.Tensor, List[tf.Tensor] `Dict[str, tf.Tensor] or Dict[str, np.ndarray] and each example must have the shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and
PreTrainedTokenizer.encode() for details.
What are input IDs?   pixel_values (np.ndarray, tf.Tensor, List[tf.Tensor] Dict[str, tf.Tensor] or Dict[str, np.ndarray] and each example must have the shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can