torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), o


ptional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors


 for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   start_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the start of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
are not taken into account for computing the loss.   end_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the end of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
are not taken into account for computing the los


s.   Returns
transformers.modeling_outputs.QuestionAnsweringModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.QuestionAnsweringModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (Data2VecTextConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Total span extraction loss is the sum of a Cross-Entropy for the start and end positions.
start_logits (torch.FloatTensor of shape (batch_size, sequence_length)) — Span-start scores (before SoftMax).
end_logits (torch.FloatTensor of shape (batch_size, sequence_length)) — Span-end scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the outp


ut of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The Data2VecTextForQuestionAnswering forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the la


tter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, Data2VecTextForQuestionAnswering
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/data2vec-text-base")
>>> model = Data2VecTextForQuestionAnswering.from_pretrained("facebook/data2vec-text-base")
>>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
>>> inputs = tokenizer(question, text, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> answer_start_index = outputs.start_logits.argmax()
>>> answer_end_index = outputs.end_logits.argmax()
>>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
>>> # target is "nice puppet"
>>> target_start_index = torch.tensor([14])
>>> target_end_index = torch.tensor([15])
>>> outputs = model(**inputs, start_positions=target_start_index, end_positions=target_end_index)
>>> loss = outputs.


Data2VecVisionModel

class transformers.Data2VecVisionModel  < source > ( config: Data2VecVisionConfig add_pooling_layer: bool = False  )   Parameters   config (Data2VecVisionConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare Data2VecVision Model transformer outputting raw hidden-states without any specific head on top.
This model is a PyTorch torch.nn.Module subclass. Use it
as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: typing.Optional[torch.Tensor] = None bool_masked_pos: typing.Optional[torch.BoolTensor] = None head_mask: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[


Data2VecVisionModel

bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.data2vec.modeling_data2vec_vision.Data2VecVisionModelOutputWithPooling or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
BeitImageProcessor.call() for details.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors


Data2VecVisionModel

for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   bool_masked_pos (torch.BoolTensor of shape (batch_size, num_patches), optional) —
Boolean masked positions. Indicates which patches are masked (1) and which aren’t (0).   Returns
transformers.models.data2vec.modeling_data2vec_vision.Data2VecVisionModelOutputWithPooling or tuple(torch.FloatTensor)

A transformers.models.data2vec.modeling_data2vec_vision.Data2VecVisionModelOutputWithPooling or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (Data2VecVisionConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — Average of the l


Data2VecVisionModel

ast layer hidden states of the patch tokens (excluding the [CLS] token) if
config.use_mean_pooling is set to True. If set to False, then the final hidden state of the [CLS] token
will be returned.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of
shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.


Data2VecVisionModel

The Data2VecVisionModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoImageProcessor, Data2VecVisionModel
>>> import torch
>>> from datasets import load_dataset
>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]
>>> image_processor = AutoImageProcessor.from_pretrained("facebook/data2vec-vision-base")
>>> model = Data2VecVisionModel.from_pretrained("facebook/data2vec-vision-base")
>>> inputs = image_processor(image, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1, 197, 76


class transformers.Data2VecVisionForImageClassification  < source > ( config: Data2VecVisionConfig  )   Parameters   config (Data2VecVisionConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Data2VecVision Model transformer with an image classification head on top (a linear layer on top of the average of
the final hidden states of the patch tokens) e.g. for ImageNet. This model is a PyTorch torch.nn.Module subclass. Use it
as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden


_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.ImageClassifierOutput or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
BeitImageProcessor.call() for details.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (boo


l, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the image classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-Entropy).   Returns
transformers.modeling_outputs.ImageClassifierOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.ImageClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (Data2VecVisionConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, config.num_labels)) — Classifi


cation (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each stage) of shape (batch_size, sequence_length, hidden_size). Hidden-states
(also called feature maps) of the model at the output of each stage.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, patch_size, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The Data2VecVisionForImageClassification forward method, overrides the __call__ special method. Although the recipe for f


orward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoImageProcessor, Data2VecVisionForImageClassification
>>> import torch
>>> from datasets import load_dataset
>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]
>>> image_processor = AutoImageProcessor.from_pretrained("facebook/data2vec-vision-base-ft1k")
>>> model = Data2VecVisionForImageClassification.from_pretrained("facebook/data2vec-vision-base-ft1k")
>>> inputs = image_processor(image, return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> # model predicts one of the 1000 ImageNet classes
>>> predicted_label = logits.argmax(-1).item()
>>> print(model.config.id2label[predicted_label])
remote control, remot


class transformers.Data2VecVisionForSemanticSegmentation  < source > ( config: Data2VecVisionConfig  )   Parameters   config (Data2VecVisionConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Data2VecVision Model transformer with a semantic segmentation head on top e.g. for ADE20k, CityScapes. This model is a PyTorch torch.nn.Module subclass. Use it
as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transfor


mers.modeling_outputs.SemanticSegmenterOutput or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
BeitImageProcessor.call() for details.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of


 shape (batch_size, height, width), optional) —
Ground truth semantic segmentation maps for computing the loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels > 1, a classification loss is computed (Cross-Entropy).   Returns
transformers.modeling_outputs.SemanticSegmenterOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.SemanticSegmenterOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (Data2VecVisionConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, config.num_labels, logits_height, logits_width)) — Classification scores for each pixel.
The logits returned do not necessarily have the same size as the pixel_values passed as inputs. This is
to avoid doing two i


nterpolations and lose some quality when a user needs to resize the logits to the
original image size as post-processing. You should always check your logits shape and resize as needed.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, patch_size, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, patch_size, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The Data2VecV


isionForSemanticSegmentation forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoImageProcessor, Data2VecVisionForSemanticSegmentation
>>> from PIL import Image
>>> import requests
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> image_processor = AutoImageProcessor.from_pretrained("facebook/data2vec-vision-base")
>>> model = Data2VecVisionForSemanticSegmentation.from_pretrained("facebook/data2vec-vision-base")
>>> inputs = image_processor(images=image, return_tensors="pt")
>>> outputs = model(**inputs)
>>> # logits are of shape (batch_size, num_labels, height, width)
>>> logits = outputs.logits


class transformers.TFData2VecVisionModel  < source > ( *args **kwargs  )   Parameters   config (Data2VecVisionConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare Data2VecVision Model transformer outputting raw hidden-states without any specific head on top.
This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.). This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior. TensorFlow models and layers in transformers accept two formats as input: ha


ving all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with pixel_values only and nothing else: model(pixel_values) a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([pixel_values, attention_mask]) or model([pixel_


values, attention_mask, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"pixel_values": pixel_values, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( pixel_values: TFModelInputType | None = None bool_masked_pos: tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None training: bool = False  ) → transformers.models.data2vec.modeling_tf_data2vec_vision.TFData2VecVisionModelOutputWithPooling or tuple(tf.Tensor)  Parameters   pixel_values (np.ndarray, tf.Tensor, List[tf.Tensor] Dict[str, tf.Tensor] or Dict[str, np.ndarray] and each example must h


ave the shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
BeitImageProcessor.call() for details.   head_mask (np.ndarray or tf.Tensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used
in eager mode, in graph mode the value will always be set to True.


   training (bool, optional, defaults to `False“) —
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).   bool_masked_pos (tf.Tensor of shape (batch_size, num_patches), optional) —
Boolean masked positions. Indicates which patches are masked (1) and which aren’t (0).   Returns
transformers.models.data2vec.modeling_tf_data2vec_vision.TFData2VecVisionModelOutputWithPooling or tuple(tf.Tensor)

A transformers.models.data2vec.modeling_tf_data2vec_vision.TFData2VecVisionModelOutputWithPooling or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (Data2VecVisionConfig) and inputs.
last_hidden_state (tf.Tensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (tf.Ten


sor of shape (batch_size, hidden_size)) — Average of the last layer hidden states of the patch tokens (excluding the [CLS] token) if
config.use_mean_pooling is set to True. If set to False, then the final hidden state of the [CLS] token
will be returned.
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted averag


e in the self-attention
heads.
  The TFData2VecVisionModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoImageProcessor, TFData2VecVisionModel
>>> from datasets import load_dataset
>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]
>>> image_processor = AutoImageProcessor.from_pretrained("facebook/data2vec-vision-base")
>>> model = TFData2VecVisionModel.from_pretrained("facebook/data2vec-vision-base")
>>> inputs = image_processor(image, return_tensors="tf")
>>> outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1, 197, 768


class transformers.TFData2VecVisionForImageClassification  < source > ( *args **kwargs  )   Parameters   config (Data2VecVisionConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Data2VecVision Model transformer with an image classification head on top (a linear layer on top of the average of
the final hidden states of the patch tokens) e.g. for ImageNet. This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.). This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior. TensorFlow mode


ls and layers in transformers accept two formats as input: having all inputs as keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with pixel_values only and nothing else: model(pixel_values) a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([pixel_values, at


tention_mask]) or model([pixel_values, attention_mask, token_type_ids]) a dictionary with one or several input Tensors associated to the input names given in the docstring:
model({"pixel_values": pixel_values, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( pixel_values: TFModelInputType | None = None head_mask: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None labels: np.ndarray | tf.Tensor | None = None training: Optional[bool] = False  ) → transformers.modeling_tf_outputs.TFSequenceClassifierOutput or tuple(tf.Tensor)  Parameters   pixel_values (np.ndarray, tf.Tensor, List[tf.Tensor] Dict[str, tf.Tensor] or Dict[str, np.ndarray] and each example must have the shape (batch_


size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
BeitImageProcessor.call() for details.   head_mask (np.ndarray or tf.Tensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used
in eager mode, in graph mode the value will always be set to True.   training (bool, optional, defaults to `False“) —


Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).   labels (tf.Tensor or np.ndarray of shape (batch_size,), optional) —
Labels for computing the image classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-Entropy).   Returns
transformers.modeling_tf_outputs.TFSequenceClassifierOutput or tuple(tf.Tensor)

A transformers.modeling_tf_outputs.TFSequenceClassifierOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (Data2VecVisionConfig) and inputs.
loss (tf.Tensor of shape (batch_size, ), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits


(tf.Tensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape
(batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The TFData2VecVisionForImageClassification forward method, overrides the __call__ special method. Although the recipe


 for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoImageProcessor, TFData2VecVisionForImageClassification
>>> import tensorflow as tf
>>> from datasets import load_dataset
>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]
>>> image_processor = AutoImageProcessor.from_pretrained("facebook/data2vec-vision-base-ft1k")
>>> model = TFData2VecVisionForImageClassification.from_pretrained("facebook/data2vec-vision-base-ft1k")
>>> inputs = image_processor(image, return_tensors="tf")
>>> logits = model(**inputs).logits
>>> # model predicts one of the 1000 ImageNet classes
>>> predicted_label = int(tf.math.argmax(logits, axis=-1))
>>> print(model.config.id2label[predicted_label])
remote control, remo


class transformers.TFData2VecVisionForSemanticSegmentation  < source > ( *args **kwargs  )   Parameters   config (Data2VecVisionConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Data2VecVision Model transformer with a semantic segmentation head on top e.g. for ADE20k, CityScapes. This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.). This model is also a tf.keras.Model subclass. Use it
as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and
behavior. TensorFlow models and layers in transformers accept two formats as input: having all inputs a


s keyword arguments (like PyTorch models), or having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models
and layers. Because of this support, when using methods like model.fit() things should “just work” for you - just
pass your inputs and labels in any format that model.fit() supports! If, however, you want to use the second
format outside of Keras methods like fit() and predict(), such as when creating your own layers or models with
the Keras Functional API, there are three possibilities you can use to gather all the input Tensors in the first
positional argument: a single Tensor with pixel_values only and nothing else: model(pixel_values) a list of varying length with one or several input Tensors IN THE ORDER given in the docstring:
model([pixel_values, attention_mask]) or model([pixel_values, attention_mask, token_type_ids]) a diction


ary with one or several input Tensors associated to the input names given in the docstring:
model({"pixel_values": pixel_values, "token_type_ids": token_type_ids}) Note that when creating models and layers with
subclassing then you don’t need to worry
about any of this, as you can just pass inputs like you would to any other Python function!  call  < source > ( pixel_values: tf.Tensor | None = None head_mask: tf.Tensor | None = None labels: tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None  ) → transformers.modeling_tf_outputs.TFSemanticSegmenterOutput or tuple(tf.Tensor)  Parameters   pixel_values (np.ndarray, tf.Tensor, List[tf.Tensor] Dict[str, tf.Tensor] or Dict[str, np.ndarray] and each example must have the shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
BeitImageProcessor.call() for details.


 head_mask (np.ndarray or tf.Tensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used
in eager mode, in graph mode the value will always be set to True.   training (bool, optional, defaults to `False“) —
Whether or not to use the model in training mode (some modules like dropout modules have different
behaviors between training and evaluation).   labels (


tf.Tensor of shape (batch_size, height, width), optional) —
Ground truth semantic segmentation maps for computing the loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels > 1, a classification loss is computed (Cross-Entropy).   Returns
transformers.modeling_tf_outputs.TFSemanticSegmenterOutput or tuple(tf.Tensor)

A transformers.modeling_tf_outputs.TFSemanticSegmenterOutput or a tuple of tf.Tensor (if
return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the
configuration (Data2VecVisionConfig) and inputs.
loss (tf.Tensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (tf.Tensor of shape (batch_size, config.num_labels, logits_height, logits_width)) — Classification scores for each pixel.
The logits returned do not necessarily have the same size as the pixel_values passed as inputs. This is
to avoid doing tw


o interpolations and lose some quality when a user needs to resize the logits to the
original image size as post-processing. You should always check your logits shape and resize as needed.
hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of tf.Tensor (one for the output of the embeddings, if the model has an embedding layer, + one for
the output of each layer) of shape (batch_size, patch_size, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, patch_size, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The TFData2VecVisionForSemant


icSegmentation forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoImageProcessor, TFData2VecVisionForSemanticSegmentation
>>> from PIL import Image
>>> import requests
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> image_processor = AutoImageProcessor.from_pretrained("facebook/data2vec-vision-base")
>>> model = TFData2VecVisionForSemanticSegmentation.from_pretrained("facebook/data2vec-vision-base")
>>> inputs = image_processor(images=image, return_tensors="pt")
>>> outputs = model(**inputs)
>>> # logits are of shape (batch_size, num_labels, height, width)
>>> logits = outputs.log


Overview

The CLAP model was proposed in Large Scale Contrastive Language-Audio pretraining with
feature fusion and keyword-to-caption augmentation by Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, Shlomo Dubnov.
CLAP (Contrastive Language-Audio Pretraining) is a neural network trained on a variety of (audio, text) pairs. It can be instructed in to predict the most relevant text snippet, given an audio, without directly optimizing for the task. The CLAP model uses a SWINTransformer to get audio features from a log-Mel spectrogram input, and a RoBERTa model to get text features. Both the text and audio features are then projected to a latent space with identical dimension. The dot product between the p


Overview

rojected audio and text features is then used as a similar score.
The abstract from the paper is the following:
Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augm


Overview

entation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zeroshot setting and is able to obtain performance comparable to models’ results in the non-zero-shot setting. LAION-Audio-6
This model was contributed by Younes Belkada and Arthur Zucker .
The original code can be found her


ClapConfig

class transformers.ClapConfig  < source > ( text_config = None audio_config = None logit_scale_init_value = 14.285714285714285 projection_dim = 512 projection_hidden_act = 'relu' initializer_factor = 1.0 **kwargs  )   Parameters   text_config (dict, optional) —
Dictionary of configuration options used to initialize ClapTextConfig.   audio_config (dict, optional) —
Dictionary of configuration options used to initialize ClapAudioConfig.   logit_scale_init_value (float, optional, defaults to 14.29) —
The inital value of the logit_scale paramter. Default is used as per the original CLAP implementation.   projection_dim (int, optional, defaults to 512) —
Dimentionality of text and audio projection layers.   projection_hidden_act (str, optional, defaults to "relu") —
Activation function for the projection layers.   initializer_factor (float,


ClapConfig

optional, defaults to 1.0) —
Factor to scale the initialization of the model weights.   kwargs (optional) —
Dictionary of keyword arguments.    ClapConfig is the configuration class to store the configuration of a ClapModel. It is used to instantiate
a CLAP model according to the specified arguments, defining the text model and audio model configs. Instantiating a
configuration with the defaults will yield a similar configuration to that of the CLAP
laion/clap-htsat-fused architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import ClapConfig, ClapModel
>>> # Initializing a ClapConfig with laion-ai/base style configuration
>>> configuration = ClapConfig()
>>> # Initializing


ClapConfig

 a ClapModel (with random weights) from the laion-ai/base style configuration
>>> model = ClapModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config
>>> # We can also initialize a ClapConfig from a ClapTextConfig and a ClapAudioConfig
>>> from transformers import ClapTextConfig, ClapAudioConfig
>>> # Initializing a ClapText and ClapAudioConfig configuration
>>> config_text = ClapTextConfig()
>>> config_audio = ClapAudioConfig()
>>> config = ClapConfig.from_text_audio_configs(config_text, config_audio)  from_text_audio_configs  < source > ( text_config: ClapTextConfig audio_config: ClapAudioConfig **kwargs  ) → ClapConfig  Returns
ClapConfig

An instance of a configuration object
  Instantiate a ClapConfig (or a derived class) from clap text model configuration and clap audio model
configuration.


ClapTextConfig

class transformers.ClapTextConfig  < source > ( vocab_size = 50265 hidden_size = 768 num_hidden_layers = 12 num_attention_heads = 12 intermediate_size = 3072 hidden_act = 'gelu' hidden_dropout_prob = 0.1 attention_probs_dropout_prob = 0.1 max_position_embeddings = 514 type_vocab_size = 1 initializer_factor = 1.0 layer_norm_eps = 1e-12 projection_dim = 512 pad_token_id = 1 bos_token_id = 0 eos_token_id = 2 position_embedding_type = 'absolute' use_cache = True projection_hidden_act = 'relu' **kwargs  )   Parameters   vocab_size (int, optional, defaults to 30522) —
Vocabulary size of the CLAP model. Defines the number of different tokens that can be represented by the
inputs_ids passed when calling ClapTextModel.   hidden_size (int, optional, defaults to 768) —
Dimensionality of the encoder layers and the pooler laye


ClapTextConfig

r.   num_hidden_layers (int, optional, defaults to 12) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 12) —
Number of attention heads for each attention layer in the Transformer encoder.   intermediate_size (int, optional, defaults to 3072) —
Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.   hidden_act (str or Callable, optional, defaults to "relu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "relu",
"relu", "silu" and "relu_new" are supported.   hidden_dropout_prob (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.   attention_probs_dropout_prob (float, optional, defaults to 0.


ClapTextConfig

1) —
The dropout ratio for the attention probabilities.   max_position_embeddings (int, optional, defaults to 512) —
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).   type_vocab_size (int, optional, defaults to 2) —
The vocabulary size of the token_type_ids passed when calling ClapTextModel.   layer_norm_eps (float, optional, defaults to 1e-12) —
The epsilon used by the layer normalization layers.   position_embedding_type (str, optional, defaults to "absolute") —
Type of position embedding. Choose one of "absolute", "relative_key", "relative_key_query". For
positional embeddings use "absolute". For more information on "relative_key", please refer to
Self-Attention with Relative Position Representations (Shaw et al


ClapTextConfig

.).
For more information on "relative_key_query", please refer to Method 4 in Improve Transformer Models
with Better Relative Position Embeddings (Huang et al.).   is_decoder (bool, optional, defaults to False) —
Whether the model is used as a decoder or not. If False, the model is used as an encoder.   use_cache (bool, optional, defaults to True) —
Whether or not the model should return the last key/values attentions (not used by all models). Only
relevant if config.is_decoder=True.   projection_hidden_act (str, optional, defaults to "relu") —
The non-linear activation function (function or string) in the projection layer. If string, "gelu",
"relu", "silu" and "gelu_new" are supported.   projection_dim (int, optional, defaults to 512) —
Dimension of the projection head of the ClapTextModelWithProjection.    This


ClapTextConfig

is the configuration class to store the configuration of a ClapTextModel. It is used to instantiate a CLAP
model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
defaults will yield a similar configuration to that of the CLAP
calp-hsat-fused architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Examples:   Copied >>> from transformers import ClapTextConfig, ClapTextModel
>>> # Initializing a CLAP text configuration
>>> configuration = ClapTextConfig()
>>> # Initializing a model (with random weights) from the configuration
>>> model = ClapTextModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.con


ClapAudioConfig

class transformers.ClapAudioConfig  < source > ( window_size = 8 num_mel_bins = 64 spec_size = 256 hidden_act = 'gelu' patch_size = 4 patch_stride = [4, 4] num_classes = 527 hidden_size = 768 projection_dim = 512 depths = [2, 2, 6, 2] num_attention_heads = [4, 8, 16, 32] enable_fusion = False hidden_dropout_prob = 0.1 fusion_type = None patch_embed_input_channels = 1 flatten_patch_embeds = True patch_embeds_hidden_size = 96 enable_patch_layer_norm = True drop_path_rate = 0.0 attention_probs_dropout_prob = 0.0 qkv_bias = True mlp_ratio = 4.0 aff_block_r = 4 num_hidden_layers = 4 projection_hidden_act = 'relu' layer_norm_eps = 1e-05 initializer_factor = 1.0 **kwargs  )   Parameters   window_size (int, optional, defaults to 8) —
Image size of the spectrogram   num_mel_bins (int, optional, defaults to 64) —
Number of mel features used per frames.


ClapAudioConfig

 Should correspond to the value used in the ClapProcessor class.   spec_size (int, optional, defaults to 256) —
Desired input size of the spectrogram that the model supports. It can be different from the output of the
ClapFeatureExtractor, in which case the input features will be resized. Corresponds to the image_size
of the audio models.   hidden_act (str, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "silu" and "gelu_new" are supported.   patch_size (int, optional, defaults to 4) —
Patch size for the audio spectrogram   patch_stride (list, optional, defaults to [4, 4]) —
Patch stride for the audio spectrogram   num_classes (int, optional, defaults to 527) —
Number of classes used for the head training   hidden_size (int, optional, defaults to 768


ClapAudioConfig

) —
Hidden size of the output of the audio encoder. Correspond to the dimension of the penultimate layer’s
output,which is sent to the projection MLP layer.   projection_dim (int, optional, defaults to 512) —
Hidden size of the projection layer.   depths (list, optional, defaults to [2, 2, 6, 2]) —
Depths used for the Swin Layers of the audio model   num_attention_heads (list, optional, defaults to [4, 8, 16, 32]) —
Number of attention heads used for the Swin Layers of the audio model   enable_fusion (bool, optional, defaults to False) —
Whether or not to enable patch fusion. This is the main contribution of the authors, and should give the
best results.   hidden_dropout_prob (float, optional, defaults to 0.1) —
The dropout probabilitiy for all fully connected layers in the encoder.   fusion_type ([type], optional) —
Fusion type used for the


ClapAudioConfig

patch fusion.   patch_embed_input_channels (int, optional, defaults to 1) —
Number of channels used for the input spectrogram   flatten_patch_embeds (bool, optional, defaults to True) —
Whether or not to flatten the patch embeddings   patch_embeds_hidden_size (int, optional, defaults to 96) —
Hidden size of the patch embeddings. It is used as the number of output channels.   enable_patch_layer_norm (bool, optional, defaults to True) —
Whether or not to enable layer normalization for the patch embeddings   drop_path_rate (float, optional, defaults to 0.0) —
Drop path rate for the patch fusion   attention_probs_dropout_prob (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.   qkv_bias (bool, optional, defaults to True) —
Whether or not to add a bias to the query, key, value projections.   mlp_ratio (float, o


ClapAudioConfig

ptional, defaults to 4.0) —
Ratio of the mlp hidden dim to embedding dim.   aff_block_r (int, optional, defaults to 4) —
downsize_ratio used in the AudioFF block   num_hidden_layers (int, optional, defaults to 4) —
Number of hidden layers in the Transformer encoder.   projection_hidden_act (str, optional, defaults to "relu") —
The non-linear activation function (function or string) in the projection layer. If string, "gelu",
"relu", "silu" and "gelu_new" are supported.   layer_norm_eps ([type], optional, defaults to 1e-05) —
The epsilon used by the layer normalization layers.   initializer_factor (float, optional, defaults to 1.0) —
A factor for initializing all weight matrices (should be kept to 1, used internally for initialization
testing).    This is the configuration class to store the configuration of a ClapAudioModel. It is used to ins


ClapAudioConfig

tantiate a
CLAP audio encoder according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the audio encoder of the CLAP
laion/clap-htsat-fused architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import ClapAudioConfig, ClapAudioModel
>>> # Initializing a ClapAudioConfig with laion/clap-htsat-fused style configuration
>>> configuration = ClapAudioConfig()
>>> # Initializing a ClapAudioModel (with random weights) from the laion/clap-htsat-fused style configuration
>>> model = ClapAudioModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config


ClapFeatureExtractor

class transformers.ClapFeatureExtractor  < source > ( feature_size = 64 sampling_rate = 48000 hop_length = 480 max_length_s = 10 fft_window_size = 1024 padding_value = 0.0 return_attention_mask = False frequency_min: float = 0 frequency_max: float = 14000 top_db: int = None truncation: str = 'fusion' padding: str = 'repeatpad' **kwargs  )   Parameters   feature_size (int, optional, defaults to 64) —
The feature dimension of the extracted Mel spectrograms. This corresponds to the number of mel filters
(n_mels).   sampling_rate (int, optional, defaults to 48000) —
The sampling rate at which the audio files should be digitalized expressed in hertz (Hz). This only serves
to warn users if the audio fed to the feature extractor does not have the same sampling rate.   hop_length (int,optional, defaults to 480) —
Length of the overlaping windows for the STFT used to obtain the Mel Spectrogram. T


ClapFeatureExtractor

he audio will be split
in smaller frames with a step of hop_length between each frame.   max_length_s (int, optional, defaults to 10) —
The maximum input length of the model in seconds. This is used to pad the audio.   fft_window_size (int, optional, defaults to 1024) —
Size of the window (in samples) on which the Fourier transform is applied. This controls the frequency
resolution of the spectrogram. 400 means that the fourrier transform is computed on windows of 400 samples.   padding_value (float, optional, defaults to 0.0) —
Padding value used to pad the audio. Should correspond to silences.   return_attention_mask (bool, optional, defaults to False) —
Whether or not the model should return the attention masks coresponding to the input.   frequency_min (float, optional, defaults to 0) —
The lowest frequency of interest. The STFT will not be computed for values below this.   frequency


ClapFeatureExtractor

_max (float, optional, defaults to 14000) —
The highest frequency of interest. The STFT will not be computed for values above this.   top_db (float, optional) —
The highest decibel value used to convert the mel spectrogram to the log scale. For more details see the
audio_utils.power_to_db function   truncation (str, optional, defaults to "fusion") —
Truncation pattern for long audio inputs. Two patterns are available:
fusion will use _random_mel_fusion, which stacks 3 random crops from the mel spectrogram and a
downsampled version of the entire mel spectrogram.
If config.fusion is set to True, shorter audios also need to to return 4 mels, which will just be a copy
of the original mel obtained from the padded audio.
rand_trunc will select a random crop of the mel spectrogram.
   padding (str, optional, defaults to "repeatpad") —
Padding pattern for shorter audio inputs. Three patterns wer


ClapFeatureExtractor

e originally implemented:
repeatpad: the audio is repeated, and then padded to fit the max_length.
repeat: the audio is repeated and then cut to fit the max_length
pad: the audio is padded.
    Constructs a CLAP feature extractor. This feature extractor inherits from SequenceFeatureExtractor which contains
most of the main methods. Users should refer to this superclass for more information regarding those methods. This class extracts mel-filter bank features from raw speech using a custom numpy implementation of the Short Time
Fourier Transform (STFT) which should match pytorch’s torch.stft equivalent.  to_dict  < source > ( ) → Dict[str, Any]  Returns
Dict[str, Any]

Dictionary of all the attributes that make up this configuration instance, excpet for the
mel filter banks, which do not need to be saved or printed as they are too long.
  Serializes this instance to a Python dictionary


ClapProcessor

class transformers.ClapProcessor  < source > ( feature_extractor tokenizer  )   Parameters   feature_extractor (ClapFeatureExtractor) —
The audio processor is a required input.   tokenizer (RobertaTokenizerFast) —
The tokenizer is a required input.    Constructs a CLAP processor which wraps a CLAP feature extractor and a RoBerta tokenizer into a single processor. ClapProcessor offers all the functionalities of ClapFeatureExtractor and RobertaTokenizerFast. See the
__call__() and decode() for more information.  batch_decode  < source > ( *args **kwargs  )    This method forwards all its arguments to RobertaTokenizerFast’s batch_decode(). Please
refer to the docstring of this method for more information.  decode  < source > ( *args **kwargs  )    This method forwards all its arguments to RobertaTokenizerFast’s decode(). Please refer
to the docstring of this method for more information.


ClapModel

class transformers.ClapModel  < source > ( config: ClapConfig  )   Parameters   config (ClapConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None input_features: typing.Optional[torch.FloatTensor] = None is_longer: typing.Optional[torch.BoolTens


ClapModel

or] = None attention_mask: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.LongTensor] = None return_loss: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.clap.modeling_clap.ClapOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not mas


ClapModel

ked,
0 for tokens that are masked.
What are attention masks?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   input_features (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Input audio features. This should be returnes by the ClapFeatureExtractor class that you can also
retrieve from AutoFeatureExtractor. See ClapFeatureExtractor.__call__() for details.   return_loss (bool, optional) —
Whether or not to return the contrastive loss.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all lay


ClapModel

ers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.models.clap.modeling_clap.ClapOutput or tuple(torch.FloatTensor)

A transformers.models.clap.modeling_clap.ClapOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapConfig'>) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when return_loss is True) — Contrastive loss for audio-text similarity.
logits_per_audio:(torch.FloatTensor of shape (audio_batch_size, text_batch_size)) — The scaled dot product scores between audio_embeds and text_embeds. This represents the audio-text
similarity scores.
logits_per_text:(torch.FloatTensor of shape (text_bat


ClapModel

ch_size, audio_batch_size)) — The scaled dot product scores between text_embeds and audio_embeds. This represents the text-audio
similarity scores.
text_embeds(torch.FloatTensor of shape (batch_size, output_dim) — The text embeddings obtained by applying the projection layer to the pooled output of ClapTextModel.
audio_embeds(torch.FloatTensor of shape (batch_size, output_dim) — The audio embeddings obtained by applying the projection layer to the pooled output of ClapAudioModel.
text_model_output(BaseModelOutputWithPooling):
The output of the ClapTextModel.
audio_model_output(BaseModelOutputWithPooling):
The output of the ClapAudioModel.
  The ClapModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing


ClapModel

 steps while
the latter silently ignores them.  Examples:   Copied >>> from datasets import load_dataset
>>> from transformers import AutoProcessor, ClapModel
>>> dataset = load_dataset("ashraq/esc50")
>>> audio_sample = dataset["train"]["audio"][0]["array"]
>>> model = ClapModel.from_pretrained("laion/clap-htsat-unfused")
>>> processor = AutoProcessor.from_pretrained("laion/clap-htsat-unfused")
>>> input_text = ["Sound of a dog", "Sound of vaccum cleaner"]
>>> inputs = processor(text=input_text, audios=audio_sample, return_tensors="pt", padding=True)
>>> outputs = model(**inputs)
>>> logits_per_audio = outputs.logits_per_audio  # this is the audio-text similarity score
>>> probs = logits_per_audio.softmax(dim=-1)  # we can take the softmax to get the label probabilities  get_text_features  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None position_i


ClapModel

ds: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → text_features (torch.FloatTensor of shape (batch_size, output_dim)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (torch.LongTensor of shape (batch_size, sequence_length


ClapModel

), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
text_features (torch.FloatTensor of shape (batch_size, output_dim)

The text embeddings obtained by
applying the projection layer to the pooled output of ClapTextModel.
  The ClapModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this f


ClapModel

unction, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoTokenizer, ClapModel
>>> model = ClapModel.from_pretrained("laion/clap-htsat-unfused")
>>> tokenizer = AutoTokenizer.from_pretrained("laion/clap-htsat-unfused")
>>> inputs = tokenizer(["the sound of a cat", "the sound of a dog"], padding=True, return_tensors="pt")
>>> text_features = model.get_text_features(**inputs)  get_audio_features  < source > ( input_features: typing.Optional[torch.Tensor] = None is_longer: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → audio_features (torch.FloatTensor o


ClapModel

f shape (batch_size, output_dim)  Parameters   input_features (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Input audio features. This should be returnes by the ClapFeatureExtractor class that you can also
retrieve from AutoFeatureExtractor. See ClapFeatureExtractor.__call__() for details.   is_longer (torch.FloatTensor, of shape (batch_size, 1), optional) —
Whether the audio clip is longer than max_length. If True, a feature fusion will be enabled to enhance
the features.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple


ClapModel

.   Returns
audio_features (torch.FloatTensor of shape (batch_size, output_dim)

The audio embeddings obtained by
applying the projection layer to the pooled output of ClapAudioModel.
  The ClapModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoFeatureExtractor, ClapModel
>>> import torch
>>> model = ClapModel.from_pretrained("laion/clap-htsat-unfused")
>>> feature_extractor = AutoFeatureExtractor.from_pretrained("laion/clap-htsat-unfused")
>>> random_audio = torch.rand((16_000))
>>> inputs = feature_extractor(random_audio, return_tensors="pt")
>>> audio_features = model.get_audio_features(**inputs


ClapTextModel

class transformers.ClapTextModel  < source > ( config add_pooling_layer = True  )    The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of
cross-attention is added between the self-attention layers, following the architecture described in Attention is
all you need_ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz
Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the is_decoder argument of the configuration set
to True. To be used in a Seq2Seq model, the model needs to initialized with both is_decoder argument and
add_cross_attention set to True; an encoder_hidden_states is then expected as an input to the forward pass. .. _Attention is all you need: https://arxiv.org/abs/1706.03762  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tens


ClapTextModel

or] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None encoder_hidden_states: typing.Optional[torch.Tensor] = None encoder_attention_mask: typing.Optional[torch.Tensor] = None past_key_values: typing.Optional[typing.List[torch.FloatTensor]] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  )    encoder_hidden_states  (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional):
Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if
the model is configured as a decoder.
encoder_attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional):
Mask to avoid


ClapTextModel

performing attention on the padding token indices of the encoder input. This mask is used in
the cross-attention if the model is configured as a decoder. Mask values selected in [0, 1]: 1 for tokens that are not masked, 0 for tokens that are masked.
past_key_values (tuple(tuple(torch.FloatTensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)):
Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).
use_cache (bool, optional):
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values


ClapTextModelWithProjection

class transformers.ClapTextModelWithProjection  < source > ( config: ClapTextConfig  )   Parameters   config (ClapConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    CLAP Text Model with a projection layer on top (a linear layer on top of the pooled output). This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tens


ClapTextModelWithProjection

or] = None position_ids: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.clap.modeling_clap.ClapTextModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional)


ClapTextModelWithProjection

—
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.models.clap.modeling_clap.ClapTextModelOutput or tuple(torch.FloatTensor)

A transformers.models.clap.modeling_clap.ClapTextModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapTextConfig'>)


ClapTextModelWithProjection

and inputs.
text_embeds (torch.FloatTensor of shape (batch_size, output_dim) optional returned when model is initialized with with_projection=True) — The text embeddings obtained by applying the projection layer to the pooler_output.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of


ClapTextModelWithProjection

torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The ClapTextModelWithProjection forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from transformers import AutoTokenizer, ClapTextModelWithProjection
>>> model = ClapTextModelWithProjection.from_pretrained("laion/clap-htsat-unfused")
>>> tokenizer = AutoTokenizer.from_pretrained("laion/clap-htsat-unfused")
>>> inputs = tokenizer(["a sound of a cat", "a sound of a dog"], padding=True, return_tensors="pt")
>>> outputs = model(**inputs)
>>> text_embeds = outputs.text_embe


ClapAudioModel

class transformers.ClapAudioModel  < source > ( config: ClapAudioConfig  )     forward  < source > ( input_features: typing.Optional[torch.FloatTensor] = None is_longer: typing.Optional[torch.BoolTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.BaseModelOutputWithPooling or tuple(torch.FloatTensor)  Parameters   input_features (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Input audio features. This should be returnes by the ClapFeatureExtractor class that you can also
retrieve from AutoFeatureExtractor. See ClapFeatureExtractor.__call__() for details.   is_longer (torch.FloatTensor, of shape (batch_size, 1), optional) —
Whether t


ClapAudioModel

he audio clip is longer than max_length. If True, a feature fusion will be enabled to enhance
the features.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.BaseModelOutputWithPooling or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutputWithPooling or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various


ClapAudioModel

elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapAudioConfig'>) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing
through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns
the classification token after processing through a linear layer and a tanh activation function. The linear
layer weights are trained from the next sentence prediction (classification) objective during pretraining.
hidden_states (tuple


ClapAudioModel

(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The ClapAudioModel fo


ClapAudioModel

rward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from datasets import load_dataset
>>> from transformers import AutoProcessor, ClapAudioModel
>>> dataset = load_dataset("ashraq/esc50")
>>> audio_sample = dataset["train"]["audio"][0]["array"]
>>> model = ClapAudioModel.from_pretrained("laion/clap-htsat-fused")
>>> processor = AutoProcessor.from_pretrained("laion/clap-htsat-fused")
>>> inputs = processor(audios=audio_sample, return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_state = outputs.last_hidden_s


ClapAudioModelWithProjection

class transformers.ClapAudioModelWithProjection  < source > ( config: ClapAudioConfig  )   Parameters   config (ClapConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    CLAP Audio Model with a projection layer on top (a linear layer on top of the pooled output). This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_featu


ClapAudioModelWithProjection

res: typing.Optional[torch.FloatTensor] = None is_longer: typing.Optional[torch.BoolTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.models.clap.modeling_clap.ClapAudioModelOutput or tuple(torch.FloatTensor)  Parameters   input_features (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Input audio features. This should be returnes by the ClapFeatureExtractor class that you can also
retrieve from AutoFeatureExtractor. See ClapFeatureExtractor.__call__() for details.   is_longer (torch.FloatTensor, of shape (batch_size, 1), optional) —
Whether the audio clip is longer than max_length. If True, a feature fusion will be enabled to enhance
the features.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all atten


ClapAudioModelWithProjection

tion layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.models.clap.modeling_clap.ClapAudioModelOutput or tuple(torch.FloatTensor)

A transformers.models.clap.modeling_clap.ClapAudioModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapAudioConfig'>) and inputs.
audio_embeds (torch.FloatTensor of shape (batch_size, hidden_size)) — The Audio embeddings obtained by applying the projection layer to the pooler_output.
last_hidden_state (t


ClapAudioModelWithProjection

orch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer pl


ClapAudioModelWithProjection

us the optional initial embedding outputs.
  The ClapAudioModelWithProjection forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Examples:   Copied >>> from datasets import load_dataset
>>> from transformers import ClapAudioModelWithProjection, ClapProcessor
>>> model = ClapAudioModelWithProjection.from_pretrained("laion/clap-htsat-fused")
>>> processor = ClapProcessor.from_pretrained("laion/clap-htsat-fused")
>>> dataset = load_dataset("ashraq/esc50")
>>> audio_sample = dataset["train"]["audio"][0]["array"]
>>> inputs = processor(audios=audio_sample, return_tensors="pt")
>>> outputs = model(**inputs)
>>> audio_embeds = outputs.audio_em


Overview

The mLUKE model was proposed in mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models by Ryokan Ri, Ikuya Yamada, and Yoshimasa Tsuruoka. It’s a multilingual extension
of the LUKE model trained on the basis of XLM-RoBERTa.
It is based on XLM-RoBERTa and adds entity embeddings, which helps improve performance on various downstream tasks
involving reasoning about entities such as named entity recognition, extractive question answering, relation
classification, cloze-style knowledge completion.
The abstract from the paper is the following:
Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual
alignment information fro


Overview

m Wikipedia entities. However, existing methods only exploit entity information in pretraining
and do not explicitly use entities in downstream tasks. In this study, we explore the effectiveness of leveraging
entity representations for downstream cross-lingual tasks. We train a multilingual language model with 24 languages
with entity representations and show the model consistently outperforms word-based pretrained models in various
cross-lingual transfer tasks. We also analyze the model and the key insight is that incorporating entity
representations into the input allows us to extract more language-agnostic features. We also evaluate the model with a
multilingual cloze prompt task with the mLAMA dataset. We


Overview

 show that entity-based prompt elicits correct factual
knowledge more likely than using only word representations.
One can directly plug in the weights of mLUKE into a LUKE model, like so:
  Copied from transformers import LukeModel
model = LukeModel.from_pretrained("studio-ousia/mluke-base")
Note that mLUKE has its own tokenizer, MLukeTokenizer. You can initialize it as follows:
  Copied from transformers import MLukeTokenizer
tokenizer = MLukeTokenizer.from_pretrained("studio-ousia/mluke-base")
As mLUKE’s architecture is equivalent to that of LUKE, one can refer to LUKE’s documentation page for all
tips, code examples and notebooks.
This model was contributed by ryo0634. The original code can be found here.


MLukeTokenizer

class transformers.MLukeTokenizer  < source > ( vocab_file entity_vocab_file bos_token = '<s>' eos_token = '</s>' sep_token = '</s>' cls_token = '<s>' unk_token = '<unk>' pad_token = '<pad>' mask_token = '<mask>' task = None max_entity_length = 32 max_mention_length = 30 entity_token_1 = '<ent>' entity_token_2 = '<ent2>' entity_unk_token = '[UNK]' entity_pad_token = '[PAD]' entity_mask_token = '[MASK]' entity_mask2_token = '[MASK2]' sp_model_kwargs: typing.Union[typing.Dict[str, typing.Any], NoneType] = None **kwargs  )   Parameters   vocab_file (str) —
Path to the vocabulary file.   entity_vocab_file (str) —
Path to the entity vocabulary file.   bos_token (str, optional, defaults to "<s>") —
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the cls_token.
   eos_token (str, optional, def


MLukeTokenizer

aults to "</s>") —
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the sep_token.
   sep_token (str, optional, defaults to "</s>") —
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.   cls_token (str, optional, defaults to "<s>") —
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.   unk_token (str, optional, defaults to "<unk>") —
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.   pad_token (str, optional, defaults to "<pad>") —


MLukeTokenizer

The token used for padding, for example when batching sequences of different lengths.   mask_token (str, optional, defaults to "<mask>") —
The token used for masking values. This is the token used when training this model with masked language
modeling. This is the token which the model will try to predict.   task (str, optional) —
Task for which you want to prepare sequences. One of "entity_classification",
"entity_pair_classification", or "entity_span_classification". If you specify this argument, the entity
sequence is automatically created based on the given entity span(s).   max_entity_length (int, optional, defaults to 32) —
The maximum length of entity_ids.   max_mention_length (int, optional, defaults to 30) —
The maximum number of tokens inside an entity span.   entity_token_1 (str, optional, defaults to <ent>) —
The special token used to represent an entity span in a word token sequence. This token is only used when
task is set to "entity_classification" or "entit


MLukeTokenizer

y_pair_classification".   entity_token_2 (str, optional, defaults to <ent2>) —
The special token used to represent an entity span in a word token sequence. This token is only used when
task is set to "entity_pair_classification".   additional_special_tokens (List[str], optional, defaults to ["<s>NOTUSED", "</s>NOTUSED"]) —
Additional special tokens used by the tokenizer.   sp_model_kwargs (dict, optional) —
Will be passed to the SentencePieceProcessor.__init__() method. The Python wrapper for
SentencePiece can be used, among other things,
to set:
enable_sampling: Enable subword regularization.
nbest_size: Sampling parameters for unigram. Invalid for BPE-Dropout.
nbest_size = {0,1}: No sampling is performed.
nbest_size > 1: samples from the nbest_size results.
nbest_size < 0: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
using forward-filtering-and-backward-sampling algorithm.
alpha: Smoothing parameter for unigram sampling, and dropout


MLukeTokenizer

probability of merge operations for
BPE-dropout.
   sp_model (SentencePieceProcessor) —
The SentencePiece processor that is used for every conversion (string, tokens and IDs).    Adapted from XLMRobertaTokenizer and LukeTokenizer. Based on
SentencePiece. This tokenizer inherits from PreTrainedTokenizer which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.  __call__  < source > ( text: typing.Union[str, typing.List[str]] text_pair: typing.Union[str, typing.List[str], NoneType] = None entity_spans: typing.Union[typing.List[typing.Tuple[int, int]], typing.List[typing.List[typing.Tuple[int, int]]], NoneType] = None entity_spans_pair: typing.Union[typing.List[typing.Tuple[int, int]], typing.List[typing.List[typing.Tuple[int, int]]], NoneType] = None entities: typing.Union[typing.List[str], typing.List[typing.List[str]], NoneType] = None entities_pair: typing.Union[typing.List[str], typing.List[typing.List[st


MLukeTokenizer

r]], NoneType] = None add_special_tokens: bool = True padding: typing.Union[bool, str, transformers.utils.generic.PaddingStrategy] = False truncation: typing.Union[bool, str, transformers.tokenization_utils_base.TruncationStrategy] = None max_length: typing.Optional[int] = None max_entity_length: typing.Optional[int] = None stride: int = 0 is_split_into_words: typing.Optional[bool] = False pad_to_multiple_of: typing.Optional[int] = None return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None return_token_type_ids: typing.Optional[bool] = None return_attention_mask: typing.Optional[bool] = None return_overflowing_tokens: bool = False return_special_tokens_mask: bool = False return_offsets_mapping: bool = False return_length: bool = False verbose: bool = True **kwargs  ) → BatchEncoding  Parameters   text (str, List[str], List[List[str]]) —
The sequence or batch of sequences to be encoded. Each sequence must be a string. Note that this
token


MLukeTokenizer

izer does not support tokenization based on pretokenized strings.   text_pair (str, List[str], List[List[str]]) —
The sequence or batch of sequences to be encoded. Each sequence must be a string. Note that this
tokenizer does not support tokenization based on pretokenized strings.   entity_spans (List[Tuple[int, int]], List[List[Tuple[int, int]]], optional) —
The sequence or batch of sequences of entity spans to be encoded. Each sequence consists of tuples each
with two integers denoting character-based start and end positions of entities. If you specify
"entity_classification" or "entity_pair_classification" as the task argument in the constructor,
the length of each sequence must be 1 or 2, respectively. If you specify entities, the length of each
sequence must be equal to the length of each sequence of entities.   entity_spans_pair (List[Tuple[int, int]], List[List[Tuple[int, int]]], optional) —
The sequence or batch of sequences of entity spans to be encoded. Each sequ


MLukeTokenizer

ence consists of tuples each
with two integers denoting character-based start and end positions of entities. If you specify the
task argument in the constructor, this argument is ignored. If you specify entities_pair, the
length of each sequence must be equal to the length of each sequence of entities_pair.   entities (List[str], List[List[str]], optional) —
The sequence or batch of sequences of entities to be encoded. Each sequence consists of strings
representing entities, i.e., special entities (e.g., [MASK]) or entity titles of Wikipedia (e.g., Los
Angeles). This argument is ignored if you specify the task argument in the constructor. The length of
each sequence must be equal to the length of each sequence of entity_spans. If you specify
entity_spans without specifying this argument, the entity sequence or the batch of entity sequences
is automatically constructed by filling it with the [MASK] entity.   entities_pair (List[str], List[List[str]], optional) —
The sequenc


MLukeTokenizer

e or batch of sequences of entities to be encoded. Each sequence consists of strings
representing entities, i.e., special entities (e.g., [MASK]) or entity titles of Wikipedia (e.g., Los
Angeles). This argument is ignored if you specify the task argument in the constructor. The length of
each sequence must be equal to the length of each sequence of entity_spans_pair. If you specify
entity_spans_pair without specifying this argument, the entity sequence or the batch of entity
sequences is automatically constructed by filling it with the [MASK] entity.   max_entity_length (int, optional) —
The maximum length of entity_ids.   add_special_tokens (bool, optional, defaults to True) —
Whether or not to add special tokens when encoding the sequences. This will use the underlying
PretrainedTokenizerBase.build_inputs_with_special_tokens function, which defines which tokens are
automatically added to the input ids. This is usefull if you want to add bos or eos tokens
automatically.


MLukeTokenizer

 padding (bool, str or PaddingStrategy, optional, defaults to False) —
Activates and controls padding. Accepts the following values:
True or 'longest': Pad to the longest sequence in the batch (or no padding if only a single
sequence if provided).
'max_length': Pad to a maximum length specified with the argument max_length or to the maximum
acceptable input length for the model if that argument is not provided.
False or 'do_not_pad' (default): No padding (i.e., can output a batch with sequences of different
lengths).
   truncation (bool, str or TruncationStrategy, optional, defaults to False) —
Activates and controls truncation. Accepts the following values:
True or 'longest_first': Truncate to a maximum length specified with the argument max_length or
to the maximum acceptable input length for the model if that argument is not provided. This will
truncate token by token, removing a token from the longest sequence in the pair if a pair of
sequences (or a batch of pairs) is