CodeGenForCausalLM

class transformers.CodeGenForCausalLM  < source > ( config  )   Parameters   config (CodeGenConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The CodeGen Model transformer with a language modeling head on top. This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None past_key_values: typing.Optional[typing.Tuple[typing.Tuple[torch.Tensor]]] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.LongTensor] = None head_mask: typ


CodeGenForCausalLM

ing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.LongTensor] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.CausalLMOutputWithPast or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoProcenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are mask


CodeGenForCausalLM

ed.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.n_positions - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_attention_heads,) or (n_layer, num_attention_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_dim), optional) —


CodeGenForCausalLM

Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Labels for language modeling. Note that the labels are shifted inside the model, i.e. you can set
labels = input_ids Indices are selected in [-100, 0, ..., config.vocab_size]


CodeGenForCausalLM

All labels set to -100
are ignored (masked), the loss is only computed for labels in [0, ..., config.vocab_size]   Returns
transformers.modeling_outputs.CausalLMOutputWithPast or tuple(torch.FloatTensor)

A transformers.modeling_outputs.CausalLMOutputWithPast or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (CodeGenConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Language modeling loss (for next-token prediction).
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTen


CodeGenForCausalLM

sor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head))
Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see
past_key_values input) to speed up sequential decoding.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batc


CodeGenForCausalLM

h_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The CodeGenForCausalLM forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> import torch
>>> from transformers import AutoTokenizer, CodeGenForCausalLM
>>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/codegen-2B-mono")
>>> model = CodeGenForCausalLM.from_pretrained("Salesforce/codegen-2B-mono")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> outputs = model(**inputs, labels=inputs["input_ids"])
>>> loss = outputs.loss
>>> logits = outputs.logit


Overview

The PEGASUS-X model was proposed in Investigating Efficiently Extending Transformers for Long Input Summarization  by Jason Phang, Yao Zhao and Peter J. Liu.
PEGASUS-X (PEGASUS eXtended) extends the PEGASUS models for long input summarization through additional long input pretraining and using staggered block-local attention with global tokens in the encoder.
The abstract from the paper is the following:
While large pretrained Transformer models have proven highly capable at tackling natural language tasks, handling long sequence inputs continues to be a significant challenge. One such task is long input summarization, where inputs are longer than the maximum input context of most pretrained models. Through an extensive set of experiments, we investigate what model architectural changes and pretraining paradigms can most


Overview

efficiently adapt a pretrained Transformer for long input summarization. We find that a staggered, block-local Transformer with global encoder tokens strikes a good balance of performance and efficiency, and that an additional pretraining phase on long sequences meaningfully improves downstream summarization performance. Based on our findings, we introduce PEGASUS-X, an extension of the PEGASUS model with additional long input pretraining to handle inputs of up to 16K tokens. PEGASUS-X achieves strong performance on long input summarization tasks comparable with much larger models while adding few additional parameters and not requiring model parallelism to train.
Tips:
PEGASUS-X uses the same tokenizer as PEGASUS.
This model was contributed by [zphang](<https://huggingface.co/zphang). The original code can be found here.


PegasusXConfig

class transformers.PegasusXConfig  < source > ( vocab_size = 96103 max_position_embeddings = 16384 encoder_layers = 16 encoder_ffn_dim = 4096 encoder_attention_heads = 16 decoder_layers = 16 decoder_ffn_dim = 4096 decoder_attention_heads = 16 encoder_layerdrop = 0.0 decoder_layerdrop = 0.0 use_cache = True is_encoder_decoder = True activation_function = 'gelu' d_model = 1024 dropout = 0.1 attention_dropout = 0.0 activation_dropout = 0.0 init_std = 0.02 decoder_start_token_id = 0 scale_embedding = True pad_token_id = 0 eos_token_id = 1 forced_eos_token_id = 1 num_global_tokens = 32 block_size = 512 stagger_local_blocks = True **kwargs  )   Parameters   vocab_size (int, optional, defaults to 96103) —
Vocabulary size of the PEGASUS-X model. Defines the number of different tokens that can be represented by
the inputs_ids passed when calling PegasusXModel.   d_model (int, optional, defaults to 1024) —
Dimension of th


PegasusXConfig

e layers and the pooler layer.   encoder_layers (int, optional, defaults to 16) —
Number of encoder layers.   decoder_layers (int, optional, defaults to 16) —
Number of decoder layers.   encoder_attention_heads (int, optional, defaults to 16) —
Number of attention heads for each attention layer in the Transformer encoder.   decoder_attention_heads (int, optional, defaults to 16) —
Number of attention heads for each attention layer in the Transformer decoder.   decoder_ffn_dim (int, optional, defaults to 4096) —
Dimension of the “intermediate” (often named feed-forward) layer in decoder.   encoder_ffn_dim (int, optional, defaults to 4096) —
Dimension of the “intermediate” (often named feed-forward) layer in decoder.   activation_function (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "silu" and "gelu_n


PegasusXConfig

ew" are supported.   dropout (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.   attention_dropout (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.   activation_dropout (float, optional, defaults to 0.0) —
The dropout ratio for activations inside the fully connected layer.   max_position_embeddings (int, optional, defaults to 16384) —
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).   init_std (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   encoder_layerdrop (float, optional, defaults to 0.0) —
The LayerDrop probability for the encoder. See the [LayerDrop paper](see https://arxiv.org/abs/1909.11556)
for more


PegasusXConfig

 details.   decoder_layerdrop (float, optional, defaults to 0.0) —
The LayerDrop probability for the decoder. See the [LayerDrop paper](see https://arxiv.org/abs/1909.11556)
for more details.   use_cache (bool, optional, defaults to True) —
Whether or not the model should return the last key/values attentions (not used by all models)   forced_eos_token_id (int, optional, defaults to 1) —
The id of the token to force as the last generated token when max_length is reached. Usually set to
eos_token_id.   num_global_tokens (int, optional, defaults to 128) —
Number of global tokens to use for the encoder   block_size (int, optional, defaults to 512) —
Block size for encoder local attention. Sequence length should be an exact multiple of block size.
block_size must be a multiple of 2 if stagger_local_block is True   stagger_local_block (bool, optional, defaults to True) —
Whether to stagger every other local attention


PegasusXConfig

 by half a block    This is the configuration class to store the configuration of a PegasusXModel. It is used to instantiate a
PEGASUS-X model according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the PEGASUS-X
google/pegasus-x-large architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import PegasusXConfig, PegasusXModel
>>> # Initializing a PEGASUS google/pegasus-x-large style configuration
>>> configuration = PegasusXConfig()
>>> # Initializing a model (with random weights) from the google/pegasus-x-large style configuration
>>> model = PegasusXModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config


PegasusXModel

class transformers.PegasusXModel  < source > ( config: PegasusXConfig  )   Parameters   config (PegasusXConfig) —
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
from_pretrained() method to load the model weights.    The bare PEGASUS-X Model outputting raw hidden-states without any specific head on top.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention_mask:


PegasusXModel

 typing.Optional[torch.Tensor] = None decoder_input_ids: typing.Optional[torch.Tensor] = None decoder_attention_mask: typing.Optional[torch.Tensor] = None encoder_outputs: typing.Optional[typing.Tuple[torch.FloatTensor]] = None past_key_values: typing.Optional[typing.Tuple[torch.FloatTensor]] = None inputs_embeds: typing.Optional[torch.Tensor] = None decoder_inputs_embeds: typing.Optional[torch.Tensor] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.Seq2SeqModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer


PegasusXModel

.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation.   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   decoder_input_ids (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Indices of decoder input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are decoder input IDs?
PEGASUS-X uses the pad_token_id as the starting token for decoder_input_ids generation. If


PegasusXModel

past_key_values is used, optionally only the last decoder_input_ids have to be input (see
past_key_values).   decoder_attention_mask (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also
be used by default.   encoder_outputs (tuple(tuple(torch.FloatTensor), optional) —
Tuple consists of (last_hidden_state, optional: hidden_states, optional: attentions)
last_hidden_state of shape (batch_size, sequence_length, hidden_size), optional) is a sequence of
hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.   past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) —
Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, n


PegasusXModel

um_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length). inputs_embeds (torch.FloatTensor of shape
(batch_size, sequence_length, hidden_size), optional): Optionally, instead of passing input_ids you
can choose to directly pass an embedded representation. This is useful if you want more control over how to
convert input_ids indices into associated vectors than the model’s internal emb


PegasusXModel

edding lookup matrix.   decoder_inputs_embeds (torch.FloatTensor of shape (batch_size, target_sequence_length, hidden_size), optional) —
Optionally, instead of passing decoder_input_ids you can choose to directly pass an embedded
representation. If past_key_values is used, optionally only the last decoder_inputs_embeds have to be
input (see past_key_values). This is useful if you want more control over how to convert
decoder_input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
If decoder_input_ids and decoder_inputs_embeds are both unset, decoder_inputs_embeds takes the value
of inputs_embeds.   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
te


PegasusXModel

nsors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.Seq2SeqModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.Seq2SeqModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (PegasusXConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the decoder of the model.
If past_key_values is used only the last hidden-state of the sequences of shape (batch_size, 1, hidden_size) is output.
past_key_values (tuple(tuple(tor


PegasusXModel

ch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the decoder


PegasusXModel

 at the output of each layer plus the optional initial embedding outputs.
decoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
encoder_last_hidden_state (torch.FloatTensor of shape (batch_


PegasusXModel

size, sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the optional initial embedding outputs.
encoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the encoder, after the attention softmax, used to compute the weighted av


PegasusXModel

erage in the
self-attention heads.
  The PegasusXModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, PegasusModel
>>> tokenizer = AutoTokenizer.from_pretrained("google/pegasus-x-large")
>>> model = PegasusModel.from_pretrained("google/pegasus-x-large")
>>> inputs = tokenizer("Studies have been shown that owning a dog is good for you", return_tensors="pt")
>>> decoder_inputs = tokenizer("Studies show that", return_tensors="pt")
>>> outputs = model(input_ids=inputs.input_ids, decoder_input_ids=decoder_inputs.input_ids)
>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1


PegasusXForConditionalGeneration

class transformers.PegasusXForConditionalGeneration  < source > ( config: PegasusXConfig  )   Parameters   config (PegasusXConfig) —
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
from_pretrained() method to load the model weights.    The PEGASUS-X for conditional generation (e.g. summarization).
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None attention


PegasusXForConditionalGeneration

_mask: typing.Optional[torch.Tensor] = None decoder_input_ids: typing.Optional[torch.Tensor] = None decoder_attention_mask: typing.Optional[torch.Tensor] = None encoder_outputs: typing.Optional[typing.Tuple[torch.FloatTensor]] = None past_key_values: typing.Optional[typing.Tuple[torch.FloatTensor]] = None inputs_embeds: typing.Optional[torch.Tensor] = None decoder_inputs_embeds: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.Seq2SeqLMOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices


PegasusXForConditionalGeneration

can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation.   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   decoder_input_ids (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Indices of decoder input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are decoder input IDs?
PEGASUS-X uses the


PegasusXForConditionalGeneration

 pad_token_id as the starting token for decoder_input_ids generation. If
past_key_values is used, optionally only the last decoder_input_ids have to be input (see
past_key_values).   decoder_attention_mask (torch.LongTensor of shape (batch_size, target_sequence_length), optional) —
Default behavior: generate a tensor that ignores pad tokens in decoder_input_ids. Causal mask will also
be used by default.   encoder_outputs (tuple(tuple(torch.FloatTensor), optional) —
Tuple consists of (last_hidden_state, optional: hidden_states, optional: attentions)
last_hidden_state of shape (batch_size, sequence_length, hidden_size), optional) is a sequence of
hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.   past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) —
Tuple of tuple(torch.FloatTensor)


PegasusXForConditionalGeneration

 of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length). inputs_embeds (torch.FloatTensor of shape
(batch_size, sequence_length, hidden_size), optional): Optionally, instead of passing input_ids you
can choose to directly pass an embedded representation. This is useful if you want more con


PegasusXForConditionalGeneration

trol over how to
convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.   decoder_inputs_embeds (torch.FloatTensor of shape (batch_size, target_sequence_length, hidden_size), optional) —
Optionally, instead of passing decoder_input_ids you can choose to directly pass an embedded
representation. If past_key_values is used, optionally only the last decoder_inputs_embeds have to be
input (see past_key_values). This is useful if you want more control over how to convert
decoder_input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
If decoder_input_ids and decoder_inputs_embeds are both unset, decoder_inputs_embeds takes the value
of inputs_embeds.   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional)


PegasusXForConditionalGeneration

 —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Labels for computing the masked language modeling loss. Indices should either be in [0, ..., config.vocab_size] or -100 (see input_ids docstring). Tokens with indices set to -100 are ignored
(masked), the loss is only computed for the tokens with labels in [0, ..., config.vocab_size].   Returns
transformers.modeling_outputs.Seq2SeqLMOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.Seq2SeqLMOutput or a tuple of
torch.FloatTensor (if re


PegasusXForConditionalGeneration

turn_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (PegasusXConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Language modeling loss.
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks


PegasusXForConditionalGeneration

 and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the decoder at the output of each layer plus the initial embedding outputs.
decoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-a


PegasusXForConditionalGeneration

ttention heads.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the
weighted average in the cross-attention heads.
encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape


PegasusXForConditionalGeneration

 (batch_size, sequence_length, hidden_size).
Hidden-states of the encoder at the output of each layer plus the initial embedding outputs.
encoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
  The PegasusXForConditionalGeneration forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Summarization example:   Copied >>> from transfo


PegasusXForConditionalGeneration

rmers import AutoTokenizer, PegasusXForConditionalGeneration
>>> model = PegasusXForConditionalGeneration.from_pretrained("google/pegasus-x-base")
>>> tokenizer = AutoTokenizer.from_pretrained("google/pegasus-x-large")
>>> ARTICLE_TO_SUMMARIZE = (
...     "PG&E stated it scheduled the blackouts in response to forecasts for high winds "
...     "amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were "
...     "scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow."
... )
>>> inputs = tokenizer(ARTICLE_TO_SUMMARIZE, max_length=1024, return_tensors="pt")
>>> # Generate Summary
>>> summary_ids = model.generate(inputs["input_ids"])
>>> tokenizer.batch_decode(summary_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
"California's largest electricity provider has turned off power to hundreds of thousands of c


Overview

The Jukebox model was proposed in Jukebox: A generative model for music
by Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford,
Ilya Sutskever. It introduces a generative music model which can produce minute long samples that can be conditioned on
an artist, genres and lyrics.
The abstract from the paper is the following:
We introduce Jukebox, a model that generates music with singing in the raw audio domain. We tackle the long context of raw audio using a multiscale VQ-VAE to compress it to discrete codes, and modeling those using autoregressive Transformers. We show that the combined model at scale can generate high-fidelity and diverse songs with coherence up to multiple minutes. We can condition on artist and genre to steer the musical and vocal style, and on unaligned lyrics to make the singing more controllable. We are releasin


Overview

g thousands of non cherry-picked samples, along with model weights and code.
As shown on the following figure, Jukebox is made of 3 priors which are decoder only models. They follow the architecture described in Generating Long Sequences with Sparse Transformers, modified to support longer context length.
First, a autoencoder is used to encode the text lyrics. Next, the first (also called top_prior) prior attends to the last hidden states extracted from the lyrics encoder. The priors are linked to the previous priors respectively via an AudioConditionner module. TheAudioConditioner upsamples the outputs of the previous prior to raw tokens at a certain audio frame per second resolution.
The metadata such as artist, genre and timing are passed to each prior, in the form of a start token and positionnal embedding for the timing data.  The hidden states are map


Overview

ped to the closest codebook vector from the VQVAE in order to convert them to raw audio.
Tips:
This model only supports inference. This is for a few reasons, mostly because it requires a crazy amount of memory to train. Feel free to open a PR and add what’s missing to have a full integration with the hugging face traineer! This model is very slow, and takes 8h to generate a minute long audio using the 5b top prior on a V100 GPU. In order automaticallay handle the device on which the model should execute, use accelerate. Contrary to the paper, the order of the priors goes from 0 to 1 as it felt more intuitive : we sample starting from 0. Primed sampling (conditionning the sampling on raw audio) requires more memory than ancestral sampling and should be used with fp16 set to True.
This model was contributed by Arthur Zucker.
The original code can be found her


JukeboxConfig

class transformers.JukeboxConfig  < source > ( vqvae_config = None prior_config_list = None nb_priors = 3 sampling_rate = 44100 timing_dims = 64 min_duration = 0 max_duration = 600.0 max_nb_genres = 5 metadata_conditioning = True **kwargs  )   Parameters   vqvae_config (JukeboxVQVAEConfig, optional) —
Configuration for the JukeboxVQVAE model.   prior_config_list (List[JukeboxPriorConfig], optional) —
List of the configs for each of the JukeboxPrior of the model. The original architecture uses 3 priors.   nb_priors (int, optional, defaults to 3) —
Number of prior models that will sequentially sample tokens. Each prior is conditional auto regressive
(decoder) model, apart from the top prior, which can include a lyric encoder. The available models were
trained using a to


JukeboxConfig

p prior and 2 upsampler priors.   sampling_rate (int, optional, defaults to 44100) —
Sampling rate of the raw audio.   timing_dims (int, optional, defaults to 64) —
Dimensions of the JukeboxRangeEmbedding layer which is equivalent to traditional positional embedding
layer. The timing embedding layer converts the absolute and relative position in the currently sampled
audio to a tensor of length timing_dims that will be added to the music tokens.   min_duration (int, optional, defaults to 0) —
Minimum duration of the audios to generate   max_duration (float, optional, defaults to 600.0) —
Maximum duration of the audios to generate   max_nb_genres (int, optional, defaults to 5) —
Maximum number of genres that can be used to condition a single sample.   metadata_conditio


JukeboxConfig

ning (bool, optional, defaults to True) —
Whether or not to use metadata conditioning, corresponding to the artist, the genre and the min/maximum
duration.    This is the configuration class to store the configuration of a JukeboxModel. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information. Instantiating a configuration with the defaults will
yield a similar configuration to that of
openai/jukebox-1b-lyrics architecture. The downsampling and stride are used to determine downsampling of the input sequence. For example, downsampling =
(5,3), and strides = (2, 2) will downsample the audio by 2^5 = 32 to get the first level of codes, and 2**8 = 256
to get the seco


JukeboxConfig

nd level codes. This is mostly true for training the top level prior and the upsamplers.  Example:   Copied >>> from transformers import JukeboxModel, JukeboxConfig
>>> # Initializing a Jukebox configuration
>>> configuration = JukeboxConfig()
>>> # Initializing a model from the configuration
>>> model = JukeboxModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config  from_configs  < source > ( prior_configs: typing.List[transformers.models.jukebox.configuration_jukebox.JukeboxPriorConfig] vqvae_config: JukeboxVQVAEConfig **kwargs  ) → JukeboxConfig  Returns
JukeboxConfig

An instance of a configuration object
  Instantiate a JukeboxConfig (or a derived class) from clip text model configuration and clip vision model
configuratio


JukeboxPriorConfig

class transformers.JukeboxPriorConfig  < source > ( act_fn = 'quick_gelu' level = 0 alignment_head = 2 alignment_layer = 68 attention_multiplier = 0.25 attention_pattern = 'enc_dec_with_lyrics' attn_dropout = 0 attn_res_scale = False blocks = 64 conv_res_scale = None num_layers = 72 emb_dropout = 0 encoder_config = None encoder_loss_fraction = 0.4 hidden_size = 2048 init_scale = 0.2 is_encoder_decoder = True lyric_vocab_size = 80 mask = False max_duration = 600 max_nb_genres = 1 merged_decoder = True metadata_conditioning = True metadata_dims = [604, 7898] min_duration = 0 mlp_multiplier = 1.0 music_vocab_size = 2048 n_ctx = 6144 n_heads = 2 nb_relevant_lyric_tokens = 384 res_conv_depth = 3 res_conv_width = 128 res_convolution_multiplier = 1 res_dilation_cycle = None res_dilation_growth_rate = 1 res_downs_t = [3, 2, 2] res_strides_t = [2, 2, 2] resid_dropout = 0 sa


JukeboxPriorConfig

mpling_rate = 44100 spread = None timing_dims = 64 zero_out = False **kwargs  )   Parameters   act_fn (str, optional, defaults to "quick_gelu") —
Activation function.   alignment_head (int, optional, defaults to 2) —
Head that is responsible of the alignment between lyrics and music. Only used to compute the lyric to audio
alignment   alignment_layer (int, optional, defaults to 68) —
Index of the layer that is responsible of the alignment between lyrics and music. Only used to compute the
lyric to audio alignment   attention_multiplier (float, optional, defaults to 0.25) —
Multiplier coefficient used to define the hidden dimension of the attention layers. 0.25 means that
0.25*width of the model will be used.   attention_pattern (str, optional, defaults to "enc_dec_with_lyrics") —
Which attention pattern to use for the decoder/   attn_dropout (int, optional, default


JukeboxPriorConfig

s to 0) —
Dropout probability for the post-attention layer dropout in the decoder.   attn_res_scale (bool, optional, defaults to False) —
Whether or not to scale the residuals in the attention conditioner block.   blocks (int, optional, defaults to 64) —
Number of blocks used in the block_attn. A sequence of length seq_len is factored as [blocks, seq_len // blocks] in the JukeboxAttention layer.   conv_res_scale (int, optional) —
Whether or not to scale the residuals in the conditioner block. Since the top level prior does not have a
conditioner, the default value is to None and should not be modified.   num_layers (int, optional, defaults to 72) —
Number of layers of the transformer architecture.   emb_dropout (int, optional, defaults to 0) —
Embedding dropout used in the lyric decoder.   encoder_config (JukeboxPriorConfig, optional)  —
Configuration of the encode


JukeboxPriorConfig

r which models the prior on the lyrics.   encoder_loss_fraction (float, optional, defaults to 0.4) —
Multiplication factor used in front of the lyric encoder loss.   hidden_size (int, optional, defaults to 2048) —
Hidden dimension of the attention layers.   init_scale (float, optional, defaults to 0.2) —
Initialization scales for the prior modules.   is_encoder_decoder (bool, optional, defaults to True) —
Whether or not the prior is an encoder-decoder model. In case it is not, and nb_relevant_lyric_tokens is
greater than 0, the encoder args should be specified for the lyric encoding.   mask (bool, optional, defaults to False) —
Whether or not to mask the previous positions in the attention.   max_duration (int, optional, defaults to 600) —
Maximum supported duration of the generated song in seconds.   max_nb_genres (int, optional, defaults to 1) —
Maximum number of


JukeboxPriorConfig

 genres that can be used to condition the model.   merged_decoder (bool, optional, defaults to True) —
Whether or not the decoder and the encoder inputs are merged. This is used for the separated
encoder-decoder architecture   metadata_conditioning (bool, optional, defaults to True) —
Whether or not to condition on the artist and genre metadata.   metadata_dims (List[int], optional, defaults to [604, 7898]) —
Number of genres and the number of artists that were used to train the embedding layers of the prior
models.   min_duration (int, optional, defaults to 0) —
Minimum duration of the generated audio on which the model was trained.   mlp_multiplier (float, optional, defaults to 1.0) —
Multiplier coefficient used to define the hidden dimension of the MLP layers. 0.25 means that 0.25*width of
the model will be used.   music_vocab_size (int, optional, defaults to 20


JukeboxPriorConfig

48) —
Number of different music tokens. Should be similar to the JukeboxVQVAEConfig.nb_discrete_codes.   n_ctx (int, optional, defaults to 6144) —
Number of context tokens for each prior. The context tokens are the music tokens that are attended to when
generating music tokens.   n_heads (int, optional, defaults to 2) —
Number of attention heads.   nb_relevant_lyric_tokens (int, optional, defaults to 384) —
Number of lyric tokens that are used when sampling a single window of length n_ctx   res_conv_depth (int, optional, defaults to 3) —
Depth of the JukeboxDecoderConvBock used to upsample the previously sampled audio in the
JukeboxMusicTokenConditioner.   res_conv_width (int, optional, defaults to 128) —
Width of the JukeboxDecoderConvBock used to upsample the previously sampled audio in the
JukeboxMusicTokenConditioner.   res_convolution_multiplier (int, optional


JukeboxPriorConfig

, defaults to 1) —
Multiplier used to scale the hidden_dim of the JukeboxResConv1DBlock.   res_dilation_cycle (int, optional) —
Dilation cycle used to define the JukeboxMusicTokenConditioner. Usually similar to the ones used in the
corresponding level of the VQVAE. The first prior does not use it as it is not conditioned on upper level
tokens.   res_dilation_growth_rate (int, optional, defaults to 1) —
Dilation grow rate used between each convolutionnal block of the JukeboxMusicTokenConditioner   res_downs_t (List[int], optional, defaults to [3, 2, 2]) —
Downsampling rates used in the audio conditioning network   res_strides_t (List[int], optional, defaults to [2, 2, 2]) —
Striding used in the audio conditioning network   resid_dropout (int, optional, defaults to 0) —
Residual dropout used in the attention pattern.   sampling_rate (int, optional, defaults to 44100)


JukeboxPriorConfig

 —
Sampling rate used for training.   spread (int, optional) —
Spread used in the summary_spread_attention pattern   timing_dims (int, optional, defaults to 64) —
Dimension of the timing embedding.   zero_out (bool, optional, defaults to False) —
Whether or not to zero out convolution weights when initializing.    This is the configuration class to store the configuration of a JukeboxPrior. It is used to instantiate a
JukeboxPrior according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the top level prior from the
[openai/jukebox-1b-lyrics](https://huggingface.co/openai/jukebox
-1b-lyrics) architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more infor


JukeboxVQVAEConfig

class transformers.JukeboxVQVAEConfig  < source > ( act_fn = 'relu' nb_discrete_codes = 2048 commit = 0.02 conv_input_shape = 1 conv_res_scale = False embed_dim = 64 hop_fraction = [0.125, 0.5, 0.5] levels = 3 lmu = 0.99 multipliers = [2, 1, 1] res_conv_depth = 4 res_conv_width = 32 res_convolution_multiplier = 1 res_dilation_cycle = None res_dilation_growth_rate = 3 res_downs_t = [3, 2, 2] res_strides_t = [2, 2, 2] sample_length = 1058304 init_scale = 0.2 zero_out = False **kwargs  )   Parameters   act_fn (str, optional, defaults to "relu") —
Activation function of the model.   nb_discrete_codes (int, optional, defaults to 2048) —
Number of codes of the VQVAE.   commit (float, optional, defaults to 0.02) —
Commit loss multiplier.   conv_input_shape (int, optional, defaults to 1) —
Number of audio channels.   conv_res_scale (bool, optional, defaults to


JukeboxVQVAEConfig

 False) —
Whether or not to scale the residuals of the JukeboxResConv1DBlock.   embed_dim (int, optional, defaults to 64) —
Embedding dimension of the codebook vectors.   hop_fraction (List[int], optional, defaults to [0.125, 0.5, 0.5]) —
Fraction of non-intersecting window used when continuing the sampling process.   levels (int, optional, defaults to 3) —
Number of hierarchical levels that used in the VQVAE.   lmu (float, optional, defaults to 0.99) —
Used in the codebook update, exponential moving average coefficient. For more detail refer to Appendix A.1
of the original VQVAE paper   multipliers (List[int], optional, defaults to [2, 1, 1]) —
Depth and width multipliers used for each level. Used on the res_conv_width and res_conv_depth   res_conv_depth (int, optional, defaults to 4) —
Depth of the encoder and decoder block. If no multipliers are use


JukeboxVQVAEConfig

d, this is the same for each level.   res_conv_width (int, optional, defaults to 32) —
Width of the encoder and decoder block. If no multipliers are used, this is the same for each level.   res_convolution_multiplier (int, optional, defaults to 1) —
Scaling factor of the hidden dimension used in the JukeboxResConv1DBlock.   res_dilation_cycle (int, optional) —
Dilation cycle value used in the JukeboxResnet. If an int is used, each new Conv1 block will have a depth
reduced by a power of res_dilation_cycle.   res_dilation_growth_rate (int, optional, defaults to 3) —
Resnet dilation growth rate used in the VQVAE (dilation_growth_rate ** depth)   res_downs_t (List[int], optional, defaults to [3, 2, 2]) —
Downsampling rate for each level of the hierarchical VQ-VAE.   res_strides_t (List[int], optional, defaults to [2, 2, 2]) —
Stride used for each level of


JukeboxVQVAEConfig

the hierarchical VQ-VAE.   sample_length (int, optional, defaults to 1058304) —
Provides the max input shape of the VQVAE. Is used to compute the input shape of each level.   init_scale (float, optional, defaults to 0.2) —
Initialization scale.   zero_out (bool, optional, defaults to False) —
Whether or not to zero out convolution weights when initializing.    This is the configuration class to store the configuration of a JukeboxVQVAE. It is used to instantiate a
JukeboxVQVAE according to the specified arguments, defining the model architecture. Instantiating a configuration
with the defaults will yield a similar configuration to that of the VQVAE from
openai/jukebox-1b-lyrics architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information


JukeboxTokenizer

class transformers.JukeboxTokenizer  < source > ( artists_file genres_file lyrics_file version = ['v3', 'v2', 'v2'] max_n_lyric_tokens = 512 n_genres = 5 unk_token = '<|endoftext|>' **kwargs  )   Parameters   artists_file (str) —
Path to the vocabulary file which contains a mapping between artists and ids. The default file supports
both “v2” and “v3”   genres_file (str) —
Path to the vocabulary file which contain a mapping between genres and ids.   lyrics_file (str) —
Path to the vocabulary file which contains the accepted characters for the lyrics tokenization.   version (List[str], optional, default to ["v3", "v2", "v2"])  —
List of the tokenizer versions. The 5b-lyrics’s top level prior model was trained using v3 instead of
v2.   n_genres (int, optional, defaults to 1) —
Maximum numbe


JukeboxTokenizer

r of genres to use for composition.   max_n_lyric_tokens (int, optional, defaults to 512) —
Maximum number of lyric tokens to keep.   unk_token (str, optional, defaults to "<|endoftext|>") —
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.    Constructs a Jukebox tokenizer. Jukebox can be conditioned on 3 different inputs : Artists, unique ids are associated to each artist from the provided dictionary. Genres, unique ids are associated to each genre from the provided dictionary. Lyrics, character based tokenization. Must be initialized with the list of characters that are inside the
vocabulary. This tokenizer does not require training. It should be able to process a different number of inputs:
as the conditioning o


JukeboxTokenizer

f the model can be done on the three different queries. If None is provided, defaults values will be used.: Depending on the number of genres on which the model should be conditioned (n_genres).    Copied >>> from transformers import JukeboxTokenizer
>>> tokenizer = JukeboxTokenizer.from_pretrained("openai/jukebox-1b-lyrics")
>>> tokenizer("Alan Jackson", "Country Rock", "old town road")["input_ids"]
[tensor([[   0,    0,    0, 6785,  546,   41,   38,   30,   76,   46,   41,   49,
           40,   76,   44,   41,   27,   30]]), tensor([[  0,   0,   0, 145,   0]]), tensor([[  0,   0,   0, 145,   0]])] You can get around that behavior by passing add_prefix_space=True when instantiating this tokenizer or when you
call it on some text, but since the model was not pretrained this way, it migh


JukeboxTokenizer

t yield a decrease in performance. If nothing is provided, the genres and the artist will either be selected randomly or set to None This tokenizer inherits from PreTrainedTokenizer which contains most of the main methods. Users should refer to:
this superclass for more information regarding those methods. However the code does not allow that and only supports composing from various genres.  save_vocabulary  < source > ( save_directory: str filename_prefix: typing.Optional[str] = None  )   Parameters   save_directory (str) —
A path to the directory where to saved. It will be created if it doesn’t exist.   filename_prefix (Optional[str], optional) —
A prefix to add to the names of the files saved by the tokenizer.    Saves the tokenizer’s vocabulary dictionary to the provided save_directo


JukeboxModel

class transformers.JukeboxModel  < source > ( config  )   Parameters   config (JukeboxConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare JUKEBOX Model used for music generation. 4 sampling techniques are supported : primed_sample, upsample,
continue_sample and ancestral_sample. It does not have a forward method as the training is not end to end. If
you want to fine-tune the model, it is recommended to use the JukeboxPrior class and train each prior
individually. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use


JukeboxModel

 it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  ancestral_sample  < source > ( labels n_samples = 1 **sampling_kwargs  )   Parameters   labels (List[torch.LongTensor])  —
List of length n_sample, and shape (self.levels, 4 + self.config.max_nb_genre + lyric_sequence_length) metadata such as artist_id, genre_id and the full list of lyric tokens
which are used to condition the generation.   n_samples (int, optional, default to 1)  —
Number of samples to be generated in parallel.    Generates music tokens based on the provided labels. Will start at the desired prior level and automatically upsample the sequence. If you want to create the audio, you should call model.decode(tokens)`, which will use
the VQ-VAE decoder to convert the music tokens to raw audio.  Example:   Copied >>> from transformers import AutoTokenizer, JukeboxModel, set_seed
>>> model = JukeboxModel.from_pr


JukeboxModel

etrained("openai/jukebox-1b-lyrics", min_duration=0).eval()
>>> tokenizer = AutoTokenizer.from_pretrained("openai/jukebox-1b-lyrics")
>>> lyrics = "Hey, are you awake? Can you talk to me?"
>>> artist = "Zac Brown Band"
>>> genre = "Country"
>>> metas = tokenizer(artist=artist, genres=genre, lyrics=lyrics)
>>> set_seed(0)
>>> music_tokens = model.ancestral_sample(metas.input_ids, sample_length=400)
>>> with torch.no_grad():
...     model.decode(music_tokens)[:, :10].squeeze(-1)
tensor([[-0.0219, -0.0679, -0.1050, -0.1203, -0.1271, -0.0936, -0.0396, -0.0405,
    -0.0818, -0.0697]])  primed_sample  < source > ( raw_audio labels **sampling_kwargs  )   Parameters   raw_audio (List[torch.Tensor] of length n_samples )  —
A list of raw audio that will be used as conditioning information for each samples that will be
generated.   labels (List[torch.LongTensor] of length n_sample, and shape (self.levels, self.config.max_nb_genre + lyric_sequence_length)  —


JukeboxModel

List of metadata such as artist_id, genre_id and the full list of lyric tokens which are used to
condition the generation.   sampling_kwargs (Dict[Any]) —
Various additional sampling arguments that are used by the _sample function. A detail list of the
arguments can bee seen in the _sample function documentation.    Generate a raw audio conditioned on the provided raw_audio which is used as conditioning at each of the
generation levels. The audio is encoded to music tokens using the 3 levels of the VQ-VAE. These tokens are
used: as conditioning for each level, which means that no ancestral sampling is required.  continue_sample  < source > ( music_tokens labels **sampling_kwargs  )   Parameters   music_tokens (List[torch.LongTensor] of length self.levels )  —
A sequence of music tokens which will be used as context to continue the sampling process. Should have
self.levels tensors, each corresponding to the generation at a certain level.   labels


JukeboxModel

 (List[torch.LongTensor] of length n_sample, and shape (self.levels, self.config.max_nb_genre + lyric_sequence_length)  —
List of metadata such as artist_id, genre_id and the full list of lyric tokens which are used to
condition the generation.   sampling_kwargs (Dict[Any]) —
Various additional sampling arguments that are used by the _sample function. A detail list of the
arguments can bee seen in the _sample function documentation.    Generates a continuation of the previously generated tokens.  upsample  < source > ( music_tokens labels **sampling_kwargs  )   Parameters   music_tokens (List[torch.LongTensor] of length self.levels )  —
A sequence of music tokens which will be used as context to continue the sampling process. Should have
self.levels tensors, each corresponding to the generation at a certain level.   labels (List[torch.LongTensor] of length n_sample, and shape (self.levels, self.config.max_nb_genre + lyric_sequence_length)  —
List


JukeboxModel

 of metadata such as artist_id, genre_id and the full list of lyric tokens which are used to
condition the generation.   sampling_kwargs (Dict[Any]) —
Various additional sampling arguments that are used by the _sample function. A detail list of the
arguments can bee seen in the _sample function documentation.    Upsamples a sequence of music tokens using the prior at level level.  _sample  < source > ( music_tokens labels sample_levels metas = None chunk_size = 32 sampling_temperature = 0.98 lower_batch_size = 16 max_batch_size = 16 sample_length_in_seconds = 24 compute_alignments = False sample_tokens = None offset = 0 save_results = True sample_length = None  )   Parameters   music_tokens (List[torch.LongTensor]) —
A sequence of music tokens of length self.levels which will be used as context to continue the
sampling process. Should have self.levels tensors, each corresponding to the generation at a certain
level.   labels (List[torch.LongTenso


JukeboxModel

r]) —
List of length n_sample, and shape (self.levels, 4 + self.config.max_nb_genre + lyric_sequence_length) metadata such as artist_id, genre_id and the full list of lyric tokens
which are used to condition the generation.   sample_levels (List[int]) —
List of the desired levels at which the sampling will be done. A level is equivalent to the index of
the prior in the list of priors   metas (List[Any], optional) —
Metadatas used to generate the labels   chunk_size (int, optional, defaults to 32) —
Size of a chunk of audio, used to fill up the memory in chuncks to prevent OOM erros. Bigger chunks
means faster memory filling but more consumption.   sampling_temperature (float, optional, defaults to 0.98) —
Temperature used to ajust the randomness of the sampling.   lower_batch_size (int, optional, defaults to 16) —
Maximum batch size for the lower level priors   max_batch_size (int, optional, defaults to 16) —
Maximum batch size for the top level


JukeboxModel

priors   sample_length_in_seconds (int, optional, defaults to 24) —
Desired length of the generation in seconds   compute_alignments (bool, optional, defaults to False) —
Whether or not to compute the alignment between the lyrics and the audio using the top_prior   sample_tokens (int, optional) —
Precise number of tokens that should be sampled at each level. This is mostly useful for running dummy
experiments   offset (int, optional, defaults to 0) —
Audio offset used as conditioning, corresponds to the starting sample in the music. If the offset is
greater than 0, the lyrics will be shifted take that intoaccount   save_results (bool, optional, defaults to True) —
Whether or not to save the intermediate results. If True, will generate a folder named with the start
time.   sample_length (int, optional) —
Desired length of the generation in samples.    Core sampling function used to generate music tokens. Iterates over the provided list of levels,


JukeboxModel

while saving
the generated raw audio at each step. Returns: torch.Tensor  Example:   Copied >>> from transformers import AutoTokenizer, JukeboxModel, set_seed
>>> import torch
>>> metas = dict(artist="Zac Brown Band", genres="Country", lyrics="I met a traveller from an antique land")
>>> tokenizer = AutoTokenizer.from_pretrained("openai/jukebox-1b-lyrics")
>>> model = JukeboxModel.from_pretrained("openai/jukebox-1b-lyrics", min_duration=0).eval()
>>> labels = tokenizer(**metas)["input_ids"]
>>> set_seed(0)
>>> zs = [torch.zeros(1, 0, dtype=torch.long) for _ in range(3)]
>>> zs = model._sample(zs, labels, [0], sample_length=40 * model.priors[0].raw_to_tokens, save_results=False)
>>> zs[0]
tensor([[1853, 1369, 1150, 1869, 1379, 1789,  519,  710, 1306, 1100, 1229,  519,
      353, 1306, 1379, 1053,  519,  653, 1631, 1467, 1229, 1229,   10, 1647,
     1254, 1229, 1306, 1528, 1789,  216, 1631, 1434,  653,  475, 1150, 1528,
     1804,  541, 1804, 1434]


JukeboxPrior

class transformers.JukeboxPrior  < source > ( config: JukeboxPriorConfig level = None nb_priors = 3 vqvae_encoder = None vqvae_decoder = None  )   Parameters   config (JukeboxPriorConfig) —
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
from_pretrained() method to load the model weights.   level (int, optional) —
Current level of the Prior. Should be in range [0,nb_priors].   nb_priors (int, optional, defaults to 3) —
Total number of priors.   vqvae_encoder (Callable, optional) —
Encoding method of the VQVAE encoder used in the forward pass of the model. Passing functions instead of
the vqvae module to avoid getting the parameters.   vqvae_decoder (Callable, optional) —
Decoding method


JukeboxPrior

of the VQVAE decoder used in the forward pass of the model. Passing functions instead of
the vqvae module to avoid getting the parameters.    The JukeboxPrior class, which is a wrapper around the various conditioning and the transformer. JukeboxPrior can be
seen as language models trained on music. They model the next music token prediction task. If a (lyric) encoderù is defined, it also models the next character` prediction on the lyrics. Can be conditionned on timing, artist,
genre, lyrics and codes from lower-levels Priors.  sample  < source > ( n_samples music_tokens = None music_tokens_conds = None metadata = None temp = 1.0 top_k = 0 top_p = 0.0 chunk_size = None sample_tokens = None  )   Parameters   n_samples (int) —
Number of samples to generate.   music_tokens (List[torch.LongTensor], optional) —
Previously gemera


JukeboxPrior

ted tokens at the current level. Used as context for the generation.   music_tokens_conds (List[torch.FloatTensor], optional) —
Upper-level music tokens generated by the previous prior model. Is None if the generation is not
conditionned on the upper-level tokens.   metadata (List[torch.LongTensor], optional) —
List containing the metatdata tensor with the artist, genre and the lyric tokens.   temp (float, optional, defaults to 1.0) —
Sampling temperature.   top_k (int, optional, defaults to 0) —
Top k probabilities used for filtering.   top_p (float, optional, defaults to 0.0) —
Top p probabilities used for filtering.   chunk_size (int, optional) —
Size of the chunks used to prepare the cache of the transformer.   sample_tokens (int, optional) —
Number of tokens to sample.    Ancestral/Prime sampling a window of tokens usi


JukeboxPrior

ng the provided conditioning and metadatas.  forward  < source > ( hidden_states: Tensor metadata: typing.Optional[typing.List[torch.LongTensor]] decode: typing.Optional[bool] = False get_preds: typing.Optional[bool] = False  )   Parameters   hidden_states (torch.Tensor) —
Hidden states which should be raw audio   metadata (List[torch.LongTensor], optional) —
List containing the metadata conditioning tensorwith the lyric and the metadata tokens.   decode (bool, optional, defaults to False) —
Whether or not to decode the encoded to tokens.   get_preds (bool, optional, defaults to False) —
Whether or not to return the actual predicitons of the model.    Encode the hidden states using the vqvae encoder, and then predicts the next token in the forward_tokens
function. The loss is the sum of the encoder loss and the decoder loss


JukeboxVQVAE

class transformers.JukeboxVQVAE  < source > ( config: JukeboxVQVAEConfig  )   Parameters   config (JukeboxConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The Hierarchical VQ-VAE model used in Jukebox. This model follows the Hierarchical VQVAE paper from Will Williams, Sam
Ringer, Tom Ash, John Hughes, David MacLeod, Jamie Dougherty. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source >


JukeboxVQVAE

( raw_audio: FloatTensor  )   Parameters   raw_audio (torch.FloatTensor) —
Audio input which will be encoded and decoded.    Forward pass of the VQ-VAE, encodes the raw_audio to latent states, which are then decoded for each level.
The commit loss, which ensure that the encoder’s computed embeddings are close to the codebook vectors, is
computed.  Example:   Copied >>> from transformers import JukeboxVQVAE, set_seed
>>> import torch
>>> model = JukeboxVQVAE.from_pretrained("openai/jukebox-1b-lyrics").eval()
>>> set_seed(0)
>>> zs = [torch.randint(100, (4, 1))]
>>> model.decode(zs).shape
torch.Size([4, 8, 1])  encode  < source > ( input_audio start_level = 0 end_level = None bs_chunks = 1  )   Parameters   input_audio (torch.Tensor) —
Raw audio which will be encoded to its discrete representation using the codebook. The closest code
form the codebook will be computed for each sequence of samples.   start_level (int, optional, defaults to 0) —
Level at whi


JukeboxVQVAE

ch the encoding process will start. Default to 0.   end_level (int, optional) —
Level at which the encoding process will start. Default to None.   bs_chunks (int, optional, defaults to 1) —
Number of chunks of raw audio to process at the same time.    Transforms the input_audio to a discrete representation made out of music_tokens.  decode  < source > ( music_tokens start_level = 0 end_level = None bs_chunks = 1  )   Parameters   music_tokens (torch.LongTensor) —
Tensor of music tokens which will be decoded to raw audio by using the codebook. Each music token
should be an index to a corresponding code vector in the codebook.   start_level (int, optional) —
Level at which the decoding process will start. Default to 0.   end_level (int, optional) —
Level at which the decoding process will start. Default to None.   bs_chunks (int, optional) —
Number of chunks to process at the same time.    Transforms the input music_tokens to their raw_audio representation


Overview

The Speech2Text2 model is used together with Wav2Vec2 for Speech Translation models proposed in
Large-Scale Self- and Semi-Supervised Learning for Speech Translation by
Changhan Wang, Anne Wu, Juan Pino, Alexei Baevski, Michael Auli, Alexis Conneau.
Speech2Text2 is a decoder-only transformer model that can be used with any speech encoder-only, such as
Wav2Vec2 or HuBERT for Speech-to-Text tasks. Please refer to the
SpeechEncoderDecoder class on how to combine Speech2Text2 with any speech encoder-only
model.
This model was contributed by Patrick von Platen.
The original code can be found here.
Tips:
Speech2Text2 achieves state-of-the-art results on the CoVoST Speech Translation dataset. For more information, see
the official models . Speech2Text2 is always used within the SpeechEncoderDecoder framework. Speech2Text2’s tokenizer is based on fastBPE.


Inference

Speech2Text2’s SpeechEncoderDecoderModel model accepts raw waveform input values from speech and
makes use of generate() to translate the input speech
autoregressively to the target language.
The Wav2Vec2FeatureExtractor class is responsible for preprocessing the input speech and
Speech2Text2Tokenizer decodes the generated target tokens to the target string. The
Speech2Text2Processor wraps Wav2Vec2FeatureExtractor and
Speech2Text2Tokenizer into a single instance to both extract the input features and decode the
predicted token ids.
Step-by-step Speech Translation
  Copied >>> import torch
>>> from transformers import Speech2Text2Processor, SpeechEncoderDecoderModel
>>> fr


Inference

om datasets import load_dataset
>>> import soundfile as sf
>>> model = SpeechEncoderDecoderModel.from_pretrained("facebook/s2t-wav2vec2-large-en-de")
>>> processor = Speech2Text2Processor.from_pretrained("facebook/s2t-wav2vec2-large-en-de")
>>> def map_to_array(batch):
...     speech, _ = sf.read(batch["file"])
...     batch["speech"] = speech
...     return batch
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> ds = ds.map(map_to_array)
>>> inputs = processor(ds["speech"][0], sampling_rate=16_000, return_tensors="pt")
>>> generated_ids = model.generate(inputs=inputs["input_values"], attention_mask=inputs["attention_mask"


Inference

])
>>> transcription = processor.batch_decode(generated_ids)
Speech Translation via Pipelines The automatic speech recognition pipeline can also be used to translate speech in just a couple lines of code
  Copied >>> from datasets import load_dataset
>>> from transformers import pipeline
>>> librispeech_en = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> asr = pipeline(
...     "automatic-speech-recognition",
...     model="facebook/s2t-wav2vec2-large-en-de",
...     feature_extractor="facebook/s2t-wav2vec2-large-en-de",
... )
>>> translation_de = asr(librispeech_en[0]["file"])
See model hub to look for Speech2Text2 checkpoints.


Speech2Text2Config

class transformers.Speech2Text2Config  < source > ( vocab_size = 10000 decoder_layers = 6 decoder_ffn_dim = 2048 decoder_attention_heads = 4 decoder_layerdrop = 0.0 use_cache = True activation_function = 'relu' d_model = 256 dropout = 0.1 attention_dropout = 0.0 activation_dropout = 0.0 init_std = 0.02 decoder_start_token_id = 2 scale_embedding = True pad_token_id = 1 bos_token_id = 0 eos_token_id = 2 max_target_positions = 1024 **kwargs  )   Parameters   vocab_size (int, optional, defaults to 50265) —
Vocabulary size of the Speech2Text model. Defines the number of different tokens that can be represented by
the inputs_ids passed when calling Speech2TextModel   d_model (int, optional, defaults to 1024) —
Dimensionality of the layers and the pooler layer.   decoder_layers (int, optional, defaults to 12) —
Number of decoder layers.   de


Speech2Text2Config

coder_attention_heads (int, optional, defaults to 16) —
Number of attention heads for each attention layer in the Transformer decoder.   decoder_ffn_dim (int, optional, defaults to 4096) —
Dimensionality of the “intermediate” (often named feed-forward) layer in decoder.   activation_function (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the pooler. If string, "gelu", "relu",
"silu" and "gelu_new" are supported.   dropout (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the embeddings, and pooler.   attention_dropout (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.   activation_dropout (float, optional, defaults to 0.0) —
The dropout ratio for activations inside the fully connected layer.


Speech2Text2Config

 init_std (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
https://arxiv.org/abs/1909.11556>`__ for more details.   decoder_layerdrop (float, optional, defaults to 0.0) —
The LayerDrop probability for the decoder. See the [LayerDrop paper](see https://arxiv.org/abs/1909.11556)
for more details.   use_cache (bool, optional, defaults to True) —
Whether or not the model should return the last key/values attentions (not used by all models).   max_target_positions (int, optional, defaults to 1024) —
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).    This is the configuration class to store the configuration of a Speech2Text2ForCausalLM. It is used to
insta


Speech2Text2Config

ntiate an Speech2Text2 model according to the specified arguments, defining the model architecture.
Instantiating a configuration with the defaults will yield a similar configuration to that of the Speech2Text2
facebook/s2t-wav2vec2-large-en-de architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import Speech2Text2Config, Speech2Text2ForCausalLM
>>> # Initializing a Speech2Text2 s2t_transformer_s style configuration
>>> configuration = Speech2Text2Config()
>>> # Initializing a model (with random weights) from the s2t_transformer_s style configuration
>>> model = Speech2Text2ForCausalLM(configuration)
>>> # Accessing the model configuration
>>> configuration = model.conf


class transformers.Speech2Text2Tokenizer  < source > ( vocab_file bos_token = '<s>' pad_token = '<pad>' eos_token = '</s>' unk_token = '<unk>' do_lower_case = False merges_file = None **kwargs  )   Parameters   vocab_file (str) —
File containing the vocabulary.   bos_token (str, optional, defaults to "<s>") —
The beginning of sentence token.   eos_token (str, optional, defaults to "</s>") —
The end of sentence token.   unk_token (str, optional, defaults to "<unk>") —
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.   pad_token (str, optional, defaults to "<pad>") —
The token used for padding, for example when batching sequences of different lengths.
**kwargs —
Additional keyword arguments passed along to PreT


rainedTokenizer    Constructs a Speech2Text2Tokenizer. This tokenizer inherits from PreTrainedTokenizer which contains some of the main methods. Users should refer to
the superclass for more information regarding such methods.  batch_decode  < source > ( sequences: typing.Union[typing.List[int], typing.List[typing.List[int]], ForwardRef('np.ndarray'), ForwardRef('torch.Tensor'), ForwardRef('tf.Tensor')] skip_special_tokens: bool = False clean_up_tokenization_spaces: bool = None **kwargs  ) → List[str]  Parameters   sequences (Union[List[int], List[List[int]], np.ndarray, torch.Tensor, tf.Tensor]) —
List of tokenized input ids. Can be obtained using the __call__ method.   skip_special_tokens (bool, optional, defaults to False) —
Whether or not to remove special tokens in the decodin


g.   clean_up_tokenization_spaces (bool, optional) —
Whether or not to clean up the tokenization spaces. If None, will default to
self.clean_up_tokenization_spaces.   kwargs (additional keyword arguments, optional) —
Will be passed to the underlying model specific decode method.   Returns
List[str]

The list of decoded sentences.
  Convert a list of lists of token ids into a list of strings by calling decode.  decode  < source > ( token_ids: typing.Union[int, typing.List[int], ForwardRef('np.ndarray'), ForwardRef('torch.Tensor'), ForwardRef('tf.Tensor')] skip_special_tokens: bool = False clean_up_tokenization_spaces: bool = None **kwargs  ) → str  Parameters   token_ids (Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]) —
List of tokenized input ids. Can be obtained us


ing the __call__ method.   skip_special_tokens (bool, optional, defaults to False) —
Whether or not to remove special tokens in the decoding.   clean_up_tokenization_spaces (bool, optional) —
Whether or not to clean up the tokenization spaces. If None, will default to
self.clean_up_tokenization_spaces.   kwargs (additional keyword arguments, optional) —
Will be passed to the underlying model specific decode method.   Returns
str

The decoded sentence.
  Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special
tokens and clean up tokenization spaces. Similar to doing self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids)).  save_vocabulary  < source > ( save_directory: str filename_prefix: typing.Optional[str] = None


class transformers.Speech2Text2Processor  < source > ( feature_extractor tokenizer  )   Parameters   feature_extractor (AutoFeatureExtractor) —
An instance of AutoFeatureExtractor. The feature extractor is a required input.   tokenizer (Speech2Text2Tokenizer) —
An instance of Speech2Text2Tokenizer. The tokenizer is a required input.    Constructs a Speech2Text2 processor which wraps a Speech2Text2 feature extractor and a Speech2Text2 tokenizer into
a single processor. Speech2Text2Processor offers all the functionalities of AutoFeatureExtractor and Speech2Text2Tokenizer.
See the call() and decode() for more information.  __call__  < source > ( *args **kwargs  )    When used in normal mode, this method forwards all its arguments to AutoFeatureExtractor’s
__call__() and returns its output. If used in the context
as_target_processor() this method forwards all its arguments to
Speech2Text2Tokenizer’s call(). Please r


efer to the doctsring of the above two
methods for more information.  from_pretrained  < source > ( pretrained_model_name_or_path: typing.Union[str, os.PathLike] cache_dir: typing.Union[str, os.PathLike, NoneType] = None force_download: bool = False local_files_only: bool = False token: typing.Union[bool, str, NoneType] = None revision: str = 'main' **kwargs  )   Parameters   pretrained_model_name_or_path (str or os.PathLike) —
This can be either:
a string, the model id of a pretrained feature_extractor hosted inside a model repo on
huggingface.co. Valid model ids can be located at the root-level, like bert-base-uncased, or
namespaced under a user or organization name, like dbmdz/bert-base-german-cased.
a path to a directory containing a feature extractor file saved using the
save_pretrained() method, e.g., ./my_model_directory/.
a path or url to a saved feature extractor JSON file, e.g.,
./my_model_directory/pr


eprocessor_config.json.
**kwargs —
Additional keyword arguments passed along to both
from_pretrained() and
~tokenization_utils_base.PreTrainedTokenizer.from_pretrained.
    Instantiate a processor associated with a pretrained model. This class method is simply calling the feature extractor
from_pretrained(), image processor
ImageProcessingMixin and the tokenizer
~tokenization_utils_base.PreTrainedTokenizer.from_pretrained methods. Please refer to the docstrings of the
methods above for more information.  save_pretrained  < source > ( save_directory push_to_hub: bool = False **kwargs  )   Parameters   save_directory (str or os.PathLike) —
Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will
be created if it does not exist).   push_to_hub (bool, optional, defaults to False) —
Whether or not to push your model to the Hugging Face model hub after saving it. You can sp


ecify the
repository you want to push to with repo_id (will default to the name of save_directory in your
namespace).   kwargs (Dict[str, Any], optional) —
Additional key word arguments passed along to the push_to_hub() method.    Saves the attributes of this processor (feature extractor, tokenizer…) in the specified directory so that it
can be reloaded using the from_pretrained() method. This class method is simply calling save_pretrained() and
save_pretrained(). Please refer to the docstrings of the
methods above for more information.  batch_decode  < source > ( *args **kwargs  )    This method forwards all its arguments to Speech2Text2Tokenizer’s batch_decode(). Please
refer to the docstring of this method for more information.  decode  < source > ( *args **kwargs  )    This method forwards all its arguments to Speech2Text2Tokenizer’s decode(). Please refer
to the docstring of this method for more information


class transformers.Speech2Text2ForCausalLM  < source > ( config  )   Parameters   config (Speech2Text2Config) —
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
from_pretrained() method to load the model weights.    The Speech2Text2 Decoder with a language modeling head. Can be used as the decoder part of EncoderDecoderModel and SpeechEncoderDecoder.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward


< source > ( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.Tensor] = None encoder_hidden_states: typing.Optional[torch.FloatTensor] = None encoder_attention_mask: typing.Optional[torch.FloatTensor] = None head_mask: typing.Optional[torch.Tensor] = None cross_attn_head_mask: typing.Optional[torch.Tensor] = None past_key_values: typing.Optional[typing.Tuple[typing.Tuple[torch.FloatTensor]]] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.LongTensor] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.CausalLMOutputWithCrossAttentions or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length))


—
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you
provide it.
Indices can be obtained using Speech2Text2Tokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   encoder_hidden_states  (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention
if the model is configured as a decoder.   encoder_attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attentio


n on the padding token indices of the encoder input. This mask is used
in the cross-attention if the model is configured as a decoder. Mask values selected in [0, 1]:   head_mask (torch.Tensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   cross_attn_head_mask (torch.Tensor of shape (decoder_layers, decoder_attention_heads), optional) —
Mask to nullify selected heads of the cross-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) —
Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors o


f
shape (batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of
shape (batch_size, num_heads, encoder_sequence_length, embed_size_per_head). The two additional
tensors are only required when the model is used as a decoder in a Sequence to Sequence model.
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the
cross-attention blocks) that can be used (see past_key_values input) to speed up sequential decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those
that don’t have their past key value states given to this model) of shape (batch_size, 1) instead of
all decoder_input_ids of shape (batch_size, sequence_length).   labels (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Labels for computing the masked language modeling loss. Indices should either be in [


0, ..., config.vocab_size] or -100 (see input_ids docstring). Tokens with indices set to -100 are ignored
(masked), the loss is only computed for the tokens with labels in [0, ..., config.vocab_size].   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding
(see past_key_values).
1 for tokens that are not masked,
0 for tokens that are masked.
   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under
returned tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors
for more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.CausalLMOutputWithCrossAtt


entions or tuple(torch.FloatTensor)

A transformers.modeling_outputs.CausalLMOutputWithCrossAttentions or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (Speech2Text2Config) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Language modeling loss (for next-token prediction).
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shap


e (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Cross attentions weights after the attention softmax, used to compute the weighted average in the
cross-attention heads.
past_key_values (t


uple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of torch.FloatTensor tuples of length config.n_layers, with each tuple containing the cached key,
value states of the self-attention and the cross-attention layers if model is used in encoder-decoder
setting. Only relevant if config.is_decoder = True.
Contains pre-computed hidden-states (key and values in the attention blocks) that can be used (see
past_key_values input) to speed up sequential decoding.
   Example:   Copied >>> from transformers import (
...     SpeechEncoderDecoderModel,
...     Speech2Text2ForCausalLM,
...     Wav2Vec2Model,
...     Speech2Text2Config,
...     Wav2Vec2Config,
...     Wav2Vec2FeatureExtractor,
...     Speech2Text2Tokenizer,
... )
>>> from datasets import load_dataset
>>> feature_extractor = Wav2Vec2FeatureExtractor()
>>> tokenizer = Speech2Text


2Tokenizer.from_pretrained("facebook/s2t-wav2vec2-large-en-de")
>>> encoder = Wav2Vec2Model(Wav2Vec2Config())
>>> decoder = Speech2Text2ForCausalLM(Speech2Text2Config())
>>> # init random speech2text model
>>> model = SpeechEncoderDecoderModel(encoder=encoder, decoder=decoder)
>>> model.config.pad_token_id = tokenizer.pad_token_id
>>> model.config.decoder_start_token_id = tokenizer.bos_token_id
>>> # pre-process inputs and labels
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> inputs = feature_extractor(
...     ds[0]["audio"]["array"], sampling_rate=ds[0]["audio"]["sampling_rate"], return_tensors="pt"
... )
>>> input_values = inputs.input_values
>>> decoder_input_ids = tokenizer(ds[0]["text"], return_tensors="pt").input_ids
>>> # compute loss
>>> loss = model(inputs=input_values, labels=decoder_input_ids).loss
>>> # backprop loss
>>> loss.backw


Overview

The CLIPSeg model was proposed in Image Segmentation Using Text and Image Prompts by Timo Lüddecke
and Alexander Ecker. CLIPSeg adds a minimal decoder on top of a frozen CLIP model for zero- and one-shot image segmentation.
The abstract from the paper is the following:
Image segmentation is usually addressed by training a
model for a fixed set of object classes. Incorporating additional classes or more complex queries later is expensive
as it requires re-training the model on a dataset that encompasses these expressions. Here we propose a system
that can generate image segmentations based on arbitrary
prompts at test time. A prompt can be either a text or an
image. This approach enables us to create a unified mo


Overview

del
(trained once) for three common segmentation tasks, which
come with distinct challenges: referring expression segmentation, zero-shot segmentation and one-shot segmentation.
We build upon the CLIP model as a backbone which we extend with a transformer-based decoder that enables dense
prediction. After training on an extended version of the
PhraseCut dataset, our system generates a binary segmentation map for an image based on a free-text prompt or on
an additional image expressing the query. We analyze different variants of the latter image-based prompts in detail.
This novel hybrid input allows for dynamic adaptation not
only to the three segmentation tasks mentioned above, but
to any binary segmentation ta


Overview

sk where a text or image query
can be formulated. Finally, we find our system to adapt well
to generalized queries involving affordances or properties
Tips:
CLIPSegForImageSegmentation adds a decoder on top of CLIPSegModel. The latter is identical to CLIPModel. CLIPSegForImageSegmentation can generate image segmentations based on arbitrary prompts at test time. A prompt can be either a text
(provided to the model as input_ids) or an image (provided to the model as conditional_pixel_values). One can also provide custom
conditional embeddings (provided to the model as conditional_embeddings).
CLIPSeg overview. Taken from the original paper.
This model was contributed by nielsr.
The original code can be found here.


Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with CLIPSeg. If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we’ll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.
 Image Segmentation
A notebook that illustrates zero-shot image segmentation with CLIPSeg.


CLIPSegConfig

class transformers.CLIPSegConfig  < source > ( text_config = None vision_config = None projection_dim = 512 logit_scale_init_value = 2.6592 extract_layers = [3, 6, 9] reduce_dim = 64 decoder_num_attention_heads = 4 decoder_attention_dropout = 0.0 decoder_hidden_act = 'quick_gelu' decoder_intermediate_size = 2048 conditional_layer = 0 use_complex_transposed_convolution = False **kwargs  )   Parameters   text_config (dict, optional) —
Dictionary of configuration options used to initialize CLIPSegTextConfig.   vision_config (dict, optional) —
Dictionary of configuration options used to initialize CLIPSegVisionConfig.   projection_dim (int, optional, defaults to 512) —
Dimensionality of text and vision projection layers.   logit_scale_init_value (float, optional, defaults to 2.6592) —
The inital value of the logit_scale paramter. Default is used as per the original CLIPSeg implementation.   extract_layers (List[int], optional, defaults to [3, 6, 9])


CLIPSegConfig

 —
Layers to extract when forwarding the query image through the frozen visual backbone of CLIP.   reduce_dim (int, optional, defaults to 64) —
Dimensionality to reduce the CLIP vision embedding.   decoder_num_attention_heads (int, optional, defaults to 4) —
Number of attention heads in the decoder of CLIPSeg.   decoder_attention_dropout (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.   decoder_hidden_act (str or function, optional, defaults to "quick_gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "selu" and "gelu_new" `"quick_gelu" are supported.   decoder_intermediate_size (int, optional, defaults to 2048) —
Dimensionality of the “intermediate” (i.e., feed-forward) layers in the Transformer decoder.   conditional_layer (int, optional, defaults to 0) —
The layer to use of the Transformer encoder whose activations will be combined with


CLIPSegConfig

 the condition
embeddings using FiLM (Feature-wise Linear Modulation). If 0, the last layer is used.   use_complex_transposed_convolution (bool, optional, defaults to False) —
Whether to use a more complex transposed convolution in the decoder, enabling more fine-grained
segmentation.   kwargs (optional) —
Dictionary of keyword arguments.    CLIPSegConfig is the configuration class to store the configuration of a CLIPSegModel. It is used to
instantiate a CLIPSeg model according to the specified arguments, defining the text model and vision model configs.
Instantiating a configuration with the defaults will yield a similar configuration to that of the CLIPSeg
CIDAS/clipseg-rd64 architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import CLIPSegConfig, CLIPSegModel
>>> # Initializing


CLIPSegConfig

 a CLIPSegConfig with CIDAS/clipseg-rd64 style configuration
>>> configuration = CLIPSegConfig()
>>> # Initializing a CLIPSegModel (with random weights) from the CIDAS/clipseg-rd64 style configuration
>>> model = CLIPSegModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config
>>> # We can also initialize a CLIPSegConfig from a CLIPSegTextConfig and a CLIPSegVisionConfig
>>> # Initializing a CLIPSegText and CLIPSegVision configuration
>>> config_text = CLIPSegTextConfig()
>>> config_vision = CLIPSegVisionConfig()
>>> config = CLIPSegConfig.from_text_vision_configs(config_text, config_vision)  from_text_vision_configs  < source > ( text_config: CLIPSegTextConfig vision_config: CLIPSegVisionConfig **kwargs  ) → CLIPSegConfig  Returns
CLIPSegConfig

An instance of a configuration object
  Instantiate a CLIPSegConfig (or a derived class) from clipseg text model configuration and clipseg vision
model configurati


CLIPSegTextConfig

class transformers.CLIPSegTextConfig  < source > ( vocab_size = 49408 hidden_size = 512 intermediate_size = 2048 num_hidden_layers = 12 num_attention_heads = 8 max_position_embeddings = 77 hidden_act = 'quick_gelu' layer_norm_eps = 1e-05 attention_dropout = 0.0 initializer_range = 0.02 initializer_factor = 1.0 pad_token_id = 1 bos_token_id = 49406 eos_token_id = 49407 **kwargs  )   Parameters   vocab_size (int, optional, defaults to 49408) —
Vocabulary size of the CLIPSeg text model. Defines the number of different tokens that can be represented
by the inputs_ids passed when calling CLIPSegModel.   hidden_size (int, optional, defaults to 512) —
Dimensionality of the encoder layers and the pooler layer.   intermediate_size (int, optional, defa