PyTorch

ores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.EpsilonLogitsWarper  < source > ( epsilon: float filter_value: float = -inf min_tokens_to_keep: int = 1  )   Parameters   epsilon (float) —
If set to > 0, only the most tokens with probabilities epsilon or higher are kept for generation.   filter_value (float, optional, defaults to -inf) —
All filtered values will be set to this float value.   min_tokens_to_keep (int, optional, defaults to 1) —
Minimum number of tokens that cannot be filtered.    LogitsWarper that performs epsilon-sampling, i.e. restricting to tokens with prob >= epsilon. Takes the
largest min_tokens_to_keep tokens if no tokens satisfy this constraint. See Truncation Sampling as Language Model
Desmoothing for more information.  E


PyTorch

xamples:   Copied >>> from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed
>>> set_seed(0)
>>> model = AutoModelForCausalLM.from_pretrained("distilgpt2")
>>> tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
>>> inputs = tokenizer("A sequence: 1, 2", return_tensors="pt")
>>> # With sampling, the output is unexpected -- sometimes too unexpected.
>>> outputs = model.generate(**inputs, do_sample=True)
>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True)[0])
A sequence: 1, 2, 0, 2, 2. 2, 2, 2, 2
>>> # With epsilon sampling, the output gets restricted to high-probability tokens. Note that this is similar to
>>> # Top P sampling, which restricts tokens based on their cumulative probability.
>>> # Pro tip: The paper recomends using `epsilon_cutoff` values between 3e-4 and 9e-4
>>> outputs = model.generate(**inputs, do_sample=True, epsilon_cutoff=0.1)
>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True)[0])
A sequence: 1, 2, 3, 4, 5, 6, 7


PyTorch

, 8, 9  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.EtaLogitsWarper  < source > ( epsilon: float filter_value: float = -inf min_tokens_to_keep: int = 1  )   Parameters   epsilon (float) —
A float value in the range (0, 1). Hyperparameter used to calculate the dynamic cutoff value, eta. The
suggested values from the paper ranges from 3e-4 to 4e-3 depending on the size of the mo


PyTorch

del.   filter_value (float, optional, defaults to -inf) —
All values that are found to be below the dynamic cutoff value, eta, are set to this float value. This
parameter is useful when logits need to be modified for very low probability tokens that should be excluded
from generation entirely.   min_tokens_to_keep (int, optional, defaults to 1) —
Specifies the minimum number of tokens that must be kept for generation, regardless of their probabilities.
For example, if min_tokens_to_keep is set to 1, at least one token will always be kept for generation,
even if all tokens have probabilities below the cutoff eta.    LogitsWarper that performs eta-sampling, a technique to filter out tokens with probabilities below a dynamic
cutoff value, eta, which is calculated based on a combination of the hyperparameter epsilon and the entropy of
the token probabilities, i.e. eta := min(epsilon, sqrt(epsilon * e^-entropy(probabilities))). Takes the largest
min_tokens_to_keep tokens if no tokens sati


PyTorch

sfy this constraint. It addresses the issue of poor quality in long
samples of text generated by neural language models leading to more coherent and fluent text. See Truncation
Sampling as Language Model Desmoothing for more information. Note: do_sample
must be set to True for this LogitsWarper to work.  Examples:   Copied >>> from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed
>>> set_seed(0)
>>> model = AutoModelForCausalLM.from_pretrained("distilgpt2")
>>> tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
>>> inputs = tokenizer("A sequence: 1, 2", return_tensors="pt")
>>> # With sampling, the output is unexpected -- sometimes too unexpected.
>>> outputs = model.generate(**inputs, do_sample=True)
>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True)[0])
A sequence: 1, 2, 0, 2, 2. 2, 2, 2, 2
>>> # With eta sampling, the output gets restricted to high-probability tokens. You can see it as a dynamic form of
>>> # epsilon sampling that adapts its


PyTorch

cutoff probability based on the entropy (high entropy = lower cutoff).
>>> # Pro tip: The paper recomends using `eta_cutoff` values between 3e-4 to 4e-3
>>> outputs = model.generate(**inputs, do_sample=True, eta_cutoff=0.1)
>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True)[0])
A sequence: 1, 2, 3, 4, 5, 6, 7, 8, 9  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.


PyTorch

ExponentialDecayLengthPenalty  < source > ( exponential_decay_length_penalty: typing.Tuple[int, float] eos_token_id: typing.Union[int, typing.List[int]] input_ids_seq_length: int  )   Parameters   exponential_decay_length_penalty (tuple(int, float)) —
This tuple shall consist of: (start_index, decay_factor) where start_index indicates where penalty
starts and decay_factor represents the factor of exponential decay   eos_token_id (Union[int, List[int]]) —
The id of the end-of-sequence token. Optionally, use a list to set multiple end-of-sequence tokens.   input_ids_seq_length (int) —
The length of the input sequence.    LogitsProcessor that exponentially increases the score of the eos_token_id after start_index has been
reached. This allows generating shorter sequences without having a hard cutoff, allowing the eos_token to be
predicted in a meaningful position.  Examples:   Copied >>> from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed
>>> set_seed(1)
>>> model = A


PyTorch

utoModelForCausalLM.from_pretrained("gpt2")
>>> tokenizer = AutoTokenizer.from_pretrained("gpt2")
>>> text = "Just wanted to let you know, I"
>>> inputs = tokenizer(text, return_tensors="pt")
>>> # Generate sequences without exponential penalty. We want short sentences, so we limit max_length=30
>>> # see that the answer tends to end abruptly
>>> outputs = model.generate(**inputs, do_sample=True, temperature=0.9, max_length=30, pad_token_id=50256)
>>> print(tokenizer.batch_decode(outputs)[0])
Just wanted to let you know, I'm not even a lawyer. I'm a man. I have no real knowledge of politics. I'm a
>>> # Generate sequences with exponential penalty, we add the exponential_decay_length_penalty=(start_index, decay_factor)
>>> # We see that instead of cutting at max_tokens, the output comes to an end before (at 25 tokens) and with more meaning
>>> # What happens is that starting from `start_index` the EOS token score will be increased by decay_factor exponentially
>>> outputs = model.gene


PyTorch

rate(
...     **inputs,
...     do_sample=True,
...     temperature=0.9,
...     max_length=30,
...     pad_token_id=50256,
...     exponential_decay_length_penalty=(15, 1.6),
... )
>>> print(tokenizer.batch_decode(outputs)[0])
Just wanted to let you know, I've got a very cool t-shirt educating people on how to use the Internet<|endoftext|>
>>> # Generate sequences with smaller decay_factor, still improving the hard cutoff mid-sentence
>>> outputs = model.generate(
...     **inputs,
...     do_sample=True,
...     temperature=0.9,
...     max_length=30,
...     pad_token_id=50256,
...     exponential_decay_length_penalty=(15, 1.05),
... )
>>> print(tokenizer.batch_decode(outputs)[0])
Just wanted to let you know, I've been working on it for about 6 months and now it's in Alpha.<|endoftext|>  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, seq


PyTorch

uence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.ForcedBOSTokenLogitsProcessor  < source > ( bos_token_id: int  )   Parameters   bos_token_id (int) —
The id of the token to force as the first generated token.    LogitsProcessor that enforces the specified token as the first generated token.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?


PyTorch

scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.ForcedEOSTokenLogitsProcessor  < source > ( max_length: int eos_token_id: typing.Union[int, typing.List[int]]  )   Parameters   max_length (int) —
The maximum length of the sequence to be generated.   eos_token_id (Union[int, List[int]]) —
The id of the token to force as the last generated token when max_length is reached. Optionally, use a
list to set multiple end-of-sequence tokens.    LogitsProcessor that enforces the specified token as the last generated token when max_length is reached.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_s


PyTorch

ize)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.ForceTokensLogitsProcessor  < source > ( force_token_map: typing.List[typing.List[int]]  )    This processor takes a list of pairs of integers which indicates a mapping from generation indices to token
indices that will be forced before sampling. The processor will set their log probs to inf so that they are
sampled at their corresponding index.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size,


PyTorch

config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.HammingDiversityLogitsProcessor  < source > ( diversity_penalty: float num_beams: int num_beam_groups: int  )   Parameters   diversity_penalty (float) —
This value is subtracted from a beam’s score if it generates a token same as any beam from other group at a
particular time. Note that diversity_penalty is only effective if group beam search is enabled. The
penalty applied to a beam’s score when it generates a token that has already been ch


PyTorch

osen by another beam
within the same group during the same time step. A higher diversity_penalty will enforce greater
diversity among the beams, making it less likely for multiple beams to choose the same token. Conversely, a
lower penalty will allow beams to more freely choose similar tokens. Adjusting this value can help strike a
balance between diversity and natural likelihood.   num_beams (int) —
Number of beams used for group beam search. Beam search is a method used that maintains beams (or “multiple
hypotheses”) at each step, expanding each one and keeping the top-scoring sequences. A higher num_beams
will explore more potential sequences. This can increase chances of finding a high-quality output but also
increases computational cost.   num_beam_groups (int) —
Number of groups to divide num_beams into in order to ensure diversity among different groups of beams.
Each group of beams will operate independently, selecting tokens without considering the choices of other
groups. T


PyTorch

his division promotes diversity by ensuring that beams within different groups explore different
paths. For instance, if num_beams is 6 and num_beam_groups is 2, there will be 2 groups each containing
3 beams. The choice of num_beam_groups should be made considering the desired level of output diversity
and the total number of beams. See this paper for more details.    LogitsProcessor that enforces diverse beam search. Note that this logits processor is only effective for PreTrainedModel.group_beam_search(). See Diverse Beam
Search: Decoding Diverse Solutions from Neural Sequence Models for more
details. Diverse beam search can be particularly useful in scenarios where a variety of different outputs is desired, rather
than multiple similar sequences. It allows the model to explore different generation paths and provides a broader
coverage of possible outputs. This logits processor can be resource-intensive, especially when using large models or long sequences. Traditional beam search


PyTorch

 often generates very similar sequences across different beams.
HammingDiversityLogitsProcessor addresses this by penalizing beams that generate tokens already chosen by other
beams in the same time step. How It Works: Grouping Beams: Beams are divided into groups. Each group selects tokens independently of the others. Penalizing Repeated Tokens: If a beam in a group selects a token already chosen by another group in the
same step, a penalty is applied to that token’s score. Promoting Diversity: This penalty discourages beams within a group from selecting the same tokens as
beams in other groups. Benefits: Diverse Outputs: Produces a variety of different sequences. Exploration: Allows the model to explore different paths.  Examples:   Copied >>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
>>> import torch
>>> # Initialize the model and tokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
>>>


PyTorch

# A long text about the solar system
>>> text = "The Solar System is a gravitationally bound system comprising the Sun and the objects that orbit it, either directly or indirectly. Of the objects that orbit the Sun directly, the largest are the eight planets, with the remainder being smaller objects, such as the five dwarf planets and small Solar System bodies. The Solar System formed 4.6 billion years ago from the gravitational collapse of a giant interstellar molecular cloud."
>>> inputs = tokenizer("summarize: " + text, return_tensors="pt")
>>> # Generate diverse summary
>>> outputs_diverse = model.generate(
...     **inputs,
...     num_beam_groups=2,
...     diversity_penalty=10.0,
...     max_length=100,
...     num_beams=4,
...     num_return_sequences=2,
... )
>>> summaries_diverse = tokenizer.batch_decode(outputs_diverse, skip_special_tokens=True)
>>> # Generate non-diverse summary
>>> outputs_non_diverse = model.generate(
...     **inputs,
...     max_length=100,
...     nu


PyTorch

m_beams=4,
...     num_return_sequences=2,
... )
>>> summary_non_diverse = tokenizer.batch_decode(outputs_non_diverse, skip_special_tokens=True)
>>> # With `diversity_penalty`, the resulting beams are much more diverse
>>> print(summary_non_diverse)
['the solar system formed 4.6 billion years ago from the collapse of a giant interstellar molecular cloud. of the objects that orbit the Sun directly, the largest are the eight planets.',
'the Solar System formed 4.6 billion years ago from the collapse of a giant interstellar molecular cloud. of the objects that orbit the Sun directly, the largest are the eight planets.']
>>> print(summaries_diverse)
['the solar system formed 4.6 billion years ago from the collapse of a giant interstellar molecular cloud. of the objects that orbit the Sun directly, the largest are the eight planets.',
'the solar system formed 4.6 billion years ago from the collapse of a giant interstellar molecular cloud. of the objects that orbit the Sun directly, the la


PyTorch

rgest are the eight planets. the rest of the objects are smaller objects, such as the five dwarf planets and small solar system bodies.']  __call__  < source > ( input_ids: LongTensor scores: FloatTensor current_tokens: LongTensor beam_group_idx: int  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using
beam search or log softmax for each vocabulary token when using beam search   current_tokens (torch.LongTensor of shape (batch_size)) —
Indices of input sequence tokens in the vocabulary, corresponding to the tokens selected by the other
beam groups in the current generation step.   beam_group_idx (int) —
The index of the beam group currently being pr


PyTorch

ocessed.   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.InfNanRemoveLogitsProcessor  < source > ( )    LogitsProcessor that removes all nan and inf values to avoid the generation method to fail. Note that using
the logits processor should only be used if necessary since it can slow down the generation method.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)


PyTorch

The processed prediction scores.

 class transformers.LogitNormalization  < source > ( )    LogitsWarper and LogitsProcessor for normalizing the scores using log-softmax. It’s important to normalize
the scores during beam search, after applying the logits processors or warpers, since the search algorithm used in
this library doesn’t do it (it only does it before, but they may need re-normalization) but it still supposes that
the scores are normalized when comparing the hypotheses.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary t


PyTorch

oken when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.LogitsProcessor  < source > ( )    Abstract base class for all logit processors that can be applied during generation.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.LogitsProcessorList  < source > ( iterable = ()  )    This


PyTorch

class can be used to create a list of LogitsProcessor or LogitsWarper to subsequently process a
scores input tensor. This class inherits from list and adds a specific call method to apply each
LogitsProcessor or LogitsWarper to the inputs.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor **kwargs  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using
beam search or log softmax for each vocabulary token when using beam search   kwargs (Dict[str, Any], optional) —
Additional kwargs that are specific to a logits processor.   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class t


PyTorch

ransformers.LogitsWarper  < source > ( )    Abstract base class for all logit warpers that can be applied during generation with multinomial sampling.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.MinLengthLogitsProcessor  < source > ( min_length: int eos_token_id: typing.Union[int, typing.List[int]]  )   Parameters   min_length (int) —
The minimum length below which the score


PyTorch

of eos_token_id is set to -float("Inf").   eos_token_id (Union[int, List[int]]) —
The id of the end-of-sequence token. Optionally, use a list to set multiple end-of-sequence tokens.    LogitsProcessor enforcing a min-length by setting EOS probability to 0.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.MinNewTokensLengthLogitsProcessor  < source > ( prompt_length_to_skip: int mi


PyTorch

n_new_tokens: int eos_token_id: typing.Union[int, typing.List[int]]  )   Parameters   prompt_length_to_skip (int) —
The input tokens length. Not a valid argument when used with generate as it will automatically assign the
input length.   min_new_tokens (int) —
The minimum new tokens length below which the score of eos_token_id is set to -float("Inf").   eos_token_id (Union[int, List[int]]) —
The id of the end-of-sequence token. Optionally, use a list to set multiple end-of-sequence tokens.    LogitsProcessor enforcing a min-length of new tokens by setting EOS (End-Of-Sequence) token probability to 0.
Note that for decoder-only models, such as Llama2, min_length will compute the length of prompt + newly generated tokens whereas for other models it will behave as min_new_tokens, that is, taking only into account
the newly generated ones.  Examples:   Copied >>> from transformers import AutoTokenizer, AutoModelForCausalLM
>>> tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
>>> m


PyTorch

odel = AutoModelForCausalLM.from_pretrained("distilgpt2")
>>> model.config.pad_token_id = model.config.eos_token_id
>>> inputs = tokenizer(["Hugging Face Company is"], return_tensors="pt")
>>> # If the maximum length (default = 20) is smaller than the minimum length constraint, the latter is ignored!
>>> outputs = model.generate(**inputs, min_new_tokens=30)
>>> print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Hugging Face Company is a company that has been working on a new product for the past year.
>>> # For testing purposes, let's set `eos_token` to `"company"`, the first generated token. This will make
>>> # generation end there.
>>> outputs = model.generate(**inputs, eos_token_id=1664)
>>> print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Hugging Face Company is a company
>>> # Increasing `min_new_tokens` will make generation ignore occurences `"company"` (eos token) before the
>>> # minimum length condition is honored.
>>> outputs = model.generate(**inpu


PyTorch

ts, min_new_tokens=2, eos_token_id=1664)
>>> print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Hugging Face Company is a new company  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.NoBadWordsLogitsProcessor  < source > ( bad_words_ids: typing.List[typing.List[int]] eos_token_id: typing.Union[int, typing.List[int]]  )   Parameters   bad_words_ids (List[List[int]]) —
Li


PyTorch

st of list of token ids that are not allowed to be generated.   eos_token_id (Union[int, List[int]]) —
The id of the end-of-sequence token. Optionally, use a list to set multiple end-of-sequence tokens.    LogitsProcessor that enforces that specified sequences will never be selected. In order to get the token ids of the words that should not appear in the generated text, make sure to set
add_prefix_space=True when initializing the tokenizer, and use tokenizer(bad_words, add_special_tokens=False).input_ids. The add_prefix_space argument is only supported for some slow tokenizers,
as fast tokenizers’ prefixing behaviours come from pre tokenizers. Read more
here.  Examples:   Copied >>> from transformers import AutoTokenizer, AutoModelForCausalLM
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> tokenizer = AutoTokenizer.from_pretrained("gpt2")
>>> inputs = tokenizer(["In a word, the cake is a"], return_tensors="pt")
>>> output_ids = model.generate(inputs["input_ids"], max_ne


PyTorch

w_tokens=5, pad_token_id=tokenizer.eos_token_id)
>>> print(tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0])
In a word, the cake is a bit of a mess.
>>> # Now let's take the bad words out. Please note that the tokenizer is initialized differently
>>> tokenizer_with_prefix_space = AutoTokenizer.from_pretrained("gpt2", add_prefix_space=True)
>>> def get_tokens_as_list(word_list):
...     "Converts a sequence of words into a list of tokens"
...     tokens_list = []
...     for word in word_list:
...         tokenized_word = tokenizer_with_prefix_space([word], add_special_tokens=False).input_ids[0]
...         tokens_list.append(tokenized_word)
...     return tokens_list
>>> bad_words_ids = get_tokens_as_list(word_list=["mess"])
>>> output_ids = model.generate(
...     inputs["input_ids"], max_new_tokens=5, bad_words_ids=bad_words_ids, pad_token_id=tokenizer.eos_token_id
... )
>>> print(tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0])
In a word, the cake is


PyTorch

 a bit of a surprise.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.NoRepeatNGramLogitsProcessor  < source > ( ngram_size: int  )   Parameters   ngram_size (int) —
All ngrams of size ngram_size can only occur once.    N-grams are groups of “n” consecutive words, characters, or tokens taken from a sequence of text. Given the
sentence: “She runs fast”, the bi-grams (n=2) would be


PyTorch

 (“she”, “runs”) and (“runs”, “fast”). In text generation,
avoiding repetitions of word sequences provides a more diverse output. This LogitsProcessor enforces no
repetition of n-grams by setting the scores of banned tokens to negative infinity which eliminates those tokens
from consideration when further processing the scores.
Fairseq. Use n-gram penalties with care. For instance, penalizing 2-grams (bigrams) in an article about the city of New York
might lead to undesirable outcomes where the city’s name appears only once in the entire text.
Reference  Examples:   Copied >>> from transformers import AutoTokenizer, AutoModelForCausalLM
>>> model = AutoModelForCausalLM.from_pretrained("distilgpt2")
>>> tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
>>> inputs = tokenizer(["Today I"], return_tensors="pt")
>>> output = model.generate(**inputs)
>>> print(tokenizer.decode(output[0], skip_special_tokens=True))
Today I’m not sure if I’m going to be able to do it.
>>> # Now let's a


PyTorch

dd ngram size using `no_repeat_ngram_size`. This stops the repetitions ("I’m") in the output.
>>> output = model.generate(**inputs, no_repeat_ngram_size=2)
>>> print(tokenizer.decode(output[0], skip_special_tokens=True))
Today I’m not sure if I can get a better understanding of the nature of this issue  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.PrefixConstrainedLogitsProcess


PyTorch

or  < source > ( prefix_allowed_tokens_fn: typing.Callable[[int, torch.Tensor], typing.List[int]] num_beams: int  )   Parameters   prefix_allowed_tokens_fn (Callable[[int, torch.Tensor], List[int]]) —
This function constraints the beam search to allowed tokens only at each step. This function takes 2
arguments inputs_ids and the batch ID batch_id. It has to return a list with the allowed tokens for the
next generation step conditioned on the previously generated tokens inputs_ids and the batch ID
batch_id.    LogitsProcessor that enforces constrained generation and is useful for prefix-conditioned constrained
generation. See Autoregressive Entity Retrieval for more information.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTens


PyTorch

or of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.RepetitionPenaltyLogitsProcessor  < source > ( penalty: float  )   Parameters   repetition_penalty (float) —
The parameter for repetition penalty. 1.0 means no penalty. See this
paper for more details.    LogitsProcessor that prevents the repetition of previous tokens through an exponential penalty. This technique
shares some similarities with coverage mechanisms and other aimed at reducing repetition. During the text
generation process, the probability distribution for the next token is determined using a formula that incorporates
token scores based on their occurrence in the generated sequence. Tokens with higher scores are more likely


PyTorch

 to be
selected. The formula can be seen in the original paper. According to the
paper a penalty of around 1.2 yields a good balance between truthful generation and lack of repetition.  Examples:   Copied >>> from transformers import AutoTokenizer, AutoModelForCausalLM
>>> # Initializing the model and tokenizer for it
>>> model = AutoModelForCausalLM.from_pretrained("distilgpt2")
>>> tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
>>> inputs = tokenizer(["I'm not going to"], return_tensors="pt")
>>> # This shows a normal generate without any specific parameters
>>> summary_ids = model.generate(**inputs)
>>> print(tokenizer.batch_decode(summary_ids, skip_special_tokens=True)[0])
I'm not going to be able to do that. I'm going to be able to do that
>>> # This generates a penalty for repeated tokens
>>> penalized_ids = model.generate(**inputs, repetition_penalty=1.1)
>>> print(tokenizer.batch_decode(penalized_ids, skip_special_tokens=True)[0])
I'm not going to be able to do that.


PyTorch

 I'll just have to go out and play  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.SequenceBiasLogitsProcessor  < source > ( sequence_bias: typing.Dict[typing.Tuple[int], float]  )   Parameters   sequence_bias (Dict[Tuple[int], float]) —
Dictionary that maps a sequence of tokens to its bias term. Positive biases increase the odds of the
sequence being selected, while negative bia


PyTorch

ses do the opposite. If a sequence has a length of 1, its bias
will always be applied. Otherwise, the bias will only be applied if the sequence in question is about to be
completed (in the token selection step after this processor is applied).    LogitsProcessor that applies an additive bias on sequences. The bias is applied to the last token of a sequence
when the next generated token can complete it. Consequently, to take the most of biasing sequences with more than
one token, consider using beam methods (to gracefully work around partially completed sequences that have a
negative bias) and applying the bias to their prefixes (to ensure the bias is applied earlier). In order to get the token ids of the sequences that you want to bias, make sure to set add_prefix_space=True when
initializing the tokenizer, and use tokenizer(bad_words, add_special_tokens=False).input_ids. The
add_prefix_space argument is only supported for some slow tokenizers, as fast tokenizers’ prefixing behaviour


PyTorch

s
come from pre tokenizers. Read more here.  Examples:   Copied >>> from transformers import AutoTokenizer, AutoModelForCausalLM
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> tokenizer = AutoTokenizer.from_pretrained("gpt2")
>>> inputs = tokenizer(["The full name of Donald is Donald"], return_tensors="pt")
>>> summary_ids = model.generate(inputs["input_ids"], max_new_tokens=4)
>>> print(tokenizer.batch_decode(summary_ids, skip_special_tokens=True)[0])
The full name of Donald is Donald J. Trump Jr
>>> # Now let's control generation through a bias. Please note that the tokenizer is initialized differently!
>>> tokenizer_with_prefix_space = AutoTokenizer.from_pretrained("gpt2", add_prefix_space=True)
>>> def get_tokens_as_tuple(word):
...     return tuple(tokenizer_with_prefix_space([word], add_special_tokens=False).input_ids[0])
>>> # If we add a negative bias without beam search, it may become "stuck" in a prefix without good continuations
>>> sequence_bias = {get_token


PyTorch

s_as_tuple("Trump"): -10.0}
>>> biased_ids = model.generate(inputs["input_ids"], max_new_tokens=4, sequence_bias=sequence_bias)
>>> print(tokenizer.batch_decode(biased_ids, skip_special_tokens=True)[0])
The full name of Donald is Donald J. Donald,
>>> biased_ids = model.generate(inputs["input_ids"], max_new_tokens=4, num_beams=4, sequence_bias=sequence_bias)
>>> print(tokenizer.batch_decode(biased_ids, skip_special_tokens=True)[0])
The full name of Donald is Donald Rumsfeld,
>>> # We can also add a positive bias to nudge the model towards specific tokens or continuations
>>> sequence_bias = {get_tokens_as_tuple("Donald Duck"): 10.0}
>>> biased_ids = model.generate(inputs["input_ids"], max_new_tokens=4, num_beams=4, sequence_bias=sequence_bias)
>>> print(tokenizer.batch_decode(biased_ids, skip_special_tokens=True)[0])
The full name of Donald is Donald Duck.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)


PyTorch

  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.SuppressTokensAtBeginLogitsProcessor  < source > ( begin_suppress_tokens begin_index  )    SuppressTokensAtBeginLogitsProcessor supresses a list of tokens as soon as the generate function starts
generating using begin_index tokens. This should ensure that the tokens defined by begin_suppress_tokens at not
sampled at the begining of the generation.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, co


PyTorch

nfig.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.SuppressTokensLogitsProcessor  < source > ( suppress_tokens  )    This processor can be used to suppress a list of tokens. The processor will set their log probs to -inf so that they
are not sampled.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens


PyTorch

in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.TemperatureLogitsWarper  < source > ( temperature: float  )   Parameters   temperature (float) —
Strictly positive float value used to modulate the logits distribution. A value smaller than 1 decreases
randomness (and vice versa), with 0 being equivalent to shifting all probability mass to the most likely
token.    LogitsWarper for temperature (exponential scaling output probability distribution), which effectively means
that it can control the randomness of the predicted tokens. Make sure that do_sample=True is included in the generate arguments otherwise the temperature va


PyTorch

lue won’t have
any effect.  Examples:   Copied >>> import torch
>>> from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed
>>> set_seed(0)  # for reproducibility
>>> tokenizer = AutoTokenizer.from_pretrained("gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> model.config.pad_token_id = model.config.eos_token_id
>>> inputs = tokenizer(["Hugging Face Company is"], return_tensors="pt")
>>> # With temperature=1.0, the default, we consistently get random outputs due to random sampling.
>>> generate_kwargs = {"max_new_tokens": 10, "do_sample": True, "temperature": 1.0, "num_return_sequences": 2}
>>> outputs = model.generate(**inputs, **generate_kwargs)
>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['Hugging Face Company is a joint venture between GEO Group, one of',
'Hugging Face Company is not an exact science – but what we believe does']
>>> # However, with temperature close to 0, it approximates greedy decoding strategies (invaria


PyTorch

nt)
>>> generate_kwargs["temperature"] = 0.0001
>>> outputs = model.generate(**inputs, **generate_kwargs)
>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['Hugging Face Company is a company that has been around for over 20 years',
'Hugging Face Company is a company that has been around for over 20 years']  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.TopKLo


PyTorch

gitsWarper  < source > ( top_k: int filter_value: float = -inf min_tokens_to_keep: int = 1  )   Parameters   top_k (int) —
The number of highest probability vocabulary tokens to keep for top-k-filtering.   filter_value (float, optional, defaults to -inf) —
All filtered values will be set to this float value.   min_tokens_to_keep (int, optional, defaults to 1) —
Minimum number of tokens that cannot be filtered.    LogitsWarper that performs top-k, i.e. restricting to the k highest probability elements.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for


PyTorch

each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.TopPLogitsWarper  < source > ( top_p: float filter_value: float = -inf min_tokens_to_keep: int = 1  )   Parameters   top_p (float) —
If set to < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or
higher are kept for generation.   filter_value (float, optional, defaults to -inf) —
All filtered values will be set to this float value.   min_tokens_to_keep (int, optional, defaults to 1) —
Minimum number of tokens that cannot be filtered.    LogitsWarper that performs top-p, i.e. restricting to top tokens summing to prob_cut_off <= prob_cut_off.  Examples:   Copied >>> from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed
>>> set_seed(0)
>>> model = AutoModelForCausalLM.from_pretrained("distilgpt2")
>>> tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
>>>


PyTorch

 inputs = tokenizer("A sequence: 1, 2", return_tensors="pt")
>>> # With sampling, the output is unexpected -- sometimes too unexpected.
>>> outputs = model.generate(**inputs, do_sample=True)
>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True)[0])
A sequence: 1, 2, 0, 2, 2. 2, 2, 2, 2
>>> # With `top_p` sampling, the output gets restricted to high-probability tokens.
>>> # Pro tip: In practice, LLMs use `top_p` in the 0.9-0.95 range.
>>> outputs = model.generate(**inputs, do_sample=True, top_p=0.1)
>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True)[0])
A sequence: 1, 2, 3, 4, 5, 6, 7, 8, 9  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction


PyTorch

scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.TypicalLogitsWarper  < source > ( mass: float = 0.9 filter_value: float = -inf min_tokens_to_keep: int = 1  )   Parameters   mass (float, optional, defaults to 0.9) —
Value of typical_p between 0 and 1 inclusive, defaults to 0.9.   filter_value (float, optional, defaults to -inf) —
All filtered values will be set to this float value.   min_tokens_to_keep (int, optional, defaults to 1) —
Minimum number of tokens that cannot be filtered.    LogitsWarper that performs typical decoding. See Typical Decoding for Natural Language
Generation for more information.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameter


PyTorch

s   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.UnbatchedClassifierFreeGuidanceLogitsProcessor  < source > ( guidance_scale: float model unconditional_ids: typing.Optional[torch.LongTensor] = None unconditional_attention_mask: typing.Optional[torch.LongTensor] = None use_cache: typing.Optional[bool] = True  )   Parameters   guidance_scale (float) —
The guidance scale for classifier free guidance (CFG). CFG is enabled by setting guidance_scale != 1.
Higher guidance scale encourages the model to generate samples that a


PyTorch

re more closely linked to the input
prompt, usually at the expense of poorer quality. A value smaller than 1 has the opposite effect, while
making the negative prompt provided with negative_prompt_ids (if any) act as a positive prompt.   model (PreTrainedModel) —
The model computing the unconditional scores. Supposedly the same as the one computing the conditional
scores. Both models must use the same tokenizer.   unconditional_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of input sequence tokens in the vocabulary for the unconditional branch. If unset, will default to
the last token of the prompt.   unconditional_attention_mask (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Attention mask for unconditional_ids.   use_cache (bool, optional, defaults to True) —
Whether to cache key/values during the negative prompt forward pass.    Logits processor for Classifier-Free Guidance (CFG). The processors
computes a weighted average


PyTorch

 across scores from prompt conditional and prompt unconditional (or negative) logits,
parameterized by the guidance_scale. The unconditional scores are computed internally by prompting model with
the unconditional_ids branch. See the paper for more information.  Examples:   Copied >>> from transformers import AutoTokenizer, AutoModelForCausalLM
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> tokenizer = AutoTokenizer.from_pretrained("gpt2")
>>> inputs = tokenizer(["Today, a dragon flew over Paris, France,"], return_tensors="pt")
>>> out = model.generate(inputs["input_ids"], guidance_scale=1.5)
>>> tokenizer.batch_decode(out, skip_special_tokens=True)[0]
'Today, a dragon flew over Paris, France, killing at least 50 people and injuring more than 100'
>>> # with a negative prompt
>>> neg_inputs = tokenizer(["A very happy event happened,"], return_tensors="pt")
>>> out = model.generate(inputs["input_ids"], guidance_scale=2, negative_prompt_ids=neg_inputs["input_ids"])
>>> to


PyTorch

kenizer.batch_decode(out, skip_special_tokens=True)[0]
'Today, a dragon flew over Paris, France, killing at least 130 people. French media reported that'
>>> # with a positive prompt
>>> neg_inputs = tokenizer(["A very happy event happened,"], return_tensors="pt")
>>> out = model.generate(inputs["input_ids"], guidance_scale=0, negative_prompt_ids=neg_inputs["input_ids"])
>>> tokenizer.batch_decode(out, skip_special_tokens=True)[0]
"Today, a dragon flew over Paris, France, and I'm very happy to be here. I"  __call__  < source > ( input_ids scores  )
 class transformers.WhisperTimeStampLogitsProcessor  < source > ( generate_config  )   Parameters   generate_config (GenerateConfig) —
The generate config used to generate the output. The following parameters are required:
eos_token_id (int, optional, defaults to 50257):
The id of the end-of-sequence token.
no_timestamps_token_id (int, optional, defaults to 50363):
The id of the "<|notimestamps|>" token.
max_initial_timestamp_index (int


PyTorch

, optional, defaults to 1):
Used to set the maximum value of the initial timestamp. This is used to prevent the model from
predicting timestamps that are too far in the future.    Whisper specific Processor. This processor can be used to force a list of tokens. The processor will set their log
probs to inf so that they are sampled at their corresponding index. See the paper for more information.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor  ) → torch.FloatTensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   Returns
torch.FloatTensor of shape (batch_size, config.vocab


TensorFlow

class transformers.TFForcedBOSTokenLogitsProcessor  < source > ( bos_token_id: int  )   Parameters   bos_token_id (int) —
The id of the token to force as the first generated token.    TFLogitsProcessor that enforces the specified token as the first generated token.  __call__  < source > ( input_ids: Tensor scores: Tensor cur_len: int  )
 class transformers.TFForcedEOSTokenLogitsProcessor  < source > ( max_length: int eos_token_id: int  )   Parameters   max_length (int) —
The maximum length of the sequence to be generated.   eos_token_id (int) —
The id of the token to force as the last generated token when max_length is reached.    TFLogitsProcessor that enforces the specified token as the last generated token when max_length is reached.  __call__  < source > ( input_ids: Tensor scores: Tensor cur_len: int  )
 class transformers.TFForceTokensLogitsProcessor  < source > ( force_token_map: typing.List[typing.List[int]]  )    Th


TensorFlow

is processor takes a list of pairs of integers which indicates a mapping from generation indices to token
indices that will be forced before sampling. The processor will set their log probs to 0 and all other tokens to
-inf so that they are sampled at their corresponding index.  __call__  < source > ( input_ids: Tensor scores: Tensor cur_len: int  )
 class transformers.TFLogitsProcessor  < source > ( )    Abstract base class for all logit processors that can be applied during generation.  __call__  < source > ( input_ids: Tensor scores: Tensor cur_len: int  ) → tf.Tensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (tf.Tensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using PreTrainedTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   scores (tf.Tensor of shape (batch_size, con


TensorFlow

fig.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search.   cur_len (int) —
The current length of valid input sequence tokens. In the TF implementation, the input_ids’ sequence length
is the maximum length generate can produce, and we need to know which of its tokens are valid.   kwargs (Dict[str, Any], optional) —
Additional logits processor specific kwargs.   Returns
tf.Tensor of shape (batch_size, config.vocab_size)

The processed prediction scores.
  TF method for processing logits.
 class transformers.TFLogitsProcessorList  < source > ( iterable = ()  )    This class can be used to create a list of TFLogitsProcessor to subsequently process a scores input tensor.
This class inherits from list and adds a specific call method to apply each TFLogitsProcessor to the
inputs.  __call__  < sou


TensorFlow

rce > ( input_ids: Tensor scores: Tensor cur_len: int **kwargs  ) → tf.Tensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (tf.Tensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using PreTrainedTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   scores (tf.Tensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search.   cur_len (int) —
The current length of valid input sequence tokens. In the TF implementation, the input_ids’ sequence length
is the maximum length generate can produce, and we need to know which of its tokens are valid.   kwargs (Dict[str, Any], optional) —
Additional logits processor specific kwargs.


TensorFlow

 Returns
tf.Tensor of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.TFLogitsWarper  < source > ( )    Abstract base class for all logit warpers that can be applied during generation with multinomial sampling.  __call__  < source > ( input_ids: Tensor scores: Tensor cur_len: int  ) → tf.Tensor of shape (batch_size, config.vocab_size)  Parameters   input_ids (tf.Tensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using PreTrainedTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   scores (tf.Tensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search.   cur_len (int) —
The current length o


TensorFlow

f valid input sequence tokens. In the TF implementation, the input_ids’ sequence length
is the maximum length generate can produce, and we need to know which of its tokens are valid.   kwargs (Dict[str, Any], optional) —
Additional logits processor specific kwargs.   Returns
tf.Tensor of shape (batch_size, config.vocab_size)

The processed prediction scores.
  TF method for warping logits.
 class transformers.TFMinLengthLogitsProcessor  < source > ( min_length: int eos_token_id: int  )   Parameters   min_length (int) —
The minimum length below which the score of eos_token_id is set to -float("Inf").   eos_token_id (int) —
The id of the end-of-sequence token.    TFLogitsProcessor enforcing a min-length by setting EOS probability to 0.  __call__  < source > ( input_ids: Tensor scores: Tensor cur_len: int  )
 class transformers.TFNoBadWordsLogitsProcessor  < source > ( bad_words_ids: typing.List[typing.List[int]] eos_token_id:


TensorFlow

int  )   Parameters   bad_words_ids (List[List[int]]) —
List of list of token ids that are not allowed to be generated. In order to get the tokens of the words
that should not appear in the generated text, make sure to set add_prefix_space=True when initializing
the tokenizer, and use tokenizer(bad_words, add_special_tokens=False).input_ids. The add_prefix_space
argument is only supported for some slow tokenizers, as fast tokenizers’ prefixing behaviours come from
pre tokenizers. Read more here.   eos_token_id (int) —
The id of the end-of-sequence token.    TFLogitsProcessor that enforces that specified sequences will never be sampled.  __call__  < source > ( input_ids: Tensor scores: Tensor cur_len: int  )
 class transformers.TFNoRepeatNGramLogitsProcessor  < source > ( ngram_size: int  )   Parameters   ngram_size (int) —
All ngrams of size ngram_size can only occur once.    TFLogitsProcessor that enforces no repetition of n-g


TensorFlow

rams. See
Fairseq.  __call__  < source > ( input_ids: Tensor scores: Tensor cur_len: int  )
 class transformers.TFRepetitionPenaltyLogitsProcessor  < source > ( penalty: float  )   Parameters   repetition_penalty (float) —
The parameter for repetition penalty. 1.0 means no penalty. See this
paper for more details.    TFLogitsProcessor enforcing an exponential penalty on repeated sequences.  __call__  < source > ( input_ids: Tensor scores: Tensor cur_len: int  )
 class transformers.TFSuppressTokensAtBeginLogitsProcessor  < source > ( begin_suppress_tokens begin_index  )    TFSuppressTokensAtBeginLogitsProcessor suppresses a list of tokens as soon as the generate function starts
generating using begin_index tokens. This should ensure that the tokens defined by begin_suppress_tokens at not
sampled at the begining of the generation.  __call__  < source > ( input_ids: Tensor scores: Tensor cur_len: int  )
 class transformers.T


TensorFlow

FSuppressTokensLogitsProcessor  < source > ( suppress_tokens  )    This processor can be used to suppress a list of tokens. The processor will set their log probs to -inf so that they
are not sampled.  __call__  < source > ( input_ids: Tensor scores: Tensor cur_len: int  )
 class transformers.TFTemperatureLogitsWarper  < source > ( temperature: float  )   Parameters   temperature (float) —
The value used to module the logits distribution.    TFLogitsWarper for temperature (exponential scaling output probability distribution).  __call__  < source > ( input_ids: Tensor scores: Tensor cur_len: int  )
 class transformers.TFTopKLogitsWarper  < source > ( top_k: int filter_value: float = -inf min_tokens_to_keep: int = 1  )   Parameters   top_k (int) —
The number of highest probability vocabulary tokens to keep for top-k-filtering.   filter_value (float, optional, defaults to -inf) —
All filtered values will be set to this float va


TensorFlow

lue.   min_tokens_to_keep (int, optional, defaults to 1) —
Minimum number of tokens that cannot be filtered.    TFLogitsWarper that performs top-k, i.e. restricting to the k highest probability elements.  __call__  < source > ( input_ids: Tensor scores: Tensor cur_len: int  )
 class transformers.TFTopPLogitsWarper  < source > ( top_p: float filter_value: float = -inf min_tokens_to_keep: int = 1  )   Parameters   top_p (float) —
If set to < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or
higher are kept for generation.   filter_value (float, optional, defaults to -inf) —
All filtered values will be set to this float value.   min_tokens_to_keep (int, optional, defaults to 1) —
Minimum number of tokens that cannot be filtered.    TFLogitsWarper that performs top-p, i.e. restricting to top tokens summing to <= prob_cut_off.  __call__  < source > ( input_ids: Tensor scores: Tensor cur_len:


FLAX

class transformers.FlaxForcedBOSTokenLogitsProcessor  < source > ( bos_token_id: int  )   Parameters   bos_token_id (int) —
The id of the token to force as the first generated token.    FlaxLogitsProcessor that enforces the specified token as the first generated token.  __call__  < source > ( input_ids: Array scores: Array cur_len: int  )
 class transformers.FlaxForcedEOSTokenLogitsProcessor  < source > ( max_length: int eos_token_id: int  )   Parameters   max_length (int) —
The maximum length of the sequence to be generated.   eos_token_id (int) —
The id of the token to force as the last generated token when max_length is reached.    FlaxLogitsProcessor that enforces the specified token as the last generated token when max_length is reached.  __call__  < source > ( input_ids: Array scores: Array cur_len: int  )
 class transformers.FlaxForceTokensLogitsProcessor  < source > ( force_token_map  )   Parameters   force_token_ma


FLAX

p (list) —
Map giving token ids and indices where they will be forced to be sampled.    FlaxLogitsProcessor that takes a list of pairs of integers which indicates a mapping from generation indices to
token indices that will be forced before sampling. The processor will set their log probs to 0 and all other tokens
to -inf so that they are sampled at their corresponding index.  __call__  < source > ( input_ids: Array scores: Array cur_len: int  )
 class transformers.FlaxLogitsProcessor  < source > ( )    Abstract base class for all logit processors that can be applied during generation.  __call__  < source > ( input_ids: Array scores: Array  ) → jnp.ndarray of shape (batch_size, config.vocab_size)  Parameters   input_ids (jnp.ndarray of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using PreTrainedTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedToken


FLAX

izer.call() for details.
What are input IDs?   scores (jnp.ndarray of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   kwargs (Dict[str, Any], optional) —
Additional logits processor specific kwargs.   Returns
jnp.ndarray of shape (batch_size, config.vocab_size)

The processed prediction scores.
  Flax method for processing logits.
 class transformers.FlaxLogitsProcessorList  < source > ( iterable = ()  )    This class can be used to create a list of FlaxLogitsProcessor or FlaxLogitsWarper to subsequently process
a scores input tensor. This class inherits from list and adds a specific call method to apply each
FlaxLogitsProcessor or FlaxLogitsWarper to the inputs.  __call__  < source > ( input_ids: Array scores: Array cur_len: int **kwargs  ) → jnp.ndarray o


FLAX

f shape (batch_size, config.vocab_size)  Parameters   input_ids (jnp.ndarray of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using PreTrainedTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   scores (jnp.ndarray of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   kwargs (Dict[str, Any], optional) —
Additional logits processor specific kwargs.   Returns
jnp.ndarray of shape (batch_size, config.vocab_size)

The processed prediction scores.

 class transformers.FlaxLogitsWarper  < source > ( )    Abstract base class for all logit warpers that can be applied during generation with multinomial sampling.  __call__  < source > ( input_id


FLAX

s: Array scores: Array  ) → jnp.ndarray of shape (batch_size, config.vocab_size)  Parameters   input_ids (jnp.ndarray of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using PreTrainedTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   scores (jnp.ndarray of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam
search or log softmax for each vocabulary token when using beam search   kwargs (Dict[str, Any], optional) —
Additional logits processor specific kwargs.   Returns
jnp.ndarray of shape (batch_size, config.vocab_size)

The processed prediction scores.
  Flax method for warping logits.
 class transformers.FlaxMinLengthLogitsProcessor  < source > ( min_length: int eos_token_id: int  )   Parameters   min_len


FLAX

gth (int) —
The minimum length below which the score of eos_token_id is set to -float("Inf").   eos_token_id (int) —
The id of the end-of-sequence token.    FlaxLogitsProcessor enforcing a min-length by setting EOS probability to 0.  __call__  < source > ( input_ids: Array scores: Array cur_len: int  )
 class transformers.FlaxSuppressTokensAtBeginLogitsProcessor  < source > ( begin_suppress_tokens begin_index  )   Parameters   begin_suppress_tokens (List[int]) —
Tokens to not sample.   begin_index (int) —
Index where the tokens are suppressed.    FlaxLogitsProcessor supressing a list of tokens as soon as the generate function starts generating using
begin_index tokens. This should ensure that the tokens defined by begin_suppress_tokens are not sampled at the
begining of the generation.  __call__  < source > ( input_ids scores cur_len: int  )
 class transformers.FlaxSuppressTokensLogitsProcessor  < source > ( suppress_tokens


FLAX

: list  )   Parameters   suppress_tokens (list) —
Tokens to not sample.    FlaxLogitsProcessor suppressing a list of tokens at each decoding step. The processor will set their log probs
to be -inf so they are not sampled.  __call__  < source > ( input_ids: Array scores: Array cur_len: int  )
 class transformers.FlaxTemperatureLogitsWarper  < source > ( temperature: float  )   Parameters   temperature (float) —
The value used to module the logits distribution.    FlaxLogitsWarper for temperature (exponential scaling output probability distribution).  __call__  < source > ( input_ids: Array scores: Array cur_len: int  )
 class transformers.FlaxTopKLogitsWarper  < source > ( top_k: int filter_value: float = -inf min_tokens_to_keep: int = 1  )   Parameters   top_k (int) —
The number of highest probability vocabulary tokens to keep for top-k-filtering.   filter_value (float, optional, defaults to -inf) —
All filtered values will


FLAX

 be set to this float value.   min_tokens_to_keep (int, optional, defaults to 1) —
Minimum number of tokens that cannot be filtered.    FlaxLogitsWarper that performs top-k, i.e. restricting to the k highest probability elements.  __call__  < source > ( input_ids: Array scores: Array cur_len: int  )
 class transformers.FlaxTopPLogitsWarper  < source > ( top_p: float filter_value: float = -inf min_tokens_to_keep: int = 1  )   Parameters   top_p (float) —
If set to < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or
higher are kept for generation.   filter_value (float, optional, defaults to -inf) —
All filtered values will be set to this float value.   min_tokens_to_keep (int, optional, defaults to 1) —
Minimum number of tokens that cannot be filtered.    FlaxLogitsWarper that performs top-p, i.e. restricting to top tokens summing to prob_cut_off <= prob_cut_off.  __call__  < source > (


FLAX

 input_ids: Array scores: Array cur_len: int  )
 class transformers.FlaxWhisperTimeStampLogitsProcessor  < source > ( generate_config model_config decoder_input_length  )   Parameters   generate_config (GenerateConfig) —
The generate config used to generate the output. The following parameters are required:
eos_token_id (int, optional, defaults to 50257):
The id of the end-of-sequence token.
no_timestamps_token_id (int, optional, defaults to 50363):
The id of the "<|notimestamps|>" token.
max_initial_timestamp_index (int, optional, defaults to 1):
Used to set the maximum value of the initial timestamp. This is used to prevent the model from
predicting timestamps that are too far in the future.    Whisper specific Processor. This processor can be used to force a list of tokens. The processor will set their log
probs to inf so that they are sampled at their corresponding index.  __call__  < source > ( input_ids scores cur_len  )


StoppingCriteria

A StoppingCriteria can be used to change when to stop generation (other than EOS token). Please note that this is exclusivelly available to our PyTorch implementations.
 class transformers.StoppingCriteria  < source > ( )    Abstract base class for all stopping criteria that can be applied during generation.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor **kwargs  )   Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be scores for each vocabulary token before S


StoppingCriteria

oftMax
or scores for each vocabulary token after SoftMax.   kwargs (Dict[str, Any], optional) —
Additional stopping criteria specific kwargs.
 class transformers.StoppingCriteriaList  < source > ( iterable = ()  )     __call__  < source > ( input_ids: LongTensor scores: FloatTensor **kwargs  )   Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be scores for each vocabulary token before SoftMax
or scores for each vocabulary token after SoftMax.   kwargs (Dict[str, Any], option


StoppingCriteria

al) —
Additional stopping criteria specific kwargs.
 class transformers.MaxLengthCriteria  < source > ( max_length: int max_position_embeddings: typing.Optional[int] = None  )   Parameters   max_length (int) —
The maximum length that the output sequence can have in number of tokens.   max_position_embeddings (int, optional) —
The maximum model length, as defined by the model’s config.max_position_embeddings attribute.    This class can be used to stop generation whenever the full generated number of tokens exceeds max_length. Keep
in mind for decoder-only type of transformers, this will include the initial prompted tokens.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor **kwargs  )   Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the voc


StoppingCriteria

abulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be scores for each vocabulary token before SoftMax
or scores for each vocabulary token after SoftMax.   kwargs (Dict[str, Any], optional) —
Additional stopping criteria specific kwargs.
 class transformers.MaxTimeCriteria  < source > ( max_time: float initial_timestamp: typing.Optional[float] = None  )   Parameters   max_time (float) —
The maximum allowed time in seconds for the generation.   initial_time (float, optional, defaults to time.time()) —
The start of the generation allowed time.    This class can be used to stop generation whenever the ful


StoppingCriteria

l generation exceeds some amount of time. By default, the
time will start being counted when you initialize this function. You can override this by passing an
initial_time.  __call__  < source > ( input_ids: LongTensor scores: FloatTensor **kwargs  )   Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   scores (torch.FloatTensor of shape (batch_size, config.vocab_size)) —
Prediction scores of a language modeling head. These can be scores for each vocabulary token before SoftMax
or scores for each vocabulary token after SoftMax.   kwargs (Dict[str, Any], optional) —
Additional stopping criteria specific kwa


Constraints

A Constraint can be used to force the generation to include specific tokens or sequences in the output. Please note that this is exclusivelly available to our PyTorch implementations.
 class transformers.Constraint  < source > ( )    Abstract base class for all constraints that can be applied during generation.
It must define how the constraint can be satisfied. All classes that inherit Constraint must follow the requirement that    Copied completed = False
while not completed:
    _, completed = constraint.update(constraint.advance()) will always terminate (halt).  advance  < source > ( ) → token_ids(torch.tensor)  Returns
token_ids(torch.tensor)

Must be a tensor of a list of indexable tokens, not some integer.
  When called, returns the token that would take this constraint one step closer to bei


Constraints

ng fulfilled.  copy  < source > ( stateful = False  ) → constraint(Constraint)  Returns
constraint(Constraint)

The same constraint as the one being called from.
  Creates a new instance of this constraint.  does_advance  < source > ( token_id: int  )    Reads in a token and returns whether it creates progress.  remaining  < source > ( )    Returns the number of remaining steps of advance() in order to complete this constraint.  reset  < source > ( )    Resets the state of this constraint to its initialization. We would call this in cases where the fulfillment of
a constraint is abrupted by an unwanted token.  test  < source > ( )    Tests whether this constraint has been properly defined.  update  < source > ( token_id: int  ) → stepped(bool)  Returns
stepped(bool)

Whether this constraint has b


Constraints

ecome one step closer to being fulfuilled.
completed(bool):
Whether this constraint has been completely fulfilled by this token being generated.
reset (bool):
Whether this constraint has reset its progress by this token being generated.
  Reads in a token and returns booleans that indicate the progress made by it. This function will update the
state of this object unlikes does_advance(self, token_id: int). This isn’t to test whether a certain token will advance the progress; it’s to update its state as if it has
been generated. This becomes important if token_id != desired token (refer to else statement in
PhrasalConstraint)
 class transformers.PhrasalConstraint  < source > ( token_ids: typing.List[int]  )   Parameters   token_ids (List[int]) —
The id of the token that must be generated by the output.


Constraints

   Constraint enforcing that an ordered sequence of tokens is included in the output.
 class transformers.DisjunctiveConstraint  < source > ( nested_token_ids: typing.List[typing.List[int]]  )   Parameters   nested_token_ids (List[List[int]]) —
A list of words, where each word is a list of ids. This constraint is fulfilled by generating just one from
the list of words.    A special Constraint that is fulfilled by fulfilling just one of several constraints.
 class transformers.ConstraintListState  < source > ( constraints: typing.List[transformers.generation.beam_constraints.Constraint]  )   Parameters   constraints (List[Constraint]) —
A list of Constraint objects that must be fulfilled by the beam scorer.    A class for beam scorers to track its progress through a list of constraints.  advance  < sour


Constraints

ce > ( )    The list of tokens to generate such that we can make progress.
By “list” we don’t mean the list of token that will fully fulfill a constraint. Given constraints c_i = {t_ij | j == # of tokens}, If we’re not in the middle of progressing through a
specific constraint c_i, we return: [t_k1 for k in indices of unfulfilled constraints] If we are in the middle of a constraint, then we return:
[t_ij], where i is the index of the inprogress constraint, j is the next step for the constraint. Though we don’t care which constraint is fulfilled first, if we are in the progress of fulfilling a constraint,
that’s the only one we’ll return.  reset  < source > ( token_ids: typing.Optional[typing.List[int]]  )    token_ids: the tokens generated thus far to reset the state of the progress through constraints


BeamSearch

class transformers.BeamScorer  < source > ( )    Abstract base class for all beam scorers that are used for beam_search() and
beam_sample().  process  < source > ( input_ids: LongTensor next_scores: FloatTensor next_tokens: LongTensor next_indices: LongTensor **kwargs  ) → UserDict  Parameters   input_ids (torch.LongTensor of shape (batch_size * num_beams, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using any class inheriting from PreTrainedTokenizer. See
PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.
What are input IDs?   next_scores (torch.FloatTensor of shape (batch_size, 2 * num_beams)) —
Current scores of the top 2 * num_beams non-finished beam hypotheses.   next_tokens (torch.LongTensor of shape (batch_size, 2 * num_beams)) —
input_ids of the tokens corresponding to the top 2 * num_beams non-finished beam hypotheses.   nex


BeamSearch

t_indices (torch.LongTensor of shape (batch_size, 2 * num_beams)) —
Beam indices indicating to which beam hypothesis the next_tokens correspond.   pad_token_id (int, optional) —
The id of the padding token.   eos_token_id (Union[int, List[int]], optional) —
The id of the end-of-sequence token. Optionally, use a list to set multiple end-of-sequence tokens.   beam_indices (torch.LongTensor, optional) —
Beam indices indicating to which beam hypothesis each token correspond.   group_index (int, optional) —
The index of the group of beams. Used with group_beam_search().   Returns
UserDict

A dictionary composed of the fields as defined above:
next_beam_scores (torch.FloatTensor of shape (batch_size * num_beams)) — Updated scores of all
non-finished beams.
next_beam_tokens (torch.FloatTensor of shape (batch_size * num_beams)) — Next tokens to be added
to the non-finished beam_hypotheses.
next_beam_indices (t


BeamSearch

orch.FloatTensor of shape (batch_size * num_beams)) — Beam indices
indicating to which beam the next tokens shall be added.
   finalize  < source > ( input_ids: LongTensor next_scores: FloatTensor next_tokens: LongTensor next_indices: LongTensor max_length: int **kwargs  ) → torch.LongTensor of shape (batch_size * num_return_sequences, sequence_length)  Parameters   input_ids (torch.LongTensor of shape (batch_size * num_beams, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using any class inheriting from PreTrainedTokenizer. See
PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.
What are input IDs?   final_beam_scores (torch.FloatTensor of shape (batch_size * num_beams)) —
The final scores of all non-finished beams.   final_beam_tokens (torch.FloatTensor of shape (batch_size * num_beams)) —
The last tokens to be added to the non-finish


BeamSearch

ed beam_hypotheses.   final_beam_indices (torch.FloatTensor of shape (batch_size * num_beams)) —
The beam indices indicating to which beam the final_beam_tokens shall be added.   pad_token_id (int, optional) —
The id of the padding token.   eos_token_id (Union[int, List[int]], optional) —
The id of the end-of-sequence token. Optionally, use a list to set multiple end-of-sequence tokens.   Returns
torch.LongTensor of shape (batch_size * num_return_sequences, sequence_length)

The generated sequences.
The second dimension (sequence_length) is either equal to max_length or shorter if all batches finished early
due to the eos_token_id.

 class transformers.BeamSearchScorer  < source > ( batch_size: int num_beams: int device: device length_penalty: typing.Optional[float] = 1.0 do_early_stopping: typing.Union[str, bool, NoneType] = False num_beam_hyps_to_keep: typing.Optional[int] = 1 num_beam_groups: typin


BeamSearch

g.Optional[int] = 1 max_length: typing.Optional[int] = None  )   Parameters   batch_size (int) —
Batch Size of input_ids for which standard beam search decoding is run in parallel.   num_beams (int) —
Number of beams for beam search.   device (torch.device) —
Defines the device type (e.g., "cpu" or "cuda") on which this instance of BeamSearchScorer will be
allocated.   length_penalty (float, optional, defaults to 1.0) —
Exponential penalty to the length that is used with beam-based generation. It is applied as an exponent to
the sequence length, which in turn is used to divide the score of the sequence. Since the score is the log
likelihood of the sequence (i.e. negative), length_penalty > 0.0 promotes longer sequences, while
length_penalty < 0.0 encourages shorter sequences.   do_early_stopping (bool or str, optional, defaults to False) —
Controls the stopping condition for beam-based methods, like beam-


BeamSearch

search. It accepts the following values:
True, where the generation stops as soon as there are num_beams complete candidates; False, where an
heuristic is applied and the generation stops when is it very unlikely to find better candidates;
"never", where the beam search procedure only stops when there cannot be better candidates (canonical
beam search algorithm).   num_beam_hyps_to_keep (int, optional, defaults to 1) —
The number of beam hypotheses that shall be returned upon calling
~transformer.BeamSearchScorer.finalize.   num_beam_groups (int, optional, defaults to 1) —
Number of groups to divide num_beams into in order to ensure diversity among different groups of beams.
See this paper for more details.   max_length (int, optional) —
The maximum length of the sequence to be generated.    BeamScorer implementing standard beam search decoding. Adapted in part from Facebook’s XLM beam search
code. Refere


BeamSearch

nce for the diverse beam search algorithm and implementation Ashwin Kalyan’s DBS
implementation  process  < source > ( input_ids: LongTensor next_scores: FloatTensor next_tokens: LongTensor next_indices: LongTensor pad_token_id: typing.Optional[int] = None eos_token_id: typing.Union[int, typing.List[int], NoneType] = None beam_indices: typing.Optional[torch.LongTensor] = None group_index: typing.Optional[int] = 0  )     finalize  < source > ( input_ids: LongTensor final_beam_scores: FloatTensor final_beam_tokens: LongTensor final_beam_indices: LongTensor max_length: int pad_token_id: typing.Optional[int] = None eos_token_id: typing.Union[int, typing.List[int], NoneType] = None beam_indices: typing.Optional[torch.LongTensor] = None  )
 class transformers.ConstrainedBeamSearchScorer  < source > ( batch_size: int num_beams: int constraints: typing.List[transformers.generation.beam_constraints.Constraint]


BeamSearch

device: device length_penalty: typing.Optional[float] = 1.0 do_early_stopping: typing.Union[str, bool, NoneType] = False num_beam_hyps_to_keep: typing.Optional[int] = 1 num_beam_groups: typing.Optional[int] = 1 max_length: typing.Optional[int] = None  )   Parameters   batch_size (int) —
Batch Size of input_ids for which standard beam search decoding is run in parallel.   num_beams (int) —
Number of beams for beam search.   constraints (List[Constraint]) —
A list of positive constraints represented as Constraint objects that must be fulfilled in the generation
output. For more information, the documentation of Constraint should be read.   device (torch.device) —
Defines the device type (e.g., "cpu" or "cuda") on which this instance of BeamSearchScorer will be
allocated.   length_penalty (float, optional, defaults to 1.0) —
Exponential penalty to the length that is used with beam-based generation. It is app


BeamSearch

lied as an exponent to
the sequence length, which in turn is used to divide the score of the sequence. Since the score is the log
likelihood of the sequence (i.e. negative), length_penalty > 0.0 promotes longer sequences, while
length_penalty < 0.0 encourages shorter sequences.   do_early_stopping (bool or str, optional, defaults to False) —
Controls the stopping condition for beam-based methods, like beam-search. It accepts the following values:
True, where the generation stops as soon as there are num_beams complete candidates; False, where an
heuristic is applied and the generation stops when is it very unlikely to find better candidates;
"never", where the beam search procedure only stops when there cannot be better candidates (canonical
beam search algorithm).   num_beam_hyps_to_keep (int, optional, defaults to 1) —
The number of beam hypotheses that shall be returned upon calling
~transformer.BeamSe


BeamSearch

archScorer.finalize.   num_beam_groups (int, optional, defaults to 1) —
Number of groups to divide num_beams into in order to ensure diversity among different groups of beams.
See this paper for more details.   max_length (int, optional) —
The maximum length of the sequence to be generated.    BeamScorer implementing constrained beam search decoding.  process  < source > ( input_ids: LongTensor next_scores: FloatTensor next_tokens: LongTensor next_indices: LongTensor scores_for_all_vocab: FloatTensor pad_token_id: typing.Optional[int] = None eos_token_id: typing.Union[int, typing.List[int], NoneType] = None beam_indices: typing.Optional[torch.LongTensor] = None  ) → UserDict  Parameters   input_ids (torch.LongTensor of shape (batch_size * num_beams, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using any class inheriting from PreTrainedTokenizer. See
PreTr


BeamSearch

ainedTokenizer.encode() and PreTrainedTokenizer.call() for details.
What are input IDs?   next_scores (torch.FloatTensor of shape (batch_size, 2 * num_beams)) —
Current scores of the top 2 * num_beams non-finished beam hypotheses.   next_tokens (torch.LongTensor of shape (batch_size, 2 * num_beams)) —
input_ids of the tokens corresponding to the top 2 * num_beams non-finished beam hypotheses.   next_indices (torch.LongTensor of shape (batch_size, 2 * num_beams)) —
Beam indices indicating to which beam hypothesis the next_tokens correspond.   scores_for_all_vocab (torch.FloatTensor of shape (batch_size * num_beams, sequence_length)) —
The scores of all tokens in the vocabulary for each of the beam hypotheses.   pad_token_id (int, optional) —
The id of the padding token.   eos_token_id (Union[int, List[int]], optional) —
The id of the end-of-sequence token. Optionally, use a list to set multiple end-of-sequ


BeamSearch

ence tokens.   beam_indices (torch.LongTensor, optional) —
Beam indices indicating to which beam hypothesis each token correspond.   Returns
UserDict

A dictionary composed of the fields as defined above:
next_beam_scores (torch.FloatTensor of shape (batch_size * num_beams)) — Updated scores of
all
non-finished beams.
next_beam_tokens (torch.FloatTensor of shape (batch_size * num_beams)) — Next tokens to be
added
to the non-finished beam_hypotheses.
next_beam_indices (torch.FloatTensor of shape (batch_size * num_beams)) — Beam indices
indicating to which beam the next tokens shall be added.
   finalize  < source > ( input_ids: LongTensor final_beam_scores: FloatTensor final_beam_tokens: LongTensor final_beam_indices: LongTensor max_length: int pad_token_id: typing.Optional[int] = None eos_token_id: typing.Union[int, typing.List[int], NoneType] = None beam_indices: typing.Optional[torch.LongTensor] = No


Utilities

transformers.top_k_top_p_filtering  < source > ( logits: FloatTensor top_k: int = 0 top_p: float = 1.0 filter_value: float = -inf min_tokens_to_keep: int = 1  )   Parameters   top_k (int, optional, defaults to 0) —
If > 0, only keep the top k tokens with highest probability (top-k filtering)   top_p (float, optional, defaults to 1.0) —
If < 1.0, only keep the top tokens with cumulative probability >= top_p (nucleus filtering). Nucleus
filtering is described in Holtzman et al. (http://arxiv.org/abs/1904.09751)   min_tokens_to_keep (int, optional, defaults to 1) —
Minimumber of tokens we keep per batch example in the output.    Filter a distribution of logits using top-k and/or nucleus (top-p) filtering From: https://gist.github.com/thomwolf/1a5a29f6962089e8


Utilities

71b94cbd09daf317
 transformers.tf_top_k_top_p_filtering  < source > ( logits top_k = 0 top_p = 1.0 filter_value = -inf min_tokens_to_keep = 1  )   Parameters   top_k (int, optional, defaults to 0) —
If > 0, only keep the top k tokens with highest probability (top-k filtering)   top_p (float, optional, defaults to 1.0) —
If < 1.0, only keep the top tokens with cumulative probability >= top_p (nucleus filtering). Nucleus
filtering is described in Holtzman et al. (http://arxiv.org/abs/1904.09751)   min_tokens_to_keep (int, optional, defaults to 1) —
Minimumber of tokens we keep per batch example in the output.    Filter a distribution of logits using top-k and/or nucleus (top-p) filtering From: https://gist.github.com/thomwolf/1a5a29f6962089e871b94cbd09daf317


Streamers

class transformers.TextStreamer  < source > ( tokenizer: AutoTokenizer skip_prompt: bool = False **decode_kwargs  )   Parameters   tokenizer (AutoTokenizer) —
The tokenized used to decode the tokens.   skip_prompt (bool, optional, defaults to False) —
Whether to skip the prompt to .generate() or not. Useful e.g. for chatbots.   decode_kwargs (dict, optional) —
Additional keyword arguments to pass to the tokenizer’s decode method.    Simple text streamer that prints the token(s) to stdout as soon as entire words are formed. The API for the streamer classes is still under development and may change in the future.  Examples:   Copied >>> from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
>>> tok = AutoTokenizer.from_pretrained("gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> inputs = tok(["An increasing sequence: one,"], return_tensors=


Streamers

"pt")
>>> streamer = TextStreamer(tok)
>>> # Despite returning the usual output, the streamer will also print the generated text to stdout.
>>> _ = model.generate(**inputs, streamer=streamer, max_new_tokens=20)
An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,  end  < source > ( )    Flushes any remaining cache and prints a newline to stdout.  on_finalized_text  < source > ( text: str stream_end: bool = False  )    Prints the new text to stdout. If the stream is ending, also prints a newline.  put  < source > ( value  )    Receives tokens, decodes them, and prints them to stdout as soon as they form entire words.
 class transformers.TextIteratorStreamer  < source > ( tokenizer: AutoTokenizer skip_prompt: bool = False timeout: typing.Optional[float] = None **decode_kwargs  )   Parameters   tokenizer (AutoTokenizer) —
The tokenized used t


Streamers

o decode the tokens.   skip_prompt (bool, optional, defaults to False) —
Whether to skip the prompt to .generate() or not. Useful e.g. for chatbots.   timeout (float, optional) —
The timeout for the text queue. If None, the queue will block indefinitely. Useful to handle exceptions
in .generate(), when it is called in a separate thread.   decode_kwargs (dict, optional) —
Additional keyword arguments to pass to the tokenizer’s decode method.    Streamer that stores print-ready text in a queue, to be used by a downstream application as an iterator. This is
useful for applications that benefit from acessing the generated text in a non-blocking way (e.g. in an interactive
Gradio demo). The API for the streamer classes is still under development and may change in the future.  Examples:   Copied >>> from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer
>


Streamers

>> from threading import Thread
>>> tok = AutoTokenizer.from_pretrained("gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> inputs = tok(["An increasing sequence: one,"], return_tensors="pt")
>>> streamer = TextIteratorStreamer(tok)
>>> # Run the generation in a separate thread, so that we can fetch the generated text in a non-blocking way.
>>> generation_kwargs = dict(inputs, streamer=streamer, max_new_tokens=20)
>>> thread = Thread(target=model.generate, kwargs=generation_kwargs)
>>> thread.start()
>>> generated_text = ""
>>> for new_text in streamer:
...     generated_text += new_text
>>> generated_text
'An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,'  on_finalized_text  < source > ( text: str stream_end: bool = False  )    Put the new text in the queue. If the stream is ending, also put a stop signal in the queue


General Utilities

This page lists all of Transformers general utility functions that are found in the file utils.py.
Most of those are only useful if you are studying the general code in the library.


Enums and namedtuples

class transformers.utils.ExplicitEnum  < source > ( value names = None module = None qualname = None type = None start = 1  )    Enum with more explicit error message for missing values.
 class transformers.utils.PaddingStrategy  < source > ( value names = None module = None qualname = None type = None start = 1  )    Possible values for the padding argument in PreTrainedTokenizerBase.call(). Useful for tab-completion in an
IDE.
 class transformers.TensorType  < source > ( value names = None module = None qualname = None type = None start = 1  )    Possible values for the return_tensors argument in PreTrainedTokenizerBase.call(). Useful for
tab-completion in an IDE.


Special Decorators

transformers.add_start_docstrings  < source > ( *docstr  )
 transformers.utils.add_start_docstrings_to_model_forward  < source > ( *docstr  )
 transformers.add_end_docstrings  < source > ( *docstr  )
 transformers.utils.add_code_sample_docstrings  < source > ( *docstr processor_class = None checkpoint = None output_type = None config_class = None mask = '[MASK]' qa_target_start_index = 14 qa_target_end_index = 15 model_cls = None modality = None expected_output = None expected_loss = None real_checkpoint = None  )
 transformers.utils.replace_return_docstrings  < source > ( output_type = None config_class = None  )


Special Properties

class transformers.utils.cached_property  < source > ( fget = None fset = None fdel = None doc = None  )    Descriptor that mimics @property but caches output in member variable. From tensorflow_datasets Built-in in functools from Python 3.8.


Other Utilities

class transformers.utils._LazyModule  < source > ( name module_file import_structure module_spec = None extra_objects = None  )    Module class that surfaces all objects but only performs associated imports when the objects are requested.


Utilities for Tokenizers

This page lists all the utility functions used by the tokenizers, mainly the class
PreTrainedTokenizerBase that implements the common methods between
PreTrainedTokenizer and PreTrainedTokenizerFast and the mixin
SpecialTokensMixin.
Most of those are only useful if you are studying the code of the tokenizers in the library.