GenerationMixin

oken_id (int, optional) —
The id of the padding token.   eos_token_id (Union[int, List[int]], optional) —
The id of the end-of-sequence token. Optionally, use a list to set multiple end-of-sequence tokens.   output_attentions (bool, optional, defaults to False) —
Whether or not to return the attentions tensors of all attention layers. See attentions under
returned tensors for more details.   output_hidden_states (bool, optional, defaults to False) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors
for more details.   output_scores (bool, optional, defaults to False) —
Whether or not to return the prediction scores. See scores under returned tensors for more details.   return_dict_in_generate (bool, optional, defaults to False) —
Whether or not to return a ModelOutput instead of a plain tuple.   synced_gpus (bool, optional, defaults to False) —
Whether to continue running the while loop until max_length (needed for ZeRO stage


GenerationMixin

 3)
model_kwargs —
Additional model specific kwargs will be forwarded to the forward function of the model. If model is
an encoder-decoder model the kwargs should include encoder_outputs.    Generates sequences of token ids for models with a language modeling head using beam search multinomial
sampling and can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models. In most cases, you do not need to call beam_sample() directly. Use generate()
instead. For an overview of generation strategies and code examples, check the following
guide.  Examples:   Copied >>> from transformers import (
...     AutoTokenizer,
...     AutoModelForSeq2SeqLM,
...     LogitsProcessorList,
...     MinLengthLogitsProcessor,
...     TopKLogitsWarper,
...     TemperatureLogitsWarper,
...     BeamSearchScorer,
... )
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
>>> encoder_input_str = "


GenerationMixin

translate English to German: How old are you?"
>>> encoder_input_ids = tokenizer(encoder_input_str, return_tensors="pt").input_ids
>>> # lets run beam search using 3 beams
>>> num_beams = 3
>>> # define decoder start token ids
>>> input_ids = torch.ones((num_beams, 1), device=model.device, dtype=torch.long)
>>> input_ids = input_ids * model.config.decoder_start_token_id
>>> # add encoder_outputs to model keyword arguments
>>> model_kwargs = {
...     "encoder_outputs": model.get_encoder()(
...         encoder_input_ids.repeat_interleave(num_beams, dim=0), return_dict=True
...     )
... }
>>> # instantiate beam scorer
>>> beam_scorer = BeamSearchScorer(
...     batch_size=1,
...     max_length=model.config.max_length,
...     num_beams=num_beams,
...     device=model.device,
... )
>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList(
...     [MinLengthLogitsProcessor(5, eos_token_id=model.config.eos_token_id)]
... )
>>> # instantiate logits processo


GenerationMixin

rs
>>> logits_warper = LogitsProcessorList(
...     [
...         TopKLogitsWarper(50),
...         TemperatureLogitsWarper(0.7),
...     ]
... )
>>> outputs = model.beam_sample(
...     input_ids, beam_scorer, logits_processor=logits_processor, logits_warper=logits_warper, **model_kwargs
... )
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Wie alt bist du?']  contrastive_search  < source > ( input_ids: LongTensor top_k: typing.Optional[int] = 1 penalty_alpha: typing.Optional[float] = 0 logits_processor: typing.Optional[transformers.generation.logits_process.LogitsProcessorList] = None logits_warper: typing.Optional[transformers.generation.logits_process.LogitsProcessorList] = None stopping_criteria: typing.Optional[transformers.generation.stopping_criteria.StoppingCriteriaList] = None pad_token_id: typing.Optional[int] = None eos_token_id: typing.Union[int, typing.List[int], NoneType] = None output_attentions: typing.Optional[bool] = None output_hidden_sta


GenerationMixin

tes: typing.Optional[bool] = None output_scores: typing.Optional[bool] = None return_dict_in_generate: typing.Optional[bool] = None synced_gpus: bool = False streamer: typing.Optional[ForwardRef('BaseStreamer')] = None sequential: typing.Optional[bool] = None **model_kwargs  )   Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
The sequence used as a prompt for the generation.   top_k (int, optional, defaults to 1) —
The size of the candidate set that is used to re-rank for contrastive search   penalty_alpha (float, optional, defaults to 0) —
The degeneration penalty for contrastive search; activate when it is larger than 0   logits_processor (LogitsProcessorList, optional) —
An instance of LogitsProcessorList. List of instances of class derived from LogitsProcessor
used to modify the prediction scores of the language modeling head applied at each generation step.   logits_warper (LogitsProcessorList, optional) —
An instance of LogitsProcess


GenerationMixin

orList. List of instances of class derived from LogitsWarper used
to warp the prediction score distribution of the language modeling head applied before multinomial
sampling at each generation step.   stopping_criteria (StoppingCriteriaList, optional) —
An instance of StoppingCriteriaList. List of instances of class derived from StoppingCriteria
used to tell if the generation loop should stop.   pad_token_id (int, optional) —
The id of the padding token.   eos_token_id (Union[int, List[int]], optional) —
The id of the end-of-sequence token. Optionally, use a list to set multiple end-of-sequence tokens.   output_attentions (bool, optional, defaults to False) —
Whether or not to return the attentions tensors of all attention layers. See attentions under
returned tensors for more details.   output_hidden_states (bool, optional, defaults to False) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors
for more details.   output_scor


GenerationMixin

es (bool, optional, defaults to False) —
Whether or not to return the prediction scores. See scores under returned tensors for more details.   return_dict_in_generate (bool, optional, defaults to False) —
Whether or not to return a ModelOutput instead of a plain tuple.   synced_gpus (bool, optional, defaults to False) —
Whether to continue running the while loop until max_length (needed for ZeRO stage 3)   streamer (BaseStreamer, optional) —
Streamer object that will be used to stream the generated sequences. Generated tokens are passed
through streamer.put(token_ids) and the streamer is responsible for any further processing.   sequential (bool, optional) —
Switches topk hidden state computation from parallel to sequential to reduce memory if True.
model_kwargs —
Additional model specific keyword arguments will be forwarded to the forward function of the model.
If model is an encoder-decoder model the kwargs should include encoder_outputs.    Generates sequences of token i


GenerationMixin

ds for models with a language modeling head using contrastive search and can
be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models. In most cases, you do not need to call contrastive_search() directly. Use
generate() instead. For an overview of generation strategies and code examples, check the following
guide.  Examples:   Copied >>> from transformers import (
...     AutoTokenizer,
...     AutoModelForCausalLM,
...     StoppingCriteriaList,
...     MaxLengthCriteria,
... )
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m")
>>> model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m")
>>> # set pad_token_id to eos_token_id because OPT does not have a PAD token
>>> model.config.pad_token_id = model.config.eos_token_id
>>> input_prompt = "DeepMind Company is"
>>> input_ids = tokenizer(input_prompt, return_tensors="pt")
>>> stopping_criteria = StoppingCriteriaList([MaxLengthCriteria(max_length=64)])
>>> outputs = model.contr


GenerationMixin

astive_search(
...     **input_ids, penalty_alpha=0.6, top_k=4, stopping_criteria=stopping_criteria
... )
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['DeepMind Company is a company that focuses on the development and commercialization of artificial intelligence (AI). DeepMind’s mission is to help people understand and solve problems that are difficult to solve in the world today.\n\nIn this post, we talk about the benefits of deep learning in business and how it']  group_beam_search  < source > ( input_ids: LongTensor beam_scorer: BeamScorer logits_processor: typing.Optional[transformers.generation.logits_process.LogitsProcessorList] = None stopping_criteria: typing.Optional[transformers.generation.stopping_criteria.StoppingCriteriaList] = None max_length: typing.Optional[int] = None pad_token_id: typing.Optional[int] = None eos_token_id: typing.Union[int, typing.List[int], NoneType] = None output_attentions: typing.Optional[bool] = None output_hidden_sta


GenerationMixin

tes: typing.Optional[bool] = None output_scores: typing.Optional[bool] = None return_dict_in_generate: typing.Optional[bool] = None synced_gpus: bool = False **model_kwargs  )   Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
The sequence used as a prompt for the generation.   beam_scorer (BeamScorer) —
An derived instance of BeamScorer that defines how beam hypotheses are constructed, stored and
sorted during generation. For more information, the documentation of BeamScorer should be read.   logits_processor (LogitsProcessorList, optional) —
An instance of LogitsProcessorList. List of instances of class derived from LogitsProcessor
used to modify the prediction scores of the language modeling head applied at each generation step.   stopping_criteria (StoppingCriteriaList, optional) —
An instance of StoppingCriteriaList. List of instances of class derived from StoppingCriteria
used to tell if the generation loop should stop.   max_length (


GenerationMixin

int, optional, defaults to 20) —
DEPRECATED. Use logits_processor or stopping_criteria directly to cap the number of generated
tokens. The maximum length of the sequence to be generated.   pad_token_id (int, optional) —
The id of the padding token.   eos_token_id (Union[int, List[int]], optional) —
The id of the end-of-sequence token. Optionally, use a list to set multiple end-of-sequence tokens.   output_attentions (bool, optional, defaults to False) —
Whether or not to return the attentions tensors of all attention layers. See attentions under
returned tensors for more details.   output_hidden_states (bool, optional, defaults to False) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors
for more details.   output_scores (bool, optional, defaults to False) —
Whether or not to return the prediction scores. See scores under returned tensors for more details.   return_dict_in_generate (bool, optional, defaults to False) —
Wheth


GenerationMixin

er or not to return a ModelOutput instead of a plain tuple.   synced_gpus (bool, optional, defaults to False) —
Whether to continue running the while loop until max_length (needed for ZeRO stage 3)
model_kwargs —
Additional model specific kwargs that will be forwarded to the forward function of the model. If
model is an encoder-decoder model the kwargs should include encoder_outputs.    Generates sequences of token ids for models with a language modeling head using diverse beam search
decoding and can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models. In most cases, you do not need to call group_beam_search() directly. Use
generate() instead. For an overview of generation strategies and code examples, check the following
guide.  Examples:   Copied >>> from transformers import (
...     AutoTokenizer,
...     AutoModelForSeq2SeqLM,
...     LogitsProcessorList,
...     MinLengthLogitsProcessor,
...     HammingDiversityLogitsProcessor,
...     B


GenerationMixin

eamSearchScorer,
... )
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
>>> encoder_input_str = "translate English to German: How old are you?"
>>> encoder_input_ids = tokenizer(encoder_input_str, return_tensors="pt").input_ids
>>> # lets run diverse beam search using 6 beams
>>> num_beams = 6
>>> # define decoder start token ids
>>> input_ids = torch.ones((num_beams, 1), device=model.device, dtype=torch.long)
>>> input_ids = input_ids * model.config.decoder_start_token_id
>>> # add encoder_outputs to model keyword arguments
>>> model_kwargs = {
...     "encoder_outputs": model.get_encoder()(
...         encoder_input_ids.repeat_interleave(num_beams, dim=0), return_dict=True
...     )
... }
>>> # instantiate beam scorer
>>> beam_scorer = BeamSearchScorer(
...     batch_size=1,
...     max_length=model.config.max_length,
...     num_beams=num_beams,
...     device=model.device,
...     num_


GenerationMixin

beam_groups=3,
... )
>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList(
...     [
...         HammingDiversityLogitsProcessor(5.5, num_beams=6, num_beam_groups=3),
...         MinLengthLogitsProcessor(5, eos_token_id=model.config.eos_token_id),
...     ]
... )
>>> outputs = model.group_beam_search(
...     input_ids, beam_scorer, logits_processor=logits_processor, **model_kwargs
... )
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Wie alt bist du?']  constrained_beam_search  < source > ( input_ids: LongTensor constrained_beam_scorer: ConstrainedBeamSearchScorer logits_processor: typing.Optional[transformers.generation.logits_process.LogitsProcessorList] = None stopping_criteria: typing.Optional[transformers.generation.stopping_criteria.StoppingCriteriaList] = None max_length: typing.Optional[int] = None pad_token_id: typing.Optional[int] = None eos_token_id: typing.Union[int, typing.List[int], NoneType] = None output_attentions:


GenerationMixin

 typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None output_scores: typing.Optional[bool] = None return_dict_in_generate: typing.Optional[bool] = None synced_gpus: typing.Optional[bool] = None **model_kwargs  )   Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
The sequence used as a prompt for the generation.   constrained_beam_scorer (ConstrainedBeamSearchScorer) —
A derived instance of BeamScorer that defines how beam hypotheses are constructed, stored and
sorted during generation, while satisfying a list of positive constraints. For more information, the
documentation of ConstrainedBeamSearchScorer should be read.   logits_processor (LogitsProcessorList, optional) —
An instance of LogitsProcessorList. List of instances of class derived from LogitsProcessor
used to modify the prediction scores of the language modeling head applied at each generation step.   stopping_criteria (StoppingCriteriaList, optional) —


GenerationMixin

An instance of StoppingCriteriaList. List of instances of class derived from StoppingCriteria
used to tell if the generation loop should stop.   logits_warper (LogitsProcessorList, optional) —
An instance of LogitsProcessorList. List of instances of class derived from LogitsWarper used
to warp the prediction score distribution of the language modeling head applied before multinomial
sampling at each generation step.   max_length (int, optional, defaults to 20) —
DEPRECATED. Use logits_processor or stopping_criteria directly to cap the number of generated
tokens. The maximum length of the sequence to be generated.   pad_token_id (int, optional) —
The id of the padding token.   eos_token_id (Union[int, List[int]], optional) —
The id of the end-of-sequence token. Optionally, use a list to set multiple end-of-sequence tokens.   output_attentions (bool, optional, defaults to False) —
Whether or not to return the attentions tensors of all attention layers. See attentions under
re


GenerationMixin

turned tensors for more details.   output_hidden_states (bool, optional, defaults to False) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors
for more details.   output_scores (bool, optional, defaults to False) —
Whether or not to return the prediction scores. See scores under returned tensors for more details.   return_dict_in_generate (bool, optional, defaults to False) —
Whether or not to return a ModelOutput instead of a plain tuple.   synced_gpus (bool, optional, defaults to False) —
Whether to continue running the while loop until max_length (needed for ZeRO stage 3)
model_kwargs —
Additional model specific kwargs will be forwarded to the forward function of the model. If model is
an encoder-decoder model the kwargs should include encoder_outputs.    Generates sequences of token ids for models with a language modeling head using constrained beam search
decoding and can be used for text-decoder, text-to-text, speech-t


GenerationMixin

o-text, and vision-to-text models. In most cases, you do not need to call constrained_beam_search() directly. Use
generate() instead. For an overview of generation strategies and code examples, check the following
guide.  Examples:   Copied >>> from transformers import (
...     AutoTokenizer,
...     AutoModelForSeq2SeqLM,
...     LogitsProcessorList,
...     MinLengthLogitsProcessor,
...     ConstrainedBeamSearchScorer,
...     PhrasalConstraint,
... )
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
>>> encoder_input_str = "translate English to German: How old are you?"
>>> encoder_input_ids = tokenizer(encoder_input_str, return_tensors="pt").input_ids
>>> # lets run beam search using 3 beams
>>> num_beams = 3
>>> # define decoder start token ids
>>> input_ids = torch.ones((num_beams, 1), device=model.device, dtype=torch.long)
>>> input_ids = input_ids * model.config.decoder_start_token


GenerationMixin

_id
>>> # add encoder_outputs to model keyword arguments
>>> model_kwargs = {
...     "encoder_outputs": model.get_encoder()(
...         encoder_input_ids.repeat_interleave(num_beams, dim=0), return_dict=True
...     )
... }
>>> constraint_str = "Sie"
>>> constraint_token_ids = tokenizer.encode(constraint_str)[:-1]  # slice to remove eos token
>>> constraints = [PhrasalConstraint(token_ids=constraint_token_ids)]
>>> # instantiate beam scorer
>>> beam_scorer = ConstrainedBeamSearchScorer(
...     batch_size=1, num_beams=num_beams, device=model.device, constraints=constraints
... )
>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList(
...     [
...         MinLengthLogitsProcessor(5, eos_token_id=model.config.eos_token_id),
...     ]
... )
>>> outputs = model.constrained_beam_search(
...     input_ids, beam_scorer, constraints=constraints, logits_processor=logits_processor, **model_kwargs
... )
>>> tokenizer.batch_decode(outputs, skip_special_tokens


TFGenerationMixin

class transformers.TFGenerationMixin  < source > ( )    A class containing all of the functions supporting generation, to be used as a mixin in TFPreTrainedModel. The class exposes generate(), which can be used for: greedy decoding by calling greedy_search() if num_beams=1 and
do_sample=False contrastive search by calling contrastive_search() if penalty_alpha>0 and
top_k>1 multinomial sampling by calling sample() if num_beams=1 and
do_sample=True beam-search decoding by calling beam_search() if num_beams>1 You do not need to call any of the above methods directly. Pass custom parameter values to ‘generate’ instead. To
learn more about decoding strategies refer to the text generation strategies guide.  generate  < source > ( inputs: typing.Optional[tensorflow.python.framework.ops.Tensor] = None generation_config: typing.Optional[transformers.generation.configuration_utils.GenerationConfig] = None logits_processor: typing.Optional[transformers.generation.tf_logits_process.TFLogitsPro


TFGenerationMixin

cessorList] = None seed = None **kwargs  ) → ModelOutput or tf.Tensor  Parameters   inputs (tf.Tensor of varying shape depending on the modality, optional) —
The sequence used as a prompt for the generation or as model inputs to the encoder. If None the
method initializes it with bos_token_id and a batch size of 1. For decoder-only models inputs
should of in the format of input_ids. For encoder-decoder models inputs can represent any of
input_ids, input_values, input_features, or pixel_values.   generation_config (~generation.GenerationConfig, optional) —
The generation configuration to be used as base parametrization for the generation call. **kwargs
passed to generate matching the attributes of generation_config will override them. If
generation_config is not provided, the default will be used, which had the following loading
priority: 1) from the generation_config.json model file, if it exists; 2) from the model
configuration. Please note that unspecified parameters will inherit


TFGenerationMixin

 GenerationConfig’s
default values, whose documentation should be checked to parameterize generation.   logits_processor (LogitsProcessorList, optional) —
Custom logits processors that complement the default logits processors built from arguments and
generation config. If a logit processor is passed that is already created with the arguments or a
generation config an error is thrown. This feature is intended for advanced users.   seed (List[int], optional) —
Random seed to control sampling, containing two integers, used when do_sample is True. See the
seed argument from stateless functions in tf.random.   kwargs (Dict[str, Any], optional) —
Ad hoc parametrization of generate_config and/or additional model-specific kwargs that will be
forwarded to the forward function of the model. If the model is an encoder-decoder model, encoder
specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with decoder_.   Returns
ModelOutput or tf.Tensor

A ModelOutput


TFGenerationMixin

(if return_dict_in_generate=True or when
config.return_dict_in_generate=True) or a tf.Tensor.
If the model is not an encoder-decoder model (model.config.is_encoder_decoder=False), the possible
ModelOutput types are:
TFGreedySearchDecoderOnlyOutput,
TFSampleDecoderOnlyOutput,
TFBeamSearchDecoderOnlyOutput,
TFBeamSampleDecoderOnlyOutput
If the model is an encoder-decoder model (model.config.is_encoder_decoder=True), the possible
ModelOutput types are:
TFGreedySearchEncoderDecoderOutput,
TFSampleEncoderDecoderOutput,
TFBeamSearchEncoderDecoderOutput,
TFBeamSampleEncoderDecoderOutput
  Generates sequences of token ids for models with a language modeling head. Most generation-controlling parameters are set in generation_config which, if not passed, will be set to the
model’s default generation configuration. You can override any generation_config by passing the corresponding
parameters to generate, e.g. .generate(inputs, num_beams=4, do_sample=True). For an overview of generation strate


TFGenerationMixin

gies and code examples, check out the following
guide.  compute_transition_scores  < source > ( sequences: Tensor scores: typing.Tuple[tensorflow.python.framework.ops.Tensor] beam_indices: typing.Optional[tensorflow.python.framework.ops.Tensor] = None normalize_logits: bool = False  ) → tf.Tensor  Parameters   sequences (tf.Tensor) —
The generated sequences. The second dimension (sequence_length) is either equal to max_length or
shorter if all batches finished early due to the eos_token_id.   scores (tuple(tf.Tensor)) —
Transition scores for each vocabulary token at each generation step. Beam transition scores consisting
of log probabilities of tokens conditioned on log softmax of previously generated tokens Tuple of
tf.Tensor with up to max_new_tokens elements (one element for each generated token), with each
tensor of shape (batch_size*num_beams, config.vocab_size).   beam_indices (tf.Tensor, optional) —
Beam indices of generated token id at each generation step. tf.Tensor of sha


TFGenerationMixin

pe
(batch_size*num_return_sequences, sequence_length). Only required if a num_beams>1 at
generate-time.   normalize_logits (bool, optional, defaults to False) —
Whether to normalize the logits (which, for legacy reasons, may be unnormalized).   Returns
tf.Tensor

A tf.Tensor of shape (batch_size*num_return_sequences, sequence_length) containing
the transition scores (logits)
  Computes the transition scores of sequences given the generation scores (and beam indices, if beam search was
used). This is a convenient method to quicky obtain the scores of the selected tokens at generation time.  Examples:   Copied >>> from transformers import GPT2Tokenizer, TFAutoModelForCausalLM
>>> import numpy as np
>>> tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
>>> model = TFAutoModelForCausalLM.from_pretrained("gpt2")
>>> tokenizer.pad_token_id = tokenizer.eos_token_id
>>> inputs = tokenizer(["Today is"], return_tensors="tf")
>>> # Example 1: Print the scores for each token generated with


TFGenerationMixin

Greedy Search
>>> outputs = model.generate(**inputs, max_new_tokens=5, return_dict_in_generate=True, output_scores=True)
>>> transition_scores = model.compute_transition_scores(
...     outputs.sequences, outputs.scores, normalize_logits=True
... )
>>> # input_length is the length of the input prompt for decoder-only models, like the GPT family, and 1 for
>>> # encoder-decoder models, like BART or T5.
>>> input_length = 1 if model.config.is_encoder_decoder else inputs.input_ids.shape[1]
>>> generated_tokens = outputs.sequences[:, input_length:]
>>> for tok, score in zip(generated_tokens[0], transition_scores[0]):
...     # | token | token string | logits | probability
...     print(f"| {tok:5d} | {tokenizer.decode(tok):8s} | {score.numpy():.3f} | {np.exp(score.numpy()):.2%}")
|   262 |  the     | -1.413 | 24.33%
|  1110 |  day     | -2.609 | 7.36%
|   618 |  when    | -2.009 | 13.41%
|   356 |  we      | -1.859 | 15.58%
|   460 |  can     | -2.508 | 8.14%
>>> # Example 2: Reconstru


TFGenerationMixin

ct the sequence scores from Beam Search
>>> outputs = model.generate(
...     **inputs,
...     max_new_tokens=5,
...     num_beams=4,
...     num_return_sequences=4,
...     return_dict_in_generate=True,
...     output_scores=True,
... )
>>> transition_scores = model.compute_transition_scores(
...     outputs.sequences, outputs.scores, outputs.beam_indices, normalize_logits=False
... )
>>> # If you sum the generated tokens' scores and apply the length penalty, you'll get the sequence scores.
>>> # Tip: recomputing the scores is only guaranteed to match with `normalize_logits=False`. Depending on the
>>> # use case, you might want to recompute it with `normalize_logits=True`.
>>> output_length = input_length + np.sum(transition_scores.numpy() < 0, axis=1)
>>> length_penalty = model.generation_config.length_penalty
>>> reconstructed_scores = np.sum(transition_scores, axis=1) / (output_length**length_penalty)
>>> print(np.allclose(outputs.sequences_scores, reconstructed_scores))
True


FlaxGenerationMixin

class transformers.FlaxGenerationMixin  < source > ( )    A class containing all functions for auto-regressive text generation, to be used as a mixin in
FlaxPreTrainedModel. The class exposes generate(), which can be used for: greedy decoding by calling _greedy_search() if num_beams=1 and
do_sample=False multinomial sampling by calling _sample() if num_beams=1 and
do_sample=True beam-search decoding by calling _beam_search() if num_beams>1 and
do_sample=False You do not need to call any of the above methods directly. Pass custom parameter values to ‘generate’ instead. To
learn more about decoding strategies refer to the text generation strategies guide.  generate  < source > ( input_ids: Array generation_config: typing.Optional[transformers.generation.configuration_utils.GenerationConfig] = None prng_key: typing.Optional[jax.Array] = None trace: bool = True params: typing.Union[typing.Dict[str, jax.Array], NoneType] = None logits_proc


FlaxGenerationMixin

essor: typing.Optional[transformers.generation.flax_logits_process.FlaxLogitsProcessorList] = None **kwargs  )   Parameters   input_ids (jnp.ndarray of shape (batch_size, sequence_length)) —
The sequence used as a prompt for the generation.   generation_config (~generation.GenerationConfig, optional) —
The generation configuration to be used as base parametrization for the generation call. **kwargs
passed to generate matching the attributes of generation_config will override them. If
generation_config is not provided, the default will be used, which had the following loading
priority: 1) from the generation_config.json model file, if it exists; 2) from the model
configuration. Please note that unspecified parameters will inherit GenerationConfig’s
default values, whose documentation should be checked to parameterize generation.   trace (bool, optional, defaults to True) —
Whether to trace generation. Setting trace=False should only be


FlaxGenerationMixin

 used for debugging and will lead to a
considerably slower runtime.   params (Dict[str, jnp.ndarray], optional) —
Optionally the model parameters can be passed. Can be useful for parallelized generation.   logits_processor (FlaxLogitsProcessorList , optional) —
Custom logits processors that complement the default logits processors built from arguments and
generation config. If a logit processor is passed that is already created with the arguments or a
generation config an error is thrown. This feature is intended for advanced users.   kwargs (Dict[str, Any], optional) —
Ad hoc parametrization of generate_config and/or additional model-specific kwargs that will be
forwarded to the forward function of the model. If the model is an encoder-decoder model, encoder
specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with decoder_.    Generates sequences of token ids for models with a language modeling head.


Logging

🤗 Transformers has a centralized logging system, so that you can setup the verbosity of the library easily.
Currently the default verbosity of the library is WARNING.
To change the level of verbosity, just use one of the direct setters. For instance, here is how to change the verbosity
to the INFO level.
  Copied import transformers
transformers.logging.set_verbosity_info()
You can also use the environment variable TRANSFORMERS_VERBOSITY to override the default verbosity. You can set it
to one of the following: debug, info, warning, error, critical. For example:
  Copied TRANSFORMERS_VERBOSITY=error ./myprogram.py
Additionally, some warnings can be disabled by setting the environment variable
TRANSFORMERS_NO_ADVISORY_WARNING


Logging

S to a true value, like 1. This will disable any warning that is logged using
logger.warning_advice. For example:
  Copied TRANSFORMERS_NO_ADVISORY_WARNINGS=1 ./myprogram.py
Here is an example of how to use the same logger as the library in your own module or script:
  Copied from transformers.utils import logging
logging.set_verbosity_info()
logger = logging.get_logger("transformers")
logger.info("INFO")
logger.warning("WARN")
All the methods of this logging module are documented below, the main ones are
logging.get_verbosity() to get the current level of verbosity in the logger and
logging.set_verbosity() to set the verbosity to the level of your choice. In order (from the least
verbose to the most verbose), those levels (


Logging

with their corresponding int values in parenthesis) are:
transformers.logging.CRITICAL or transformers.logging.FATAL (int value, 50): only report the most
critical errors. transformers.logging.ERROR (int value, 40): only report errors. transformers.logging.WARNING or transformers.logging.WARN (int value, 30): only reports error and
warnings. This the default level used by the library. transformers.logging.INFO (int value, 20): reports error, warnings and basic information. transformers.logging.DEBUG (int value, 10): report all information.
By default, tqdm progress bars will be displayed during model download. logging.disable_progress_bar() and logging.enable_progress_bar() can be used to suppress or unsuppress this behavior


Base setters

transformers.utils.logging.set_verbosity_error  < source > ( )    Set the verbosity to the ERROR level.
 transformers.utils.logging.set_verbosity_warning  < source > ( )    Set the verbosity to the WARNING level.
 transformers.utils.logging.set_verbosity_info  < source > ( )    Set the verbosity to the INFO level.
 transformers.utils.logging.set_verbosity_debug  < source > ( )    Set the verbosity to the DEBUG level.


Other functions

transformers.utils.logging.get_verbosity  < source > ( ) → int  Returns
int

The logging level.
  Return the current level for the 🤗 Transformers’s root logger as an int. 🤗 Transformers has following logging levels: 50: transformers.logging.CRITICAL or transformers.logging.FATAL 40: transformers.logging.ERROR 30: transformers.logging.WARNING or transformers.logging.WARN 20: transformers.logging.INFO 10: transformers.logging.DEBUG
 transformers.utils.logging.set_verbosity  < source > ( verbosity: int  )   Parameters   verbosity (int) —
Logging level, e.g., one of:
transformers.logging.CRITICAL or transformers.logging.FATAL
transformers.logging.ERROR
transformers.logging.WARNING or


Other functions

 transformers.logging.WARN
transformers.logging.INFO
transformers.logging.DEBUG
    Set the verbosity level for the 🤗 Transformers’s root logger.
 transformers.utils.logging.get_logger  < source > ( name: typing.Optional[str] = None  )    Return a logger with the specified name. This function is not supposed to be directly accessed unless you are writing a custom transformers module.
 transformers.utils.logging.enable_default_handler  < source > ( )    Enable the default handler of the HuggingFace Transformers’s root logger.
 transformers.utils.logging.disable_default_handler  < source > ( )    Disable the default handler of the HuggingFace Transformers’s root logger.
 transformers.


Other functions

utils.logging.enable_explicit_format  < source > ( )     Enable explicit formatting for every HuggingFace Transformers’s logger. The explicit formatter is as follows:   Copied     [LEVELNAME|FILENAME|LINE NUMBER] TIME >> MESSAGE
All handlers currently bound to the root logger are affected by this method.
 transformers.utils.logging.reset_format  < source > ( )    Resets the formatting for HuggingFace Transformers’s loggers. All handlers currently bound to the root logger are affected by this method.
 transformers.utils.logging.enable_progress_bar  < source > ( )    Enable tqdm progress bar.
 transformers.utils.logging.disable_progress_bar  < source > ( )    Disable tqdm progress bar


Models

The base classes PreTrainedModel, TFPreTrainedModel, and
FlaxPreTrainedModel implement the common methods for loading/saving a model either from a local
file or directory, or from a pretrained model configuration provided by the library (downloaded from HuggingFace’s AWS
S3 repository).
PreTrainedModel and TFPreTrainedModel also implement a few methods which
are common among all the models to:
resize the input token embeddings when new tokens are added to the vocabulary prune the attention heads of the model.
The other methods that are common to each model are defined in ModuleUtilsMixin
(for the PyTorch models) and ~modeling_tf_utils.TFModuleUtilsMixin (for the TensorFlow models) or
for text generation, GenerationMixin (for the PyTorch models),
TFGenerationMixin (for the TensorFlow models) and
FlaxGenerationMixin (for the Flax/JAX models).


PreTrainedModel

class transformers.PreTrainedModel  < source > ( config: PretrainedConfig *inputs **kwargs  )    Base class for all models. PreTrainedModel takes care of storing the configuration of the models and handles methods for loading,
downloading and saving models as well as a few methods common to all models to: resize the input embeddings, prune heads in the self-attention heads. Class attributes (overridden by derived classes): config_class (PretrainedConfig) — A subclass of PretrainedConfig to use as configuration class
for this model architecture. load_tf_weights (Callable) — A python method for loading a TensorFlow checkpoint in a PyTorch model,
taking as arguments: model (PreTrainedModel) — An instance of the model on which to load the TensorFlow checkpoint. config (PreTrainedConfig) — An instance of the configuration associated to the model. path (str) — A path to the TensorFlow checkpoint. base_model_prefix (str) — A string indicating the attribute a


PreTrainedModel

ssociated to the base model in derived
classes of the same architecture adding modules on top of the base model. is_parallelizable (bool) — A flag indicating whether this model supports model parallelization. main_input_name (str) — The name of the principal input to the model (often input_ids for NLP
models, pixel_values for vision models and input_values for speech models).  push_to_hub  < source > ( repo_id: str use_temp_dir: typing.Optional[bool] = None commit_message: typing.Optional[str] = None private: typing.Optional[bool] = None token: typing.Union[bool, str, NoneType] = None max_shard_size: typing.Union[int, str, NoneType] = '10GB' create_pr: bool = False safe_serialization: bool = False revision: str = None **deprecated_kwargs  )   Parameters   repo_id (str) —
The name of the repository you want to push your model to. It should contain your organization name
when pushing to a given organization.   use_temp_dir (bool, optional) —
Whether or


PreTrainedModel

not to use a temporary directory to store the files saved before they are pushed to the Hub.
Will default to True if there is no directory named like repo_id, False otherwise.   commit_message (str, optional) —
Message to commit while pushing. Will default to "Upload model".   private (bool, optional) —
Whether or not the repository created should be private.   token (bool or str, optional) —
The token to use as HTTP bearer authorization for remote files. If True, will use the token generated
when running huggingface-cli login (stored in ~/.huggingface). Will default to True if repo_url
is not specified.   max_shard_size (int or str, optional, defaults to "10GB") —
Only applicable for models. The maximum size for a checkpoint before being sharded. Checkpoints shard
will then be each of size lower than this size. If expressed as a string, needs to be digits followed
by a unit (like "5MB").   create_pr (bool, optional, defaults to False) —
Whether or no


PreTrainedModel

t to create a PR with the uploaded files or directly commit.   safe_serialization (bool, optional, defaults to False) —
Whether or not to convert the model weights in safetensors format for safer serialization.   revision (str, optional) —
Branch to push the uploaded files to.    Upload the model file to the 🤗 Model Hub.  Examples:   Copied from transformers import AutoModel
model = AutoModel.from_pretrained("bert-base-cased")
# Push the model to your namespace with the name "my-finetuned-bert".
model.push_to_hub("my-finetuned-bert")
# Push the model to an organization with the name "my-finetuned-bert".
model.push_to_hub("huggingface/my-finetuned-bert")  can_generate  < source > ( ) → bool  Returns
bool

Whether this model can generate sequences with .generate().
  Returns whether this model can generate sequences with .generate().  disable_input_require_grads  < source > ( )    Removes the _require_grads_hook.  enable_input_require_grads  < source


PreTrainedModel

 > ( )    Enables the gradients for the input embeddings. This is useful for fine-tuning adapter weights while keeping
the model weights fixed.  from_pretrained  < source > ( pretrained_model_name_or_path: typing.Union[str, os.PathLike, NoneType] *model_args config: typing.Union[transformers.configuration_utils.PretrainedConfig, str, os.PathLike, NoneType] = None cache_dir: typing.Union[str, os.PathLike, NoneType] = None ignore_mismatched_sizes: bool = False force_download: bool = False local_files_only: bool = False token: typing.Union[bool, str, NoneType] = None revision: str = 'main' use_safetensors: bool = None **kwargs  )   Parameters   pretrained_model_name_or_path (str or os.PathLike, optional) —
Can be either:
A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like bert-base-uncased, or namespaced under a
user or organization name, like dbmdz/bert-base-ge


PreTrainedModel

rman-cased.
A path to a directory containing model weights saved using
save_pretrained(), e.g., ./my_model_directory/.
A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In
this case, from_tf should be set to True and a configuration object should be provided as
config argument. This loading path is slower than converting the TensorFlow checkpoint in a
PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
A path or url to a model folder containing a flax checkpoint file in .msgpack format (e.g,
./flax_model/ containing flax_model.msgpack). In this case, from_flax should be set to
True.
None if you are both providing the configuration and state dictionary (resp. with keyword
arguments config and state_dict).
   model_args (sequence of positional arguments, optional) —
All remaining positional arguments will be passed to the underlying model’s __init__ method.   config (Union[P


PreTrainedModel

retrainedConfig, str, os.PathLike], optional) —
Can be either:
an instance of a class derived from PretrainedConfig,
a string or path valid as input to from_pretrained().
Configuration for the model to use instead of an automatically loaded configuration. Configuration can
be automatically loaded when:
The model is a model provided by the library (loaded with the model id string of a pretrained
model).
The model was saved using save_pretrained() and is reloaded by supplying the
save directory.
The model is loaded by supplying a local directory as pretrained_model_name_or_path and a
configuration JSON file named config.json is found in the directory.
   state_dict (Dict[str, torch.Tensor], optional) —
A state dictionary to use instead of a state dictionary loaded from saved weights file.
This option can be used if you want to create a model from a pretrained configuration but load your own
weights. In this case though, you should check if using save_pr


PreTrainedModel

etrained() and
from_pretrained() is not a simpler option.   cache_dir (Union[str, os.PathLike], optional) —
Path to a directory in which a downloaded pretrained model configuration should be cached if the
standard cache should not be used.   from_tf (bool, optional, defaults to False) —
Load the model weights from a TensorFlow checkpoint save file (see docstring of
pretrained_model_name_or_path argument).   from_flax (bool, optional, defaults to False) —
Load the model weights from a Flax checkpoint save file (see docstring of
pretrained_model_name_or_path argument).   ignore_mismatched_sizes (bool, optional, defaults to False) —
Whether or not to raise an error if some of the weights from the checkpoint do not have the same size
as the weights of the model (if for instance, you are instantiating a model with 10 labels from a
checkpoint with 3 labels).   force_download (bool, optional, defaults to False) —
Whether or not to force the (re-)download of


PreTrainedModel

the model weights and configuration files, overriding the
cached versions if they exist.   resume_download (bool, optional, defaults to False) —
Whether or not to delete incompletely received files. Will attempt to resume the download if such a
file exists.   proxies (Dict[str, str], optional) —
A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.   output_loading_info(bool, optional, defaults to False) —
Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.   local_files_only(bool, optional, defaults to False) —
Whether or not to only look at local files (i.e., do not try to download the model).   token (str or bool, optional) —
The token to use as HTTP bearer authorization for remote files. If True, or not specified, will use
the token generated when running huggingface-cli login (stored


PreTrainedModel

in ~/.huggingface).   revision (str, optional, defaults to "main") —
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so revision can be any
identifier allowed by git.
To test a pull request you made on the Hub, you can pass `revision=“refs/pr/“.
   mirror (str, optional) —
Mirror source to accelerate downloads in China. If you are from China and have an accessibility
problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety.
Please refer to the mirror site for more information.   _fast_init(bool, optional, defaults to True) —
Whether or not to disable fast initialization.
One should only disable _fast_init to ensure backwards compatibility with transformers.__version__ < 4.6.0 for seeded model initialization. This argument will be removed at the next major version. See
pull reque


PreTrainedModel

st 11471 for more information.
   Parameters for big model inference   low_cpu_mem_usage(bool, optional) —
Tries to not use more than 1x model size in CPU memory (including peak memory) while loading the model.
This is an experimental feature and a subject to change at any moment.   torch_dtype (str or torch.dtype, optional) —
Override the default torch.dtype and load the model under a specific dtype. The different options
are:
torch.float16 or torch.bfloat16 or torch.float: load in a specified
dtype, ignoring the model’s config.torch_dtype if one exists. If not specified
the model will get loaded in torch.float (fp32).
"auto" - A torch_dtype entry in the config.json file of the model will be
attempted to be used. If this entry isn’t found then next check the dtype of the first weight in
the checkpoint that’s of a floating point type and use that as dtype. This will load the model
using the dtype it was saved in at the end of the training. It can’t be


PreTrainedModel

 used as an indicator of how
the model was trained. Since it could be trained in one of half precision dtypes, but saved in fp32.
For some models the dtype they were trained in is unknown - you may try to check the model’s paper or
reach out to the authors and ask them to add this information to the model’s card and to insert the
torch_dtype entry in config.json on the hub.
   device_map (str or Dict[str, Union[int, str, torch.device]] or int or torch.device, optional) —
A map that specifies where each submodule should go. It doesn’t need to be refined to each
parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the
same device. If we only pass the device (e.g., "cpu", "cuda:1", "mps", or a GPU ordinal rank
like 1) on which the model will be allocated, the device map will map the entire model to this
device. Passing device_map = 0 means put the whole model on GPU 0.
To have Accelerate compute the most optimi


PreTrainedModel

zed device_map automatically, set device_map="auto". For
more information about each option see designing a device
map.   max_memory (Dict, optional) —
A dictionary device identifier to maximum memory. Will default to the maximum memory available for each
GPU and the available CPU RAM if unset.   offload_folder (str or os.PathLike, optional) —
If the device_map contains any value "disk", the folder where we will offload weights.   offload_state_dict (bool, optional) —
If True, will temporarily offload the CPU state dict to the hard drive to avoid getting out of CPU
RAM if the weight of the CPU state dict + the biggest shard of the checkpoint does not fit. Defaults to
True when there is some disk offload.   load_in_8bit (bool, optional, defaults to False) —
If True, will convert the loaded model into mixed-8bit quantized model. To use this feature please
install bitsandbytes (pip install -U bitsandbytes).   load_in_4bit (bool, optional, defaults to Fal


PreTrainedModel

se) —
If True, will convert the loaded model into 4bit precision quantized model. To use this feature
install the latest version of bitsandbytes (pip install -U bitsandbytes).   quantization_config (Union[QuantizationConfigMixin,Dict], optional) —
A dictionary of configuration parameters or a QuantizationConfigMixin object for quantization (e.g
bitsandbytes, gptq)   subfolder (str, optional, defaults to "") —
In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can
specify the folder name here.   variant (str, optional) —
If specified load weights from variant filename, e.g. pytorch_model..bin. variant is
ignored when using from_tf or from_flax.   use_safetensors (bool, optional, defaults to None) —
Whether or not to use safetensors checkpoints. Defaults to None. If not specified and safetensors
is not installed, it will be set to False.   kwargs (remaining dictionary of keyword arguments, optional) —
Can


PreTrainedModel

be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
output_attentions=True). Behaves differently depending on whether a config is provided or
automatically loaded:
If a configuration is provided with config, **kwargs will be directly passed to the
underlying model’s __init__ method (we assume all relevant updates to the configuration have
already been done)
If a configuration is not provided, kwargs will be first passed to the configuration class
initialization function (from_pretrained()). Each key of kwargs that
corresponds to a configuration attribute will be used to override said attribute with the
supplied kwargs value. Remaining keys that do not correspond to any configuration attribute
will be passed to the underlying model’s __init__ function.
     Instantiate a pretrained pytorch model from a pre-trained model configuration. The model is set in evaluation mode by default using model.eval() (Dropout


PreTrainedModel

 modules are deactivated). To train
the model, you should first set it back in training mode with model.train(). The warning Weights from XXX not initialized from pretrained model means that the weights of XXX do not come
pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning
task. The warning Weights from XXX not used in YYY means that the layer XXX is not used by YYY, therefore those
weights are discarded. Activate the special “offline-mode” to
use this method in a firewalled environment.  Examples:   Copied >>> from transformers import BertConfig, BertModel
>>> # Download model and configuration from huggingface.co and cache.
>>> model = BertModel.from_pretrained("bert-base-uncased")
>>> # Model was saved using *save_pretrained('./test/saved_model/')* (for example purposes, not runnable).
>>> model = BertModel.from_pretrained("./test/saved_model/")
>>> # Update configuration during loading.
>>> m


PreTrainedModel

odel = BertModel.from_pretrained("bert-base-uncased", output_attentions=True)
>>> assert model.config.output_attentions == True
>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower, for example purposes, not runnable).
>>> config = BertConfig.from_json_file("./tf_model/my_tf_model_config.json")
>>> model = BertModel.from_pretrained("./tf_model/my_tf_checkpoint.ckpt.index", from_tf=True, config=config)
>>> # Loading from a Flax checkpoint file instead of a PyTorch model (slower)
>>> model = BertModel.from_pretrained("bert-base-uncased", from_flax=True) low_cpu_mem_usage algorithm: This is an experimental function that loads the model using ~1x model size CPU memory Here is how it works: save which state_dict keys we have drop state_dict before the model is created, since the latter takes 1x model size CPU memory after the model has been instantiated switch to the meta device all params/buffers that
are going to be replaced from th


PreTrainedModel

e loaded state_dict load state_dict 2nd time replace the params/buffers from the state_dict Currently, it can’t handle deepspeed ZeRO stage 3 and ignores loading errors  get_input_embeddings  < source > ( ) → nn.Module  Returns
nn.Module

A torch module mapping vocabulary to hidden states.
  Returns the model’s input embeddings.  get_memory_footprint  < source > ( return_buffers = True  )   Parameters   return_buffers (bool, optional, defaults to True) —
Whether to return the size of the buffer tensors in the computation of the memory footprint. Buffers
are tensors that do not require gradients and not registered as parameters. E.g. mean and std in batch
norm layers. Please see: https://discuss.pytorch.org/t/what-pytorch-means-by-buffers/120266/2    Get the memory footprint of a model. This will return the memory footprint of the current model in bytes.
Useful to benchmark the memory footprint of the current model and design some tests. Solution in


PreTrainedModel

spired from the
PyTorch discussions: https://discuss.pytorch.org/t/gpu-memory-that-model-uses/56822/2  get_output_embeddings  < source > ( ) → nn.Module  Returns
nn.Module

A torch module mapping hidden states to vocabulary.
  Returns the model’s output embeddings.  gradient_checkpointing_disable  < source > ( )    Deactivates gradient checkpointing for the current model. Note that in other frameworks this feature can be referred to as “activation checkpointing” or “checkpoint
activations”.  gradient_checkpointing_enable  < source > ( )    Activates gradient checkpointing for the current model. Note that in other frameworks this feature can be referred to as “activation checkpointing” or “checkpoint
activations”.  init_weights  < source > ( )    If needed prunes and maybe initializes weights. If using a custom PreTrainedModel, you need to implement any
initialization logic in _init_weights.  post_init  < source > ( )    A method executed at the end


PreTrainedModel

 of each Transformer model initialization, to execute code that needs the model’s
modules properly initialized (such as weight initialization).  prune_heads  < source > ( heads_to_prune: typing.Dict[int, typing.List[int]]  )   Parameters   heads_to_prune (Dict[int, List[int]]) —
Dictionary with keys being selected layer indices (int) and associated values being the list of heads
to prune in said layer (list of int). For instance {1: [0, 2], 2: [2, 3]} will prune heads 0 and 2 on
layer 1 and heads 2 and 3 on layer 2.    Prunes heads of the base model.  register_for_auto_class  < source > ( auto_class = 'AutoModel'  )   Parameters   auto_class (str or type, optional, defaults to "AutoModel") —
The auto class to register this new model with.    Register this class with a given auto class. This should only be used for custom models as the ones in the
library are already mapped with an auto class. This API is experimental and may have some slight breaking


PreTrainedModel

changes in the next releases.  resize_token_embeddings  < source > ( new_num_tokens: typing.Optional[int] = None pad_to_multiple_of: typing.Optional[int] = None  ) → torch.nn.Embedding  Parameters   new_num_tokens (int, optional) —
The number of new tokens in the embedding matrix. Increasing the size will add newly initialized
vectors at the end. Reducing the size will remove vectors from the end. If not provided or None, just
returns a pointer to the input tokens torch.nn.Embedding module of the model without doing anything.   pad_to_multiple_of (int, optional) —
If set will pad the embedding matrix to a multiple of the provided value.If new_num_tokens is set to
None will just pad the embedding to a multiple of pad_to_multiple_of.
This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability
>= 7.5 (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128. For more
details about this,


PreTrainedModel

 or help on choosing the correct value for resizing, refer to this guide:
https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc   Returns
torch.nn.Embedding

Pointer to the input tokens Embeddings Module of the model.
  Resizes input token embeddings matrix of the model if new_num_tokens != config.vocab_size. Takes care of tying weights embeddings afterwards if the model class has a tie_weights() method.  reverse_bettertransformer  < source > ( ) → PreTrainedModel  Returns
PreTrainedModel

The model converted back to the original modeling.
  Reverts the transformation from to_bettertransformer() so that the original modeling is
used, for example in order to save the model.  save_pretrained  < source > ( save_directory: typing.Union[str, os.PathLike] is_main_process: bool = True state_dict: typing.Optional[dict] = None save_function: typing.Callable = <function save at 0x7ff0c0557ee0> pus


PreTrainedModel

h_to_hub: bool = False max_shard_size: typing.Union[int, str] = '10GB' safe_serialization: bool = False variant: typing.Optional[str] = None token: typing.Union[bool, str, NoneType] = None save_peft_format: bool = True **kwargs  )   Parameters   save_directory (str or os.PathLike) —
Directory to which to save. Will be created if it doesn’t exist.   is_main_process (bool, optional, defaults to True) —
Whether the process calling this is the main process or not. Useful when in distributed training like
TPUs and need to call this function on all processes. In this case, set is_main_process=True only on
the main process to avoid race conditions.   state_dict (nested dictionary of torch.Tensor) —
The state dictionary of the model to save. Will default to self.state_dict(), but can be used to only
save parts of the model or if special precautions need to be taken when recovering the state dictionary
of a model (like when using model parallelism).   save_fun


PreTrainedModel

ction (Callable) —
The function to use to save the state dictionary. Useful on distributed training like TPUs when one
need to replace torch.save by another method.   push_to_hub (bool, optional, defaults to False) —
Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the
repository you want to push to with repo_id (will default to the name of save_directory in your
namespace).   max_shard_size (int or str, optional, defaults to "10GB") —
The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size
lower than this size. If expressed as a string, needs to be digits followed by a unit (like "5MB").
If a single weight of the model is bigger than max_shard_size, it will be in its own checkpoint shard
which will be bigger than max_shard_size.
   safe_serialization (bool, optional, defaults to False) —
Whether to save the model using safetensors or the traditional PyTorch wa


PreTrainedModel

y (that uses pickle).   variant (str, optional) —
If specified, weights are saved in the format pytorch_model..bin.   token (str or bool, optional) —
The token to use as HTTP bearer authorization for remote files. If True, or not specified, will use
the token generated when running huggingface-cli login (stored in ~/.huggingface).   save_peft_format (bool, optional, defaults to True) —
For backward compatibility with PEFT library, in case adapter weights are attached to the model, all
keys of the state dict of adapters needs to be pre-pended with base_model.model. Advanced users can
disable this behaviours by setting save_peft_format to False.   kwargs (Dict[str, Any], optional) —
Additional key word arguments passed along to the push_to_hub() method.    Save a model and its configuration file to a directory, so that it can be re-loaded using the
from_pretrained() class method.  set_input_embeddings  < source > ( value: Module  )   Parameters   value


PreTrainedModel

(nn.Module) — A module mapping vocabulary to hidden states.    Set model’s input embeddings.  tie_weights  < source > ( )    Tie the weights between the input embeddings and the output embeddings. If the torchscript flag is set in the configuration, can’t handle parameter sharing so we are cloning the
weights instead.  to_bettertransformer  < source > ( ) → PreTrainedModel  Returns
PreTrainedModel

The model converted to BetterTransformer.
  Converts the model to use PyTorch’s native attention
implementation, integrated to
Transformers through Optimum library. Only a
subset of all Transformers models are supported. PyTorch’s attention fastpath allows to speed up inference through kernel fusions and the use of nested
tensors. Detailed benchmarks can be found in this blog
post.  warn_if_padding_and_no_attention_mask  < source > ( input_ids attention_mask  )    Shows a one-time warning if the input_ids appear to contain padding and no attention mask w


Large model loading

In Transformers 4.20.0, the from_pretrained() method has been reworked to accommodate large models using Accelerate. This requires Accelerate >= 0.9.0 and PyTorch >= 1.9.0. Instead of creating the full model, then loading the pretrained weights inside it (which takes twice the size of the model in RAM, one for the randomly initialized model, one for the weights), there is an option to create the model as an empty shell, then only materialize its parameters when the pretrained weights are loaded.
This option can be activated with low_cpu_mem_usage=True. The model is first created on the Meta device (with empty weights) and the state dict is then loaded inside it (shard by shard in the case of a sharded checkpoint). This way the maximum RAM used is the full size of the mod


Large model loading

el only.
  Copied from transformers import AutoModelForSeq2SeqLM
t0pp = AutoModelForSeq2SeqLM.from_pretrained("bigscience/T0pp", low_cpu_mem_usage=True)
Moreover, you can directly place the model on different devices if it doesn’t fully fit in RAM (only works for inference for now). With device_map="auto", Accelerate will determine where to put each layer to maximize the use of your fastest devices (GPUs) and offload the rest on the CPU, or even the hard drive if you don’t have enough GPU RAM (or CPU RAM). Even if the model is split across several devices, it will run as you would normally expect.
When passing a device_map, low_cpu_mem_usage is automatically set to True, so you don’t need to specify it:
  Copied from transformers import AutoModelForSeq2SeqLM
t0pp = AutoM


Large model loading

odelForSeq2SeqLM.from_pretrained("bigscience/T0pp", device_map="auto")
You can inspect how the model was split across devices by looking at its hf_device_map attribute:
  Copied t0pp.hf_device_map
  Copied {'shared': 0,
 'decoder.embed_tokens': 0,
 'encoder': 0,
 'decoder.block.0': 0,
 'decoder.block.1': 1,
 'decoder.block.2': 1,
 'decoder.block.3': 1,
 'decoder.block.4': 1,
 'decoder.block.5': 1,
 'decoder.block.6': 1,
 'decoder.block.7': 1,
 'decoder.block.8': 1,
 'decoder.block.9': 1,
 'decoder.block.10': 1,
 'decoder.block.11': 1,
 'decoder.block.12': 1,
 'decoder.block.13': 1,
 'decoder.block.14': 1,
 'decoder.block.15': 1,
 'decoder.block.16': 1,
 'decoder.block.17': 1,
 'decoder.block.18': 1,
 'decoder.block.19': 1,
 'decoder.block.20': 1,
 'decoder.block.21': 1,


Large model loading

 'decoder.block.22': 'cpu',
 'decoder.block.23': 'cpu',
 'decoder.final_layer_norm': 'cpu',
 'decoder.dropout': 'cpu',
 'lm_head': 'cpu'}
You can also write your own device map following the same format (a dictionary layer name to device). It should map all parameters of the model to a given device, but you don’t have to detail where all the submodules of one layer go if that layer is entirely on the same device. For instance, the following device map would work properly for T0pp (as long as you have the GPU memory):
  Copied device_map = {"shared": 0, "encoder": 0, "decoder": 1, "lm_head": 1}
Another way to minimize the memory impact of your model is to instantiate it at a lower precision dtype (like torch.float16) or use direct quantization techniques as described belo


Model Instantiation dtype

Under Pytorch a model normally gets instantiated with torch.float32 format. This can be an issue if one tries to
load a model whose weights are in fp16, since it’d require twice as much memory. To overcome this limitation, you can
either explicitly pass the desired dtype using torch_dtype argument:
  Copied model = T5ForConditionalGeneration.from_pretrained("t5", torch_dtype=torch.float16)
or, if you want the model to always load in the most optimal memory pattern, you can use the special value "auto",
and then dtype will be automatically derived from the model’s weights:
  Copied model = T5ForConditionalGeneration.from_pretrained("t5", torch_dtype="auto")
Models instantiated from scratch can also be told which dtype to use with:
  Copied config = T5Config.from_pretrained("t5")
model = AutoModel.from_config(config)
Due to Pytorch design, this functionality is only available for floating dtypes.


ModuleUtilsMixin

class transformers.modeling_utils.ModuleUtilsMixin  < source > ( )    A few utilities for torch.nn.Modules, to be used as a mixin.  add_memory_hooks  < source > ( )    Add a memory hook before and after each sub-module forward pass to record increase in memory consumption. Increase in memory consumption is stored in a mem_rss_diff attribute for each module and can be reset to zero
with model.reset_memory_hooks_state().  estimate_tokens  < source > ( input_dict: typing.Dict[str, typing.Union[torch.Tensor, typing.Any]]  ) → int  Parameters   inputs (dict) — The model inputs.   Returns
int

The total number of tokens.
  Helper function to estimate the total number of tokens from the model inputs.  floating_point_ops  < source > ( input_dict: typing.Dict[str, typing.Union[torch.Tensor, typing.Any]] exclude_embeddings: bool = True  ) → int  Parameters   batch_size (int


ModuleUtilsMixin

) —
The batch size for the forward pass.   sequence_length (int) —
The number of tokens in each line of the batch.   exclude_embeddings (bool, optional, defaults to True) —
Whether or not to count embedding and softmax operations.   Returns
int

The number of floating-point operations.
  Get number of (optionally, non-embeddings) floating-point operations for the forward and backward passes of a
batch with this transformer model. Default approximation neglects the quadratic dependency on the number of
tokens (valid if 12 * d_model << sequence_length) as laid out in this
paper section 2.1. Should be overridden for transformers with parameter
re-use e.g. Albert or Universal Transformers, or if doing long-range modeling with very high sequence lengths.  get_extended_attention_mask  < source > ( attention_mask: Tensor input_shape: typing.Tuple[int] device: device = No


ModuleUtilsMixin

ne dtype: torch.float32 = None  )   Parameters   attention_mask (torch.Tensor) —
Mask with ones indicating tokens to attend to, zeros for tokens to ignore.   input_shape (Tuple[int]) —
The shape of the input to the model.    Makes broadcastable attention and causal masks so that future and masked tokens are ignored.  get_head_mask  < source > ( head_mask: typing.Optional[torch.Tensor] num_hidden_layers: int is_attention_chunked: bool = False  )   Parameters   head_mask (torch.Tensor with shape [num_heads] or [num_hidden_layers x num_heads], optional) —
The mask indicating if we should keep the heads or not (1.0 for keep, 0.0 for discard).   num_hidden_layers (int) —
The number of hidden layers in the model.   is_attention_chunked (bool, optional, defaults to False) —
Whether or not the attentions scores are computed by chunks or not.    Prepare the head mask if neede


ModuleUtilsMixin

d.  invert_attention_mask  < source > ( encoder_attention_mask: Tensor  ) → torch.Tensor  Parameters   encoder_attention_mask (torch.Tensor) — An attention mask.   Returns
torch.Tensor

The inverted attention mask.
  Invert an attention mask (e.g., switches 0. and 1.).  num_parameters  < source > ( only_trainable: bool = False exclude_embeddings: bool = False  ) → int  Parameters   only_trainable (bool, optional, defaults to False) —
Whether or not to return only the number of trainable parameters   exclude_embeddings (bool, optional, defaults to False) —
Whether or not to return only the number of non-embeddings parameters   Returns
int

The number of parameters.
  Get number of (optionally, trainable or non-embeddings) parameters in the module.  reset_memory_hooks_state  < source > ( )    Reset the mem_rss_diff attribute of each module (see add_memory_hooks()


TFPreTrainedModel

class transformers.TFPreTrainedModel  < source > ( *args **kwargs  )    Base class for all TF models. TFPreTrainedModel takes care of storing the configuration of the models and handles methods for loading,
downloading and saving models as well as a few methods common to all models to: resize the input embeddings, prune heads in the self-attention heads. Class attributes (overridden by derived classes): config_class (PretrainedConfig) — A subclass of PretrainedConfig to use as configuration class
for this model architecture. base_model_prefix (str) — A string indicating the attribute associated to the base model in derived
classes of the same architecture adding modules on top of the base model. main_input_name (str) — The name of the principal input to the model (often input_ids for NLP
models, pixel_values for vision models and input_values for speech models).  push_to_hub  < source > ( repo_id: str use_temp_dir: Optional[bool] = None commit_message: Optional[st


TFPreTrainedModel

r] = None private: Optional[bool] = None max_shard_size: Optional[Union[int, str]] = '10GB' token: Optional[Union[bool, str]] = None use_auth_token: Optional[Union[bool, str]] = None create_pr: bool = False **base_model_card_args  )   Parameters   repo_id (str) —
The name of the repository you want to push your model to. It should contain your organization name
when pushing to a given organization.   use_temp_dir (bool, optional) —
Whether or not to use a temporary directory to store the files saved before they are pushed to the Hub.
Will default to True if there is no directory named like repo_id, False otherwise.   commit_message (str, optional) —
Message to commit while pushing. Will default to "Upload model".   private (bool, optional) —
Whether or not the repository created should be private.   token (bool or str, optional) —
The token to use as HTTP bearer authorization for remote files. If True, will use the token generated
when running huggingface-cli logi


TFPreTrainedModel

n (stored in ~/.huggingface). Will default to True if repo_url
is not specified.   max_shard_size (int or str, optional, defaults to "10GB") —
Only applicable for models. The maximum size for a checkpoint before being sharded. Checkpoints shard
will then be each of size lower than this size. If expressed as a string, needs to be digits followed
by a unit (like "5MB").   create_pr (bool, optional, defaults to False) —
Whether or not to create a PR with the uploaded files or directly commit.    Upload the model files to the 🤗 Model Hub while synchronizing a local clone of the repo in repo_path_or_name.  Examples:   Copied from transformers import TFAutoModel
model = TFAutoModel.from_pretrained("bert-base-cased")
# Push the model to your namespace with the name "my-finetuned-bert".
model.push_to_hub("my-finetuned-bert")
# Push the model to an organization with the name "my-finetuned-bert".
model.push_to_hub("huggingface/my-finetuned-bert")  can_generate  < source > (


TFPreTrainedModel

 ) → bool  Returns
bool

Whether this model can generate sequences with .generate().
  Returns whether this model can generate sequences with .generate().  compile  < source > ( optimizer = 'rmsprop' loss = 'auto_with_warning' metrics = None loss_weights = None weighted_metrics = None run_eagerly = None steps_per_execution = None **kwargs  )    This is a thin wrapper that sets the model’s loss output head as the loss if the user does not specify a loss
function themselves.  create_model_card  < source > ( output_dir model_name: str language: Optional[str] = None license: Optional[str] = None tags: Optional[str] = None finetuned_from: Optional[str] = None tasks: Optional[str] = None dataset_tags: Optional[Union[str, List[str]]] = None dataset: Optional[Union[str, List[str]]] = None dataset_args: Optional[Union[str, List[str]]] = None  )   Parameters   output_dir (str or os.PathLike) —
The folder in which to create the model card.   model_name (str, optional) —
T


TFPreTrainedModel

he name of the model.   language (str, optional) —
The language of the model (if applicable)   license (str, optional) —
The license of the model. Will default to the license of the pretrained model used, if the original
model given to the Trainer comes from a repo on the Hub.   tags (str or List[str], optional) —
Some tags to be included in the metadata of the model card.   finetuned_from (str, optional) —
The name of the model used to fine-tune this one (if applicable). Will default to the name of the repo
of the original model given to the Trainer (if it comes from the Hub).   tasks (str or List[str], optional) —
One or several task identifiers, to be included in the metadata of the model card.   dataset_tags (str or List[str], optional) —
One or several dataset tags, to be included in the metadata of the model card.   dataset (str or List[str], optional) —
One or several dataset identifiers, to be included in the metadata of the model card.   dataset_args (str


TFPreTrainedModel

 or List[str], optional) —
One or several dataset arguments, to be included in the metadata of the model card.    Creates a draft of a model card using the information available to the Trainer.  eager_serving  < source > ( inputs  )   Parameters   inputs (Dict[str, tf.Tensor]) —
The input of the saved model as a dictionary of tensors.    Method used for serving the model. This method is deprecated, and will be removed.  from_pretrained  < source > ( pretrained_model_name_or_path: Optional[Union[str, os.PathLike]] *model_args config: Optional[Union[PretrainedConfig, str, os.PathLike]] = None cache_dir: Optional[Union[str, os.PathLike]] = None ignore_mismatched_sizes: bool = False force_download: bool = False local_files_only: bool = False token: Optional[Union[str, bool]] = None revision: str = 'main' **kwargs  )   Parameters   pretrained_model_name_or_path (str, optional) —
Can be either:
A string, the model id of a pretrained model hosted inside a model repo on h


TFPreTrainedModel

uggingface.co.
Valid model ids can be located at the root-level, like bert-base-uncased, or namespaced under a
user or organization name, like dbmdz/bert-base-german-cased.
A path to a directory containing model weights saved using
save_pretrained(), e.g., ./my_model_directory/.
A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this
case, from_pt should be set to True and a configuration object should be provided as config
argument. This loading path is slower than converting the PyTorch model in a TensorFlow model
using the provided conversion scripts and loading the TensorFlow model afterwards.
None if you are both providing the configuration and state dictionary (resp. with keyword
arguments config and state_dict).
   model_args (sequence of positional arguments, optional) —
All remaining positional arguments will be passed to the underlying model’s __init__ method.   config (Union[PretrainedConfig, str], optional) —
Can be


TFPreTrainedModel

 either:
an instance of a class derived from PretrainedConfig,
a string valid as input to from_pretrained().
Configuration for the model to use instead of an automatically loaded configuration. Configuration can
be automatically loaded when:
The model is a model provided by the library (loaded with the model id string of a pretrained
model).
The model was saved using save_pretrained() and is reloaded by supplying the
save directory.
The model is loaded by supplying a local directory as pretrained_model_name_or_path and a
configuration JSON file named config.json is found in the directory.
   from_pt (bool, optional, defaults to False) —
Load the model weights from a PyTorch state_dict save file (see docstring of
pretrained_model_name_or_path argument).   ignore_mismatched_sizes (bool, optional, defaults to False) —
Whether or not to raise an error if some of the weights from the checkpoint do not have the same size
as the weights of the model (if for instance, you


TFPreTrainedModel

 are instantiating a model with 10 labels from a
checkpoint with 3 labels).   cache_dir (str, optional) —
Path to a directory in which a downloaded pretrained model configuration should be cached if the
standard cache should not be used.   force_download (bool, optional, defaults to False) —
Whether or not to force the (re-)download of the model weights and configuration files, overriding the
cached versions if they exist.   resume_download (bool, optional, defaults to False) —
Whether or not to delete incompletely received files. Will attempt to resume the download if such a
file exists.
proxies —
(Dict[str, str], optional): A dictionary of proxy servers to use by protocol or endpoint, e.g., {‘http’: ‘foo.bar:3128’, ‘http://hostname’: ‘foo.bar:4012’}. The proxies are used on each request. output_loading_info(bool, *optional*, defaults to False`): Whether ot not to also return a
dictionary containing missing keys, unexpected keys and error messages.   local_files_


TFPreTrainedModel

only(bool, optional, defaults to False) —
Whether or not to only look at local files (e.g., not try downloading the model).   token (str or bool, optional) —
The token to use as HTTP bearer authorization for remote files. If True, or not specified, will use
the token generated when running huggingface-cli login (stored in ~/.huggingface).   revision (str, optional, defaults to "main") —
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so revision can be any
identifier allowed by git.    Instantiate a pretrained TF 2.0 model from a pre-trained model configuration. The warning Weights from XXX not initialized from pretrained model means that the weights of XXX do not come
pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning
task. The warning Weights from XXX not used in YYY means that


TFPreTrainedModel

the layer XXX is not used by YYY, therefore those
weights are discarded.  Examples:   Copied >>> from transformers import BertConfig, TFBertModel
>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFBertModel.from_pretrained("bert-base-uncased")
>>> # Model was saved using *save_pretrained('./test/saved_model/')* (for example purposes, not runnable).
>>> model = TFBertModel.from_pretrained("./test/saved_model/")
>>> # Update configuration during loading.
>>> model = TFBertModel.from_pretrained("bert-base-uncased", output_attentions=True)
>>> assert model.config.output_attentions == True
>>> # Loading from a Pytorch model file instead of a TensorFlow checkpoint (slower, for example purposes, not runnable).
>>> config = BertConfig.from_json_file("./pt_model/my_pt_model_config.json")
>>> model = TFBertModel.from_pretrained("./pt_model/my_pytorch_model.bin", from_pt=True, config=config)  get_bias  < source > ( ) → tf.Variable  Returns
t


TFPreTrainedModel

f.Variable

The weights representing the bias, None if not an LM model.
  Dict of bias attached to an LM head. The key represents the name of the bias attribute.  get_head_mask  < source > ( head_mask: tf.Tensor | None num_hidden_layers: int  )   Parameters   head_mask (tf.Tensor with shape [num_heads] or [num_hidden_layers x num_heads], optional) —
The mask indicating if we should keep the heads or not (1.0 for keep, 0.0 for discard).   num_hidden_layers (int) —
The number of hidden layers in the model.    Prepare the head mask if needed.  get_input_embeddings  < source > ( ) → tf.Variable  Returns
tf.Variable

The embeddings layer mapping vocabulary to hidden states.
  Returns the model’s input embeddings layer.  get_lm_head  < source > ( ) → tf.keras.layers.Layer  Returns
tf.keras.layers.Layer

The LM head layer if the model has one, None if not.
  The LM Head layer. This method must be overwritten by all the models that have a lm head.  get_output_embe


TFPreTrainedModel

ddings  < source > ( ) → tf.Variable  Returns
tf.Variable

The new weights mapping vocabulary to hidden states.
  Returns the model’s output embeddings  get_output_layer_with_bias  < source > ( ) → tf.keras.layers.Layer  Returns
tf.keras.layers.Layer

The layer that handles the bias, None if not an LM model.
  Get the layer that handles a bias attribute in case the model has an LM head with weights tied to the
embeddings  get_prefix_bias_name  < source > ( ) → str  Returns
str

The _prefix name of the bias.
  Get the concatenated _prefix name of the bias from the model name to the parent layer  load_repo_checkpoint  < source > ( repo_path_or_name  ) → dict  Parameters   repo_path_or_name (str) —
Can either be a repository name for your {object} in the Hub or a path to a local folder (in which case
the repository will have the name of that local folder).   Returns
dict

A dictionary of extra metadata from the checkpoint, most commonly an “epoch” count.


TFPreTrainedModel

  Loads a saved checkpoint (model weights and optimizer state) from a repo. Returns the current epoch count when
the checkpoint was made.  prepare_tf_dataset  < source > ( dataset: 'datasets.Dataset' batch_size: int = 8 shuffle: bool = True tokenizer: Optional['PreTrainedTokenizerBase'] = None collate_fn: Optional[Callable] = None collate_fn_args: Optional[Dict[str, Any]] = None drop_remainder: Optional[bool] = None prefetch: bool = True  ) → Dataset  Parameters   dataset (Any) —
A [~datasets.Dataset] to be wrapped as a tf.data.Dataset.   batch_size (int, defaults to 8) —
The size of batches to return.   shuffle (bool, defaults to True) —
Whether to return samples from the dataset in random order. Usually True for training datasets and
False for validation/test datasets.   tokenizer (PreTrainedTokenizerBase, optional) —
A PreTrainedTokenizer that will be used to pad samples to create batches. Has no effect if a specific
collate_fn is passed instead.   collate_fn (


TFPreTrainedModel

Callable, optional) —
A function that collates samples from the dataset into a single batch. Defaults to
DefaultDataCollator if no tokenizer is supplied or DataCollatorWithPadding if a tokenizer is
passed.   collate_fn_args (Dict[str, Any], optional) —
A dict of arguments to pass to the collate_fn alongside the list of samples.   drop_remainder (bool, optional) —
Whether to drop the final batch, if the batch_size does not evenly divide the dataset length. Defaults
to the same setting as shuffle.   prefetch (bool, defaults to True) —
Whether to add prefetching to the end of the tf.data pipeline. This is almost always beneficial for
performance, but can be disabled in edge cases.   Returns
Dataset

A tf.data.Dataset which is ready to pass to the Keras API.
  Wraps a HuggingFace Dataset as a tf.data.Dataset with collation and batching. This method is
designed to create a “ready-to-use” dataset that can be passed directly to Keras methods like fit() without
further


TFPreTrainedModel

 modification. The method will drop columns from the dataset if they don’t match input names for the
model. If you want to specify the column names to return rather than using the names that match this model, we
recommend using Dataset.to_tf_dataset() instead.  prune_heads  < source > ( heads_to_prune  )   Parameters   heads_to_prune (Dict[int, List[int]]) —
Dictionary with keys being selected layer indices (int) and associated values being the list of heads
to prune in said layer (list of int). For instance {1: [0, 2], 2: [2, 3]} will prune heads 0 and 2 on
layer 1 and heads 2 and 3 on layer 2.    Prunes heads of the base model.  register_for_auto_class  < source > ( auto_class = 'TFAutoModel'  )   Parameters   auto_class (str or type, optional, defaults to "TFAutoModel") —
The auto class to register this new model with.    Register this class with a given auto class. This should only be used for custom models as the ones in the
library are already mapped with an


TFPreTrainedModel

 auto class. This API is experimental and may have some slight breaking changes in the next releases.  resize_token_embeddings  < source > ( new_num_tokens: Optional[int] = None  ) → tf.Variable or tf.keras.layers.Embedding  Parameters   new_num_tokens (int, optional) —
The number of new tokens in the embedding matrix. Increasing the size will add newly initialized
vectors at the end. Reducing the size will remove vectors from the end. If not provided or None, just
returns a pointer to the input tokens without doing anything.   Returns
tf.Variable or tf.keras.layers.Embedding

Pointer to the input tokens of the model.
  Resizes input token embeddings matrix of the model if new_num_tokens != config.vocab_size. Takes care of tying weights embeddings afterwards if the model class has a tie_weights() method.  save_pretrained  < source > ( save_directory saved_model = False version = 1 push_to_hub = False signatures = None max_shard_size: Union[int, str] = '10GB' cr


TFPreTrainedModel

eate_pr: bool = False safe_serialization: bool = False token: Optional[Union[str, bool]] = None **kwargs  )   Parameters   save_directory (str) —
Directory to which to save. Will be created if it doesn’t exist.   saved_model (bool, optional, defaults to False) —
If the model has to be saved in saved model format as well or not.   version (int, optional, defaults to 1) —
The version of the saved model. A saved model needs to be versioned in order to be properly loaded by
TensorFlow Serving as detailed in the official documentation
https://www.tensorflow.org/tfx/serving/serving_basic   push_to_hub (bool, optional, defaults to False) —
Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the
repository you want to push to with repo_id (will default to the name of save_directory in your
namespace).   signatures (dict or tf.function, optional) —
Model’s signature used for serving. This will be passed to the signatures argumen


TFPreTrainedModel

t of model.save().   max_shard_size (int or str, optional, defaults to "10GB") —
The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size
lower than this size. If expressed as a string, needs to be digits followed by a unit (like "5MB").
If a single weight of the model is bigger than max_shard_size, it will be in its own checkpoint shard
which will be bigger than max_shard_size.
   create_pr (bool, optional, defaults to False) —
Whether or not to create a PR with the uploaded files or directly commit.   safe_serialization (bool, optional, defaults to False) —
Whether to save the model using safetensors or the traditional PyTorch way (that uses pickle).   token (str or bool, optional) —
The token to use as HTTP bearer authorization for remote files. If True, or not specified, will use
the token generated when running huggingface-cli login (stored in ~/.huggingface).   kwargs (Dict[str, Any], optional) —
Additional key word


TFPreTrainedModel

 arguments passed along to the push_to_hub() method.    Save a model and its configuration file to a directory, so that it can be re-loaded using the
from_pretrained() class method.  serving   ( inputs  )   Parameters   Method used for serving the model. Does not have a specific signature, but will be specialized as concrete —   functions when saving with save_pretrained. —
inputs (Dict[str, tf.Tensor]):
The input of the saved model as a dictionary of tensors.     serving_output  < source > ( output  )    Prepare the output of the saved model. Can be overridden if specific serving modifications are required.  set_bias  < source > ( value  )   Parameters   value (Dict[tf.Variable]) —
All the new bias attached to an LM head.    Set all the bias in the LM head.  set_input_embeddings  < source > ( value  )   Parameters   value (tf.Variable) —
The new weights mapping hidden states to vocabulary.    Set model’s input embeddings  set_output_embeddings  < source > ( value


TFPreTrainedModel

  )   Parameters   value (tf.Variable) —
The new weights mapping hidden states to vocabulary.    Set model’s output embeddings  test_step  < source > ( data  )    A modification of Keras’s default train_step that correctly handles matching outputs to labels for our models
and supports directly training on the loss output head. In addition, it ensures input keys are copied to the
labels where appropriate. It will also copy label keys into the input dict when using the dummy loss, to ensure
that they are available to the model during the forward pass.  train_step  < source > ( data  )    A modification of Keras’s default train_step that correctly handles matching outputs to labels for our models
and supports directly training on the loss output head. In addition, it ensures input keys are copied to the
labels where appropriate. It will also copy label keys into the input dict when using the dummy loss, to ensure
that they are available to the model during the forwar


TFModelUtilsMixin

class transformers.modeling_tf_utils.TFModelUtilsMixin  < source > ( )    A few utilities for tf.keras.Model, to be used as a mixin.  num_parameters  < source > ( only_trainable: bool = False  ) → int  Parameters   only_trainable (bool, optional, defaults to False) —
Whether or not to return only the number of trainable parameters   Returns
int

The number of parameters.
  Get the number of (optionally, trainable) parameters in the model.


FlaxPreTrainedModel

class transformers.FlaxPreTrainedModel  < source > ( config: PretrainedConfig module: Module input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True  )    Base class for all models. FlaxPreTrainedModel takes care of storing the configuration of the models and handles methods for loading,
downloading and saving models. Class attributes (overridden by derived classes): config_class (PretrainedConfig) — A subclass of PretrainedConfig to use as configuration class
for this model architecture. base_model_prefix (str) — A string indicating the attribute associated to the base model in derived
classes of the same architecture adding modules on top of the base model. main_input_name (str) — The name of the principal input to the model (often input_ids for NLP
models, pixel_values for vision models and input_values for speech models).  push_to_hub  < source > ( repo_id: str use_temp_dir: typing.Opt


FlaxPreTrainedModel

ional[bool] = None commit_message: typing.Optional[str] = None private: typing.Optional[bool] = None token: typing.Union[bool, str, NoneType] = None max_shard_size: typing.Union[int, str, NoneType] = '10GB' create_pr: bool = False safe_serialization: bool = False revision: str = None **deprecated_kwargs  )   Parameters   repo_id (str) —
The name of the repository you want to push your model to. It should contain your organization name
when pushing to a given organization.   use_temp_dir (bool, optional) —
Whether or not to use a temporary directory to store the files saved before they are pushed to the Hub.
Will default to True if there is no directory named like repo_id, False otherwise.   commit_message (str, optional) —
Message to commit while pushing. Will default to "Upload model".   private (bool, optional) —
Whether or not the repository created should be private.   token (bool or str, optional) —
The token to use as HTTP bearer authorizat


FlaxPreTrainedModel

ion for remote files. If True, will use the token generated
when running huggingface-cli login (stored in ~/.huggingface). Will default to True if repo_url
is not specified.   max_shard_size (int or str, optional, defaults to "10GB") —
Only applicable for models. The maximum size for a checkpoint before being sharded. Checkpoints shard
will then be each of size lower than this size. If expressed as a string, needs to be digits followed
by a unit (like "5MB").   create_pr (bool, optional, defaults to False) —
Whether or not to create a PR with the uploaded files or directly commit.   safe_serialization (bool, optional, defaults to False) —
Whether or not to convert the model weights in safetensors format for safer serialization.   revision (str, optional) —
Branch to push the uploaded files to.    Upload the model checkpoint to the 🤗 Model Hub.  Examples:   Copied from transformers import FlaxAutoModel
model = FlaxAutoModel.from_pretrained("bert-b


FlaxPreTrainedModel

ase-cased")
# Push the model to your namespace with the name "my-finetuned-bert".
model.push_to_hub("my-finetuned-bert")
# Push the model to an organization with the name "my-finetuned-bert".
model.push_to_hub("huggingface/my-finetuned-bert")  can_generate  < source > ( )    Returns whether this model can generate sequences with .generate(). Returns:
bool: Whether this model can generate sequences with .generate().  from_pretrained  < source > ( pretrained_model_name_or_path: typing.Union[str, os.PathLike] dtype: dtype = <class 'jax.numpy.float32'> *model_args config: typing.Union[transformers.configuration_utils.PretrainedConfig, str, os.PathLike, NoneType] = None cache_dir: typing.Union[str, os.PathLike, NoneType] = None ignore_mismatched_sizes: bool = False force_download: bool = False local_files_only: bool = False token: typing.Union[bool, str, NoneType] = None revision: str = 'main' **kwargs  )   Parameters   pretrained_model_name_or_path (


FlaxPreTrainedModel

str or os.PathLike) —
Can be either:
A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like bert-base-uncased, or namespaced under a
user or organization name, like dbmdz/bert-base-german-cased.
A path to a directory containing model weights saved using
save_pretrained(), e.g., ./my_model_directory/.
A path or url to a pt index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case,
from_pt should be set to True.
   dtype (jax.numpy.dtype, optional, defaults to jax.numpy.float32) —
The data type of the computation. Can be one of jax.numpy.float32, jax.numpy.float16 (on GPUs) and
jax.numpy.bfloat16 (on TPUs).
This can be used to enable mixed-precision training or half-precision inference on GPUs or TPUs. If
specified all the computation will be performed with the given dtype.
Note that this only specifies the dtype of the computation and does not i


FlaxPreTrainedModel

nfluence the dtype of model
parameters.
If you wish to change the dtype of the model parameters, see to_fp16() and
to_bf16().   model_args (sequence of positional arguments, optional) —
All remaining positional arguments will be passed to the underlying model’s __init__ method.   config (Union[PretrainedConfig, str, os.PathLike], optional) —
Can be either:
an instance of a class derived from PretrainedConfig,
a string or path valid as input to from_pretrained().
Configuration for the model to use instead of an automatically loaded configuration. Configuration can
be automatically loaded when:
The model is a model provided by the library (loaded with the model id string of a pretrained
model).
The model was saved using save_pretrained() and is reloaded by supplying the
save directory.
The model is loaded by supplying a local directory as pretrained_model_name_or_path and a
configuration JSON file named config.json is found in the directory.
   cac


FlaxPreTrainedModel

he_dir (Union[str, os.PathLike], optional) —
Path to a directory in which a downloaded pretrained model configuration should be cached if the
standard cache should not be used.   from_pt (bool, optional, defaults to False) —
Load the model weights from a PyTorch checkpoint save file (see docstring of
pretrained_model_name_or_path argument).   ignore_mismatched_sizes (bool, optional, defaults to False) —
Whether or not to raise an error if some of the weights from the checkpoint do not have the same size
as the weights of the model (if for instance, you are instantiating a model with 10 labels from a
checkpoint with 3 labels).   force_download (bool, optional, defaults to False) —
Whether or not to force the (re-)download of the model weights and configuration files, overriding the
cached versions if they exist.   resume_download (bool, optional, defaults to False) —
Whether or not to delete incompletely received files. Will attempt to resume the


FlaxPreTrainedModel

download if such a
file exists.   proxies (Dict[str, str], optional) —
A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.   local_files_only(bool, optional, defaults to False) —
Whether or not to only look at local files (i.e., do not try to download the model).   token (str or bool, optional) —
The token to use as HTTP bearer authorization for remote files. If True, or not specified, will use
the token generated when running huggingface-cli login (stored in ~/.huggingface).   revision (str, optional, defaults to "main") —
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so revision can be any
identifier allowed by git.    Instantiate a pretrained flax model from a pre-trained model configuration. The warn


FlaxPreTrainedModel

ing Weights from XXX not initialized from pretrained model means that the weights of XXX do not come
pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning
task. The warning Weights from XXX not used in YYY means that the layer XXX is not used by YYY, therefore those
weights are discarded.  Examples:   Copied >>> from transformers import BertConfig, FlaxBertModel
>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxBertModel.from_pretrained("bert-base-cased")
>>> # Model was saved using *save_pretrained('./test/saved_model/')* (for example purposes, not runnable).
>>> model = FlaxBertModel.from_pretrained("./test/saved_model/")
>>> # Loading from a PyTorch checkpoint file instead of a PyTorch model (slower, for example purposes, not runnable).
>>> config = BertConfig.from_json_file("./pt_model/config.json")
>>> model = FlaxBertModel.from_pretrained("./pt_model/p