TrainingArguments

point is also pushed in a subfolder named
last-checkpoint, allowing you to resume training easily with
trainer.train(resume_from_checkpoint="last-checkpoint").
"all_checkpoints": like "checkpoint" but all checkpoints are pushed like they appear in the
output
folder (so you will get one checkpoint folder per folder in your final repository)
   token (str, optional) —
The token to use to push the model to the Hub. Will default to the token in the cache folder obtained
with huggingface-cli login.   private_repo (bool, optional, defaults to False) —
If True, the Hub repo will be set to private.   always_push (bool, optional, defaults to False) —
Unless this is True, the Trainer will skip pushing a checkpoint when the previous push is not
finished.    A method that regroups all arguments linked to synchronizing checkpoints with the Hub. Calling this method will set self.push_to_hub to True, which means the output_dir will begin a git
directory synced with the repo (determined


TrainingArguments

 by model_id) and the content will be pushed each time a save is
triggered (depending onself.save_strategy). Calling save_model() will also trigger a push.  Example:   Copied >>> from transformers import TrainingArguments
>>> args = TrainingArguments("working_dir")
>>> args = args.set_push_to_hub("me/awesome-model")
>>> args.hub_model_id
'me/awesome-model'  set_save  < source > ( strategy: typing.Union[str, transformers.trainer_utils.IntervalStrategy] = 'steps' steps: int = 500 total_limit: typing.Optional[int] = None on_each_node: bool = False  )   Parameters   strategy (str or IntervalStrategy, optional, defaults to "steps") —
The checkpoint save strategy to adopt during training. Possible values are:
"no": No save is done during training.
"epoch": Save is done at the end of each epoch.
"steps": Save is done every save_steps.
   steps (int, optional, defaults to 500) —
Number of updates steps before two checkpoint saves if strategy="steps".   total_limit (int, optional


TrainingArguments

) —
If a value is passed, will limit the total amount of checkpoints. Deletes the older checkpoints in
output_dir.   on_each_node (bool, optional, defaults to False) —
When doing multi-node distributed training, whether to save models and checkpoints on each node, or
only on the main one.
This should not be activated when the different nodes use the same storage as the files will be saved
with the same names for each node.    A method that regroups all arguments linked to the evaluation.  Example:   Copied >>> from transformers import TrainingArguments
>>> args = TrainingArguments("working_dir")
>>> args = args.set_save(strategy="steps", steps=100)
>>> args.save_steps
100  set_testing  < source > ( batch_size: int = 8 loss_only: bool = False jit_mode: bool = False  )   Parameters   batch_size (int optional, defaults to 8) —
The batch size per device (GPU/TPU core/CPU…) used for testing.   loss_only (bool, optional, defaults to False) —
Ignores all outputs except the loss


TrainingArguments

.   jit_mode (bool, optional) —
Whether or not to use PyTorch jit trace for inference.    A method that regroups all basic arguments linked to testing on a held-out dataset. Calling this method will automatically set self.do_predict to True.  Example:   Copied >>> from transformers import TrainingArguments
>>> args = TrainingArguments("working_dir")
>>> args = args.set_testing(batch_size=32)
>>> args.per_device_eval_batch_size
32  set_training  < source > ( learning_rate: float = 5e-05 batch_size: int = 8 weight_decay: float = 0 num_epochs: float = 3 max_steps: int = -1 gradient_accumulation_steps: int = 1 seed: int = 42 gradient_checkpointing: bool = False  )   Parameters   learning_rate (float, optional, defaults to 5e-5) —
The initial learning rate for the optimizer.   batch_size (int optional, defaults to 8) —
The batch size per device (GPU/TPU core/CPU…) used for training.   weight_decay (float, optional, defaults to 0) —
The weight decay to apply (if not zero) to a


TrainingArguments

ll layers except all bias and LayerNorm weights in the
optimizer.   num_train_epochs(float, optional, defaults to 3.0) —
Total number of training epochs to perform (if not an integer, will perform the decimal part percents
of the last epoch before stopping training).   max_steps (int, optional, defaults to -1) —
If set to a positive number, the total number of training steps to perform. Overrides
num_train_epochs. In case of using a finite iterable dataset the training may stop before reaching
the set number of steps when all data is exhausted.   gradient_accumulation_steps (int, optional, defaults to 1) —
Number of updates steps to accumulate the gradients for, before performing a backward/update pass.
When using gradient accumulation, one step is counted as one step with backward pass. Therefore,
logging, evaluation, save will be conducted every gradient_accumulation_steps * xxx_step training
examples.
   seed (int, optional, defaults to 42) —
Random seed that will be


TrainingArguments

set at the beginning of training. To ensure reproducibility across runs, use
the ~Trainer.model_init function to instantiate the model if it has some randomly initialized
parameters.   gradient_checkpointing (bool, optional, defaults to False) —
If True, use gradient checkpointing to save memory at the expense of slower backward pass.    A method that regroups all basic arguments linked to the training. Calling this method will automatically set self.do_train to True.  Example:   Copied >>> from transformers import TrainingArguments
>>> args = TrainingArguments("working_dir")
>>> args = args.set_training(learning_rate=1e-4, batch_size=32)
>>> args.learning_rate
1e-4  to_dict  < source > ( )    Serializes this instance while replace Enum by their values (for JSON serialization support). It obfuscates
the token values by removing their value.  to_json_string  < source > ( )    Serializes this instance to a JSON string.  to_sanitized_dict  < source > ( )    Sanitized serial


class transformers.Seq2SeqTrainingArguments  < source > ( output_dir: str overwrite_output_dir: bool = False do_train: bool = False do_eval: bool = False do_predict: bool = False evaluation_strategy: typing.Union[transformers.trainer_utils.IntervalStrategy, str] = 'no' prediction_loss_only: bool = False per_device_train_batch_size: int = 8 per_device_eval_batch_size: int = 8 per_gpu_train_batch_size: typing.Optional[int] = None per_gpu_eval_batch_size: typing.Optional[int] = None gradient_accumulation_steps: int = 1 eval_accumulation_steps: typing.Optional[int] = None eval_delay: typing.Optional[float] = 0 learning_rate: float = 5e-05 weight_decay: float = 0.0 adam_beta1: float = 0.9 adam_beta2: float = 0.999 adam_epsilon: float = 1e-08 max_grad_norm: float = 1.0 num_train_epochs: float = 3.0 max_steps: int = -1 lr_scheduler_type: typing.Union[transformers.trainer_utils.SchedulerType, str] = 'linear' warmup_ratio: float = 0.0 warmup_steps: int = 0 log_level: typing.O


ptional[str] = 'passive' log_level_replica: typing.Optional[str] = 'warning' log_on_each_node: bool = True logging_dir: typing.Optional[str] = None logging_strategy: typing.Union[transformers.trainer_utils.IntervalStrategy, str] = 'steps' logging_first_step: bool = False logging_steps: float = 500 logging_nan_inf_filter: bool = True save_strategy: typing.Union[transformers.trainer_utils.IntervalStrategy, str] = 'steps' save_steps: float = 500 save_total_limit: typing.Optional[int] = None save_safetensors: typing.Optional[bool] = False save_on_each_node: bool = False no_cuda: bool = False use_cpu: bool = False use_mps_device: bool = False seed: int = 42 data_seed: typing.Optional[int] = None jit_mode_eval: bool = False use_ipex: bool = False bf16: bool = False fp16: bool = False fp16_opt_level: str = 'O1' half_precision_backend: str = 'auto' bf16_full_eval: bool = False fp16_full_eval: bool = False tf32: typing.Optional[bool] = None local_rank: int = -1 ddp_backend: t


yping.Optional[str] = None tpu_num_cores: typing.Optional[int] = None tpu_metrics_debug: bool = False debug: typing.Union[str, typing.List[transformers.debug_utils.DebugOption]] = '' dataloader_drop_last: bool = False eval_steps: typing.Optional[float] = None dataloader_num_workers: int = 0 past_index: int = -1 run_name: typing.Optional[str] = None disable_tqdm: typing.Optional[bool] = None remove_unused_columns: typing.Optional[bool] = True label_names: typing.Optional[typing.List[str]] = None load_best_model_at_end: typing.Optional[bool] = False metric_for_best_model: typing.Optional[str] = None greater_is_better: typing.Optional[bool] = None ignore_data_skip: bool = False fsdp: typing.Union[typing.List[transformers.trainer_utils.FSDPOption], str, NoneType] = '' fsdp_min_num_params: int = 0 fsdp_config: typing.Optional[str] = None fsdp_transformer_layer_cls_to_wrap: typing.Optional[str] = None deepspeed: typing.Optional[str] = None label_smoothing_factor: float = 0


.0 optim: typing.Union[transformers.training_args.OptimizerNames, str] = 'adamw_torch' optim_args: typing.Optional[str] = None adafactor: bool = False group_by_length: bool = False length_column_name: typing.Optional[str] = 'length' report_to: typing.Optional[typing.List[str]] = None ddp_find_unused_parameters: typing.Optional[bool] = None ddp_bucket_cap_mb: typing.Optional[int] = None ddp_broadcast_buffers: typing.Optional[bool] = None dataloader_pin_memory: bool = True skip_memory_metrics: bool = True use_legacy_prediction_loop: bool = False push_to_hub: bool = False resume_from_checkpoint: typing.Optional[str] = None hub_model_id: typing.Optional[str] = None hub_strategy: typing.Union[transformers.trainer_utils.HubStrategy, str] = 'every_save' hub_token: typing.Optional[str] = None hub_private_repo: bool = False hub_always_push: bool = False gradient_checkpointing: bool = False include_inputs_for_metrics: bool = False fp16_backend: str = 'auto' push_to_hub_model_i


d: typing.Optional[str] = None push_to_hub_organization: typing.Optional[str] = None push_to_hub_token: typing.Optional[str] = None mp_parameters: str = '' auto_find_batch_size: bool = False full_determinism: bool = False torchdynamo: typing.Optional[str] = None ray_scope: typing.Optional[str] = 'last' ddp_timeout: typing.Optional[int] = 1800 torch_compile: bool = False torch_compile_backend: typing.Optional[str] = None torch_compile_mode: typing.Optional[str] = None dispatch_batches: typing.Optional[bool] = None include_tokens_per_second: typing.Optional[bool] = False sortish_sampler: bool = False predict_with_generate: bool = False generation_max_length: typing.Optional[int] = None generation_num_beams: typing.Optional[int] = None generation_config: typing.Union[str, pathlib.Path, transformers.generation.configuration_utils.GenerationConfig, NoneType] = None  )   Parameters   output_dir (str) —
The output directory where the model predictions and checkpoints will b


e written.   overwrite_output_dir (bool, optional, defaults to False) —
If True, overwrite the content of the output directory. Use this to continue training if output_dir
points to a checkpoint directory.   do_train (bool, optional, defaults to False) —
Whether to run training or not. This argument is not directly used by Trainer, it’s intended to be used
by your training/evaluation scripts instead. See the example
scripts for more details.   do_eval (bool, optional) —
Whether to run evaluation on the validation set or not. Will be set to True if evaluation_strategy is
different from "no". This argument is not directly used by Trainer, it’s intended to be used by your
training/evaluation scripts instead. See the example
scripts for more details.   do_predict (bool, optional, defaults to False) —
Whether to run predictions on the test set or not. This argument is not directly used by Trainer, it’s
intended to be used by your training/evaluation scripts instead. See t


he example
scripts for more details.   evaluation_strategy (str or IntervalStrategy, optional, defaults to "no") —
The evaluation strategy to adopt during training. Possible values are:
"no": No evaluation is done during training.
"steps": Evaluation is done (and logged) every eval_steps.
"epoch": Evaluation is done at the end of each epoch.
   prediction_loss_only (bool, optional, defaults to False) —
When performing evaluation and generating predictions, only returns the loss.   per_device_train_batch_size (int, optional, defaults to 8) —
The batch size per GPU/XPU/TPU/MPS/NPU core/CPU for training.   per_device_eval_batch_size (int, optional, defaults to 8) —
The batch size per GPU/XPU/TPU/MPS/NPU core/CPU for evaluation.   gradient_accumulation_steps (int, optional, defaults to 1) —
Number of updates steps to accumulate the gradients for, before performing a backward/update pass.
When using gradient accumulation, one step is counted as one step with backward pass


. Therefore, logging,
evaluation, save will be conducted every gradient_accumulation_steps * xxx_step training examples.
   eval_accumulation_steps (int, optional) —
Number of predictions steps to accumulate the output tensors for, before moving the results to the CPU. If
left unset, the whole predictions are accumulated on GPU/NPU/TPU before being moved to the CPU (faster but
requires more memory).   eval_delay (float, optional) —
Number of epochs or steps to wait for before the first evaluation can be performed, depending on the
evaluation_strategy.   learning_rate (float, optional, defaults to 5e-5) —
The initial learning rate for AdamW optimizer.   weight_decay (float, optional, defaults to 0) —
The weight decay to apply (if not zero) to all layers except all bias and LayerNorm weights in AdamW
optimizer.   adam_beta1 (float, optional, defaults to 0.9) —
The beta1 hyperparameter for the AdamW optimizer.   adam_beta2 (float, optional, defaults to 0.999) —
The beta


2 hyperparameter for the AdamW optimizer.   adam_epsilon (float, optional, defaults to 1e-8) —
The epsilon hyperparameter for the AdamW optimizer.   max_grad_norm (float, optional, defaults to 1.0) —
Maximum gradient norm (for gradient clipping).   num_train_epochs(float, optional, defaults to 3.0) —
Total number of training epochs to perform (if not an integer, will perform the decimal part percents of
the last epoch before stopping training).   max_steps (int, optional, defaults to -1) —
If set to a positive number, the total number of training steps to perform. Overrides num_train_epochs.
In case of using a finite iterable dataset the training may stop before reaching the set number of steps
when all data is exhausted   lr_scheduler_type (str or SchedulerType, optional, defaults to "linear") —
The scheduler type to use. See the documentation of SchedulerType for all possible values.   warmup_ratio (float, optional, defaults to 0.0) —
Ratio of total training steps


used for a linear warmup from 0 to learning_rate.   warmup_steps (int, optional, defaults to 0) —
Number of steps used for a linear warmup from 0 to learning_rate. Overrides any effect of warmup_ratio.   log_level (str, optional, defaults to passive) —
Logger log level to use on the main process. Possible choices are the log levels as strings: ‘debug’,
‘info’, ‘warning’, ‘error’ and ‘critical’, plus a ‘passive’ level which doesn’t set anything and keeps the
current log level for the Transformers library (which will be "warning" by default).   log_level_replica (str, optional, defaults to "warning") —
Logger log level to use on replicas. Same choices as log_level”   log_on_each_node (bool, optional, defaults to True) —
In multinode distributed training, whether to log using log_level once per node, or only on the main
node.   logging_dir (str, optional) —
TensorBoard log directory. Will default to
*output_dir/runs/CURRENT_DATETIME_HOSTNAME*.   logging_strategy (str or


 IntervalStrategy, optional, defaults to "steps") —
The logging strategy to adopt during training. Possible values are:
"no": No logging is done during training.
"epoch": Logging is done at the end of each epoch.
"steps": Logging is done every logging_steps.
   logging_first_step (bool, optional, defaults to False) —
Whether to log and evaluate the first global_step or not.   logging_steps (int or float, optional, defaults to 500) —
Number of update steps between two logs if logging_strategy="steps". Should be an integer or a float in
range [0,1). If smaller than 1, will be interpreted as ratio of total training steps.   logging_nan_inf_filter (bool, optional, defaults to True) —
Whether to filter nan and inf losses for logging. If set to True the loss of every step that is nan
or inf is filtered and the average loss of the current logging window is taken instead.
logging_nan_inf_filter only influences the logging of loss values, it does not change the behavior the
g


radient is computed or applied to the model.
   save_strategy (str or IntervalStrategy, optional, defaults to "steps") —
The checkpoint save strategy to adopt during training. Possible values are:
"no": No save is done during training.
"epoch": Save is done at the end of each epoch.
"steps": Save is done every save_steps.
   save_steps (int or float, optional, defaults to 500) —
Number of updates steps before two checkpoint saves if save_strategy="steps". Should be an integer or a
float in range [0,1). If smaller than 1, will be interpreted as ratio of total training steps.   save_total_limit (int, optional) —
If a value is passed, will limit the total amount of checkpoints. Deletes the older checkpoints in
output_dir. When load_best_model_at_end is enabled, the “best” checkpoint according to
metric_for_best_model will always be retained in addition to the most recent ones. For example, for
save_total_limit=5 and load_best_model_at_end, the four last checkpoints will


 always be retained
alongside the best model. When save_total_limit=1 and load_best_model_at_end, it is possible that two
checkpoints are saved: the last one and the best one (if they are different).   save_safetensors (bool, optional, defaults to False) —
Use safetensors saving and loading for state dicts instead of
default torch.load and torch.save.   save_on_each_node (bool, optional, defaults to False) —
When doing multi-node distributed training, whether to save models and checkpoints on each node, or only on
the main one.
This should not be activated when the different nodes use the same storage as the files will be saved with
the same names for each node.   use_cpu (bool, optional, defaults to False) —
Whether or not to use cpu. If set to False, we will use cuda or mps device if available.   seed (int, optional, defaults to 42) —
Random seed that will be set at the beginning of training. To ensure reproducibility across runs, use the
~Trainer.model_init functi


on to instantiate the model if it has some randomly initialized parameters.   data_seed (int, optional) —
Random seed to be used with data samplers. If not set, random generators for data sampling will use the
same seed as seed. This can be used to ensure reproducibility of data sampling, independent of the model
seed.   jit_mode_eval (bool, optional, defaults to False) —
Whether or not to use PyTorch jit trace for inference.   use_ipex (bool, optional, defaults to False) —
Use Intel extension for PyTorch when it is available. IPEX
installation.   bf16 (bool, optional, defaults to False) —
Whether to use bf16 16-bit (mixed) precision training instead of 32-bit training. Requires Ampere or higher
NVIDIA architecture or using CPU (use_cpu) or Ascend NPU. This is an experimental API and it may change.   fp16 (bool, optional, defaults to False) —
Whether to use fp16 16-bit (mixed) precision training instead of 32-bit training.   fp16_opt_level (str, optional, defaults to


 ‘O1’) —
For fp16 training, Apex AMP optimization level selected in [‘O0’, ‘O1’, ‘O2’, and ‘O3’]. See details on
the Apex documentation.   fp16_backend (str, optional, defaults to "auto") —
This argument is deprecated. Use half_precision_backend instead.   half_precision_backend (str, optional, defaults to "auto") —
The backend to use for mixed precision training. Must be one of "auto", "apex", "cpu_amp". "auto" will
use CPU/CUDA AMP or APEX depending on the PyTorch version detected, while the other choices will force the
requested backend.   bf16_full_eval (bool, optional, defaults to False) —
Whether to use full bfloat16 evaluation instead of 32-bit. This will be faster and save memory but can harm
metric values. This is an experimental API and it may change.   fp16_full_eval (bool, optional, defaults to False) —
Whether to use full float16 evaluation instead of 32-bit. This will be faster and save memory but can harm
metric values.   tf32 (bool, optional) —
Whethe


r to enable the TF32 mode, available in Ampere and newer GPU architectures. The default value depends
on PyTorch’s version default of torch.backends.cuda.matmul.allow_tf32. For more details please refer to
the TF32 documentation. This is an
experimental API and it may change.   local_rank (int, optional, defaults to -1) —
Rank of the process during distributed training.   ddp_backend (str, optional) —
The backend to use for distributed training. Must be one of "nccl", "mpi", "ccl", "gloo", "hccl".   tpu_num_cores (int, optional) —
When training on TPU, the number of TPU cores (automatically passed by launcher script).   dataloader_drop_last (bool, optional, defaults to False) —
Whether to drop the last incomplete batch (if the length of the dataset is not divisible by the batch size)
or not.   eval_steps (int or float, optional) —
Number of update steps between two evaluations if evaluation_strategy="steps". Will default to the same
value as logging_steps if not set.


 Should be an integer or a float in range [0,1). If smaller than 1,
will be interpreted as ratio of total training steps.   dataloader_num_workers (int, optional, defaults to 0) —
Number of subprocesses to use for data loading (PyTorch only). 0 means that the data will be loaded in the
main process.   past_index (int, optional, defaults to -1) —
Some models like TransformerXL or XLNet can make use of
the past hidden states for their predictions. If this argument is set to a positive int, the Trainer will
use the corresponding output (usually index 2) as the past state and feed it to the model at the next
training step under the keyword argument mems.   run_name (str, optional) —
A descriptor for the run. Typically used for wandb and
mlflow logging.   disable_tqdm (bool, optional) —
Whether or not to disable the tqdm progress bars and table of metrics produced by
~notebook.NotebookTrainingTracker in Jupyter Notebooks. Will default to True if the logging level is
set t


o warn or lower (default), False otherwise.   remove_unused_columns (bool, optional, defaults to True) —
Whether or not to automatically remove the columns unused by the model forward method.
(Note that this behavior is not implemented for TFTrainer yet.)   label_names (List[str], optional) —
The list of keys in your dictionary of inputs that correspond to the labels.
Will eventually default to the list of argument names accepted by the model that contain the word “label”,
except if the model used is one of the XxxForQuestionAnswering in which case it will also include the
["start_positions", "end_positions"] keys.   load_best_model_at_end (bool, optional, defaults to False) —
Whether or not to load the best model found during training at the end of training. When this option is
enabled, the best checkpoint will always be saved. See
save_total_limit
for more.
When set to True, the parameters save_strategy needs to be the same as evaluation_strategy, and in
the case i


t is “steps”, save_steps must be a round multiple of eval_steps.
   metric_for_best_model (str, optional) —
Use in conjunction with load_best_model_at_end to specify the metric to use to compare two different
models. Must be the name of a metric returned by the evaluation with or without the prefix "eval_". Will
default to "loss" if unspecified and load_best_model_at_end=True (to use the evaluation loss).
If you set this value, greater_is_better will default to True. Don’t forget to set it to False if
your metric is better when lower.   greater_is_better (bool, optional) —
Use in conjunction with load_best_model_at_end and metric_for_best_model to specify if better models
should have a greater metric or not. Will default to:
True if metric_for_best_model is set to a value that isn’t "loss" or "eval_loss".
False if metric_for_best_model is not set, or set to "loss" or "eval_loss".
   ignore_data_skip (bool, optional, defaults to False) —
When resuming training, whethe


r or not to skip the epochs and batches to get the data loading at the same
stage as in the previous training. If set to True, the training will begin faster (as that skipping step
can take a long time) but will not yield the same results as the interrupted training would have.   fsdp (bool, str or list of FSDPOption, optional, defaults to '') —
Use PyTorch Distributed Parallel Training (in distributed training only).
A list of options along the following:
"full_shard": Shard parameters, gradients and optimizer states.
"shard_grad_op": Shard optimizer states and gradients.
"offload": Offload parameters and gradients to CPUs (only compatible with "full_shard" and
"shard_grad_op").
"auto_wrap": Automatically recursively wrap layers with FSDP using default_auto_wrap_policy.
   fsdp_config (str or dict, optional) —
Config to be used with fsdp (Pytorch Distributed Parallel Training). The value is either a location of
deepspeed json config file (e.g., ds_config.json) or an


 already loaded json file as dict.
A List of config and its options:
min_num_params (int, optional, defaults to 0):
FSDP’s minimum number of parameters for Default Auto Wrapping. (useful only when fsdp field is
passed).
transformer_layer_cls_to_wrap (List[str], optional):
List of transformer layer class names (case-sensitive) to wrap, e.g, BertLayer, GPTJBlock,
T5Block … (useful only when fsdp flag is passed).
backward_prefetch (str, optional)
FSDP’s backward prefetch mode. Controls when to prefetch next set of parameters (useful only when
fsdp field is passed).
A list of options along the following:
"backward_pre" : Prefetches the next set of parameters before the current set of parameter’s
gradient
computation.
"backward_post" : This prefetches the next set of parameters after the current set of
parameter’s
gradient computation.
forward_prefetch (bool, optional, defaults to False)
FSDP’s forward prefetch mode (useful only when fsdp field is passed).
If "True", then


 FSDP explicitly prefetches the next upcoming all-gather while executing in the
forward pass.
limit_all_gathers (bool, optional, defaults to False)
FSDP’s limit_all_gathers (useful only when fsdp field is passed).
If "True", FSDP explicitly synchronizes the CPU thread to prevent too many in-flight
all-gathers.
use_orig_params (bool, optional, defaults to False)
If "True", allows non-uniform requires_grad during init, which means support for interspersed
frozen and trainable paramteres. Useful in cases such as parameter-efficient fine-tuning. Please
refer this
[blog](https://dev-discuss.pytorch.org/t/rethinking-pytorch-fully-sharded-data-parallel-fsdp-from-first-principles/1019
sync_module_states (bool, optional, defaults to True)
If "True", each individually wrapped FSDP unit will broadcast module parameters from rank 0 to
ensure they are the same across all ranks after initialization
xla (bool, optional, defaults to False):
Whether to use PyTorch/XLA Fully Sharded D


ata Parallel Training. This is an experimental feature
and its API may evolve in the future.
xla_fsdp_settings (dict, optional)
The value is a dictionary which stores the XLA FSDP wrapping parameters.
For a complete list of options, please see here.
xla_fsdp_grad_ckpt (bool, optional, defaults to False):
Will use gradient checkpointing over each nested XLA FSDP wrapped layer. This setting can only be
used when the xla flag is set to true, and an auto wrapping policy is specified through
fsdp_min_num_params or fsdp_transformer_layer_cls_to_wrap.
activation_checkpointing (bool, optional, defaults to False):
If True, activation checkpointing is a technique to reduce memory usage by clearing activations of
certain layers and recomputing them during a backward pass. Effectively, this trades extra
computation time for reduced memory usage.
   deepspeed (str or dict, optional) —
Use Deepspeed. This is an experimental feature and its API may
evolve in the future. The value i


s either the location of DeepSpeed json config file (e.g.,
ds_config.json) or an already loaded json file as a dict”   label_smoothing_factor (float, optional, defaults to 0.0) —
The label smoothing factor to use. Zero means no label smoothing, otherwise the underlying onehot-encoded
labels are changed from 0s and 1s to label_smoothing_factor/num_labels and 1 - label_smoothing_factor + label_smoothing_factor/num_labels respectively.   debug (str or list of DebugOption, optional, defaults to "") —
Enable one or more debug features. This is an experimental feature.
Possible options are:
"underflow_overflow": detects overflow in model’s input/outputs and reports the last frames that led to
the event
"tpu_metrics_debug": print debug metrics on TPU
The options should be separated by whitespaces.   optim (str or training_args.OptimizerNames, optional, defaults to "adamw_torch") —
The optimizer to use: adamw_hf, adamw_torch, adamw_torch_fused, adamw_apex_fused, adamw_anypre


cision or
adafactor.   optim_args (str, optional) —
Optional arguments that are supplied to AnyPrecisionAdamW.   group_by_length (bool, optional, defaults to False) —
Whether or not to group together samples of roughly the same length in the training dataset (to minimize
padding applied and be more efficient). Only useful if applying dynamic padding.   length_column_name (str, optional, defaults to "length") —
Column name for precomputed lengths. If the column exists, grouping by length will use these values rather
than computing them on train startup. Ignored unless group_by_length is True and the dataset is an
instance of Dataset.   report_to (str or List[str], optional, defaults to "all") —
The list of integrations to report the results and logs to. Supported platforms are "azure_ml",
"clearml", "codecarbon", "comet_ml", "dagshub", "flyte", "mlflow", "neptune",
"tensorboard", and "wandb". Use "all" to report to all integrations installed, "none" for no
integration


s.   ddp_find_unused_parameters (bool, optional) —
When using distributed training, the value of the flag find_unused_parameters passed to
DistributedDataParallel. Will default to False if gradient checkpointing is used, True otherwise.   ddp_bucket_cap_mb (int, optional) —
When using distributed training, the value of the flag bucket_cap_mb passed to DistributedDataParallel.   ddp_broadcast_buffers (bool, optional) —
When using distributed training, the value of the flag broadcast_buffers passed to
DistributedDataParallel. Will default to False if gradient checkpointing is used, True otherwise.   dataloader_pin_memory (bool, optional, defaults to True) —
Whether you want to pin memory in data loaders or not. Will default to True.   skip_memory_metrics (bool, optional, defaults to True) —
Whether to skip adding of memory profiler reports to metrics. This is skipped by default because it slows
down the training and evaluation speed.   push_to_hub (bool, optional, defa


ults to False) —
Whether or not to push the model to the Hub every time the model is saved. If this is activated,
output_dir will begin a git directory synced with the repo (determined by hub_model_id) and the content
will be pushed each time a save is triggered (depending on your save_strategy). Calling
save_model() will also trigger a push.
If output_dir exists, it needs to be a local clone of the repository to which the Trainer will be
pushed.
   resume_from_checkpoint (str, optional) —
The path to a folder with a valid checkpoint for your model. This argument is not directly used by
Trainer, it’s intended to be used by your training/evaluation scripts instead. See the example
scripts for more details.   hub_model_id (str, optional) —
The name of the repository to keep in sync with the local output_dir. It can be a simple model ID in
which case the model will be pushed in your namespace. Otherwise it should be the whole repository name,
for instance "user_name/mod


el", which allows you to push to an organization you are a member of with
"organization_name/model". Will default to user_name/output_dir_name with output_dir_name being the
name of output_dir.
Will default to the name of output_dir.   hub_strategy (str or HubStrategy, optional, defaults to "every_save") —
Defines the scope of what is pushed to the Hub and when. Possible values are:
"end": push the model, its configuration, the tokenizer (if passed along to the Trainer) and a
draft of a model card when the save_model() method is called.
"every_save": push the model, its configuration, the tokenizer (if passed along to the Trainer) and
a draft of a model card each time there is a model save. The pushes are asynchronous to not block
training, and in case the save are very frequent, a new push is only attempted if the previous one is
finished. A last push is made with the final model at the end of training.
"checkpoint": like "every_save" but the latest checkpoint is al


so pushed in a subfolder named
last-checkpoint, allowing you to resume training easily with
trainer.train(resume_from_checkpoint="last-checkpoint").
"all_checkpoints": like "checkpoint" but all checkpoints are pushed like they appear in the output
folder (so you will get one checkpoint folder per folder in your final repository)
   hub_token (str, optional) —
The token to use to push the model to the Hub. Will default to the token in the cache folder obtained with
huggingface-cli login.   hub_private_repo (bool, optional, defaults to False) —
If True, the Hub repo will be set to private.   hub_always_push (bool, optional, defaults to False) —
Unless this is True, the Trainer will skip pushing a checkpoint when the previous push is not finished.   gradient_checkpointing (bool, optional, defaults to False) —
If True, use gradient checkpointing to save memory at the expense of slower backward pass.   include_inputs_for_metrics (bool, optional, defaults to False) —
Wheth


er or not the inputs will be passed to the compute_metrics function. This is intended for metrics
that need inputs, predictions and references for scoring calculation in Metric class.   auto_find_batch_size (bool, optional, defaults to False) —
Whether to find a batch size that will fit into memory automatically through exponential decay, avoiding
CUDA Out-of-Memory errors. Requires accelerate to be installed (pip install accelerate)   full_determinism (bool, optional, defaults to False) —
If True, enable_full_determinism() is called instead of set_seed() to ensure reproducible results in
distributed training. Important: this will negatively impact the performance, so only use it for debugging.   torchdynamo (str, optional) —
If set, the backend compiler for TorchDynamo. Possible choices are "eager", "aot_eager", "inductor",
"nvfuser", "aot_nvfuser", "aot_cudagraphs", "ofi", "fx2trt", "onnxrt" and "ipex".   ray_scope (str, optional, defaults to "last") —
The scope to


 use when doing hyperparameter search with Ray. By default, "last" will be used. Ray will
then use the last checkpoint of all trials, compare those, and select the best one. However, other options
are also available. See the Ray documentation for
more options.   ddp_timeout (int, optional, defaults to 1800) —
The timeout for torch.distributed.init_process_group calls, used to avoid GPU socket timeouts when
performing slow operations in distributed runnings. Please refer the [PyTorch documentation]
(https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group) for more
information.   use_mps_device (bool, optional, defaults to False) —
This argument is deprecated.mps device will be used if it is available similar to cuda device.   torch_compile (bool, optional, defaults to False) —
Whether or not to compile the model using PyTorch 2.0
torch.compile.
This will use the best defaults for the torch.compile
API.
You can customize the defaults with


the argument torch_compile_backend and torch_compile_mode but we
don’t guarantee any of them will work as the support is progressively rolled in in PyTorch.
This flag and the whole compile API is experimental and subject to change in future releases.   torch_compile_backend (str, optional) —
The backend to use in torch.compile. If set to any value, torch_compile will be set to True.
Refer to the PyTorch doc for possible values and note that they may change across PyTorch versions.
This flag is experimental and subject to change in future releases.   torch_compile_mode (str, optional) —
The mode to use in torch.compile. If set to any value, torch_compile will be set to True.
Refer to the PyTorch doc for possible values and note that they may change across PyTorch versions.
This flag is experimental and subject to change in future releases.   include_tokens_per_second (bool, optional) —
Whether or not to compute the number of tokens per second per device for training s


peed metrics.
This will iterate over the entire training dataloader once beforehand,
and will slow down the entire process.   sortish_sampler (bool, optional, defaults to False) —
Whether to use a sortish sampler or not. Only possible if the underlying datasets are Seq2SeqDataset
for now but will become generally available in the near future.
It sorts the inputs according to lengths in order to minimize the padding size, with a bit of randomness
for the training set.   predict_with_generate (bool, optional, defaults to False) —
Whether to use generate to calculate generative metrics (ROUGE, BLEU).   generation_max_length (int, optional) —
The max_length to use on each evaluation loop when predict_with_generate=True. Will default to the
max_length value of the model configuration.   generation_num_beams (int, optional) —
The num_beams to use on each evaluation loop when predict_with_generate=True. Will default to the
num_beams value of the model configuration.   gener


ation_config (str or Path or GenerationConfig, optional) —
Allows to load a GenerationConfig from the from_pretrained method. This can be either:
a string, the model id of a pretrained model configuration hosted inside a model repo on
huggingface.co. Valid model ids can be located at the root-level, like bert-base-uncased, or namespaced
under a user or organization name, like dbmdz/bert-base-german-cased.
a path to a directory containing a configuration file saved using the
save_pretrained() method, e.g., ./my_model_directory/.
a GenerationConfig object.
    TrainingArguments is the subset of the arguments we use in our example scripts which relate to the training loop
itself. Using HfArgumentParser we can turn this class into
argparse arguments that can be specified on the
command line.  to_dict  < source > ( )    Serializes this instance while replace Enum by their values and GenerationConfig by dictionaries (for JSON
serialization support). It obfuscates the token


Checkpoints

By default, Trainer will save all checkpoints in the output_dir you set in the
TrainingArguments you are using. Those will go in subfolder named checkpoint-xxx with xxx
being the step at which the training was at.
Resuming training from a checkpoint can be done when calling Trainer.train() with either:
resume_from_checkpoint=True which will resume training from the latest checkpoint resume_from_checkpoint=checkpoint_dir which will resume training from the specific checkpoint in the directory
passed.
In addition, you can easily save your checkpoints on the Model Hub when u


Checkpoints

sing push_to_hub=True. By default, all
the models saved in intermediate checkpoints are saved in different commits, but not the optimizer state. You can adapt
the hub-strategy value of your TrainingArguments to either:
"checkpoint": the latest checkpoint is also pushed in a subfolder named last-checkpoint, allowing you to
resume training easily with trainer.train(resume_from_checkpoint="output_dir/last-checkpoint"). "all_checkpoints": all checkpoints are pushed like they appear in the output folder (so you will get one
checkpoint folder per folder in your final repository


Logging

By default Trainer will use logging.INFO for the main process and logging.WARNING for the replicas if any.
These defaults can be overridden to use any of the 5 logging levels with TrainingArguments’s
arguments:
log_level - for the main process log_level_replica - for the replicas
Further, if TrainingArguments’s log_on_each_node is set to False only the main node will
use the log level settings for its main process, all other nodes will use the log level settings for replicas.
Note that Trainer is going to set transformers’s log level separately for each node in its
Trainer.__init__(). So you may want to set this sooner (see the next example) if you tap into other
transformers functionality before creating the Train


Logging

er object.
Here is an example of how this can be used in an application:
  Copied [...]
logger = logging.getLogger(__name__)
# Setup logging
logging.basicConfig(
    format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
    datefmt="%m/%d/%Y %H:%M:%S",
    handlers=[logging.StreamHandler(sys.stdout)],
)
# set the main code and the modules it uses to the same log-level according to the node
log_level = training_args.get_process_log_level()
logger.setLevel(log_level)
datasets.utils.logging.set_verbosity(log_level)
transformers.utils.logging.set_verbosity(log_level)
trainer = Trainer(...)
And then if you only want to see warnings on the main node and all other nodes to not print any most likely duplicated
war


Logging

nings you could run it as:
  Copied my_app.py ... --log_level warning --log_level_replica error
In the multi-node environment if you also don’t want the logs to repeat for each node’s main process, you will want to
change the above to:
  Copied my_app.py ... --log_level warning --log_level_replica error --log_on_each_node 0
and then only the main process of the first node will log at the “warning” level, and all other processes on the main
node and all processes on other nodes will log at the “error” level.
If you need your application to be as quiet as possible you could do:
  Copied my_app.py ... --log_level error --log_level_replica error --log_on_each_node 0
(add --log_on_each_node 0 if on multi-node environmen


Randomness

When resuming from a checkpoint generated by Trainer all efforts are made to restore the
python, numpy and pytorch RNG states to the same states as they were at the moment of saving that checkpoint,
which should make the “stop and resume” style of training as close as possible to non-stop training.
However, due to various default non-deterministic pytorch settings this might not fully work. If you want full
determinism please refer to Controlling sources of randomness. As explained in the document, that some of those settings
that make things deterministic (.e.g., torch.backends.cudnn.deterministic) may slow things down, therefore this
can’t be done by default, but you can enable those yourself if needed.


Specific GPUs Selection

Let’s discuss how you can tell your program which GPUs are to be used and in what order.
When using DistributedDataParallel to use only a subset of your GPUs, you simply specify the number of GPUs to use. For example, if you have 4 GPUs, but you wish to use the first 2 you can do:
  Copied python -m torch.distributed.launch --nproc_per_node=2  trainer-program.py ...
if you have either accelerate or deepspeed installed you can also accomplish the same by using one of:
  Copied accelerate launch --num_processes 2 trainer-program.py ...
  Copied deepspeed --num_gpus 2 trainer-program.py ...
You don’t need to use the Accelerate or the Deepspeed integration features to use these launchers.
Until now you were able to tell the program how many GPUs to use. Now let’s discuss how to select specific GPUs and control their order.
The following environment variables help you control which GPUs to use and their order.
CUDA_VISIBLE_DEVICES
If y


Specific GPUs Selection

ou have multiple GPUs and you’d like to use only 1 or a few of those GPUs, set the environment variable CUDA_VISIBLE_DEVICES to a list of the GPUs to be used.
For example, let’s say you have 4 GPUs: 0, 1, 2 and 3. To run only on the physical GPUs 0 and 2, you can do:
  Copied CUDA_VISIBLE_DEVICES=0,2 python -m torch.distributed.launch trainer-program.py ...
So now pytorch will see only 2 GPUs, where your physical GPUs 0 and 2 are mapped to cuda:0 and cuda:1 correspondingly.
You can even change their order:
  Copied CUDA_VISIBLE_DEVICES=2,0 python -m torch.distributed.launch trainer-program.py ...
Here your physical GPUs 0 and 2 are mapped to cuda:1 and cuda:0 correspondingly.
The above examples were all for DistributedDataParallel use pattern, but the same method works for DataParallel as well:
  Copied CUDA_VISIBLE_DEVICES=2,0 python trainer-program.py ...
To emulate an environment without GPUs simply set this environment variabl


Specific GPUs Selection

e to an empty value like so:
  Copied CUDA_VISIBLE_DEVICES= python trainer-program.py ...
As with any environment variable you can, of course, export those instead of adding these to the command line, as in:
  Copied export CUDA_VISIBLE_DEVICES=0,2
python -m torch.distributed.launch trainer-program.py ...
but this approach can be confusing since you may forget you set up the environment variable earlier and not understand why the wrong GPUs are used. Therefore, it’s a common practice to set the environment variable just for a specific run on the same command line as it’s shown in most examples of this section.
CUDA_DEVICE_ORDER
There is an additional environment variable CUDA_DEVICE_ORDER that controls how the physical devices are ordered. The two choices are:
ordered by PCIe bus IDs (matches nvidia-smi’s order) - this is the default.
  Copied export CUDA_DEVICE_ORDER=PCI_BUS_ID
ordered by GPU compute capabilities
  Copied export


Specific GPUs Selection

CUDA_DEVICE_ORDER=FASTEST_FIRST
Most of the time you don’t need to care about this environment variable, but it’s very helpful if you have a lopsided setup where you have an old and a new GPUs physically inserted in such a way so that the slow older card appears to be first. One way to fix that is to swap the cards. But if you can’t swap the cards (e.g., if the cooling of the devices gets impacted) then setting CUDA_DEVICE_ORDER=FASTEST_FIRST will always put the newer faster card first. It’ll be somewhat confusing though since nvidia-smi will still report them in the PCIe order.
The other solution to swapping the order is to use:
  Copied export CUDA_VISIBLE_DEVICES=1,0
In this example we are working with just 2 GPUs, but of course the same would apply to as many GPUs as your computer has.
Also if you do set this environment variable it’s the best to set it in your ~/.bashrc file or some other startup config file and forget about


Trainer Integrations

The Trainer has been extended to support libraries that may dramatically improve your training
time and fit much bigger models.
Currently it supports third party solutions, DeepSpeed and PyTorch FSDP, which implement parts of the paper ZeRO: Memory Optimizations
Toward Training Trillion Parameter Models, by Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He.
This provided support is new and experimental as of this writing. While the support for DeepSpeed and PyTorch FSDP is active and we welcome issues around it, we don’t support the FairScale integration anymore since it has been integrated in PyTorch main (see the PyTorch FSDP integration)


CUDA Extension Installation Notes

As of this writing, Deepspeed require compilation of CUDA C++ code, before it can be used.
While all installation issues should be dealt with through the corresponding GitHub Issues of Deepspeed, there are a few common issues that one may encounter while building
any PyTorch extension that needs to build CUDA extensions.
Therefore, if you encounter a CUDA-related build issue while doing the following:
  Copied pip install deepspeed
please, read the following notes first.
In these notes we give examples for what to do when pytorch has been built with CUDA 10.2. If your situation is
different remember to adjust the version number to the one you are after.
 Possible problem #1
While, Pytorch comes with its own CUDA toolkit, to build these two projects you must have an identical version of CUDA
installed system-wide.
For example, if


CUDA Extension Installation Notes

you installed pytorch with cudatoolkit==10.2 in the Python environment, you also need to have
CUDA 10.2 installed system-wide.
The exact location may vary from system to system, but /usr/local/cuda-10.2 is the most common location on many
Unix systems. When CUDA is correctly set up and added to the PATH environment variable, one can find the
installation location by doing:
  Copied which nvcc
If you don’t have CUDA installed system-wide, install it first. You will find the instructions by using your favorite
search engine. For example, if you’re on Ubuntu you may want to search for: ubuntu cuda 10.2 install.
 Possible problem #2
Another possible common problem is that you may have more than one CUDA toolkit installed system-wide. For example you
may have:
  Copied /usr/local/cuda-10.2
/usr/local/cuda-11.0
Now, in this situation y


CUDA Extension Installation Notes

ou need to make sure that your PATH and LD_LIBRARY_PATH environment variables contain
the correct paths to the desired CUDA version. Typically, package installers will set these to contain whatever the
last version was installed. If you encounter the problem, where the package build fails because it can’t find the right
CUDA version despite you having it installed system-wide, it means that you need to adjust the 2 aforementioned
environment variables.
First, you may look at their contents:
  Copied echo $PATH
echo $LD_LIBRARY_PATH
so you get an idea of what is inside.
It’s possible that LD_LIBRARY_PATH is empty.
PATH lists the locations of where executables can be found and LD_LIBRARY_PATH is for where shared libraries
are to looked for. In both cases, earlier entries have priority over the later ones. : is used to separate mult


CUDA Extension Installation Notes

iple
entries.
Now, to tell the build program where to find the specific CUDA toolkit, insert the desired paths to be listed first by
doing:
  Copied export PATH=/usr/local/cuda-10.2/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-10.2/lib64:$LD_LIBRARY_PATH
Note that we aren’t overwriting the existing values, but prepending instead.
Of course, adjust the version number, the full path if need be. Check that the directories you assign actually do
exist. lib64 sub-directory is where the various CUDA .so objects, like libcudart.so reside, it’s unlikely
that your system will have it named differently, but if it is adjust it to reflect your reality.
 Possible problem #3
Some older CUDA versions may refuse to build with newer compilers. For example, you my have gcc-9 but it wants
gcc-7.
There are various ways to go about it.
If you can


CUDA Extension Installation Notes

 install the latest CUDA toolkit it typically should support the newer compiler.
Alternatively, you could install the lower version of the compiler in addition to the one you already have, or you may
already have it but it’s not the default one, so the build system can’t see it. If you have gcc-7 installed but the
build system complains it can’t find it, the following might do the trick:
  Copied sudo ln -s /usr/bin/gcc-7  /usr/local/cuda-10.2/bin/gcc
sudo ln -s /usr/bin/g++-7  /usr/local/cuda-10.2/bin/g++
Here, we are making a symlink to gcc-7 from /usr/local/cuda-10.2/bin/gcc and since
/usr/local/cuda-10.2/bin/ should be in the PATH environment variable (see the previous problem’s solution), it
should find gcc-7 (and g++7) and then the build will succeed.
As always make sure to edit the paths in the example to match your situat


PyTorch Fully Sharded Data parallel

To accelerate training huge models on larger batch sizes, we can use a fully sharded data parallel model.
This type of data parallel paradigm enables fitting more data and larger models by sharding the optimizer states, gradients and parameters.
To read more about it and the benefits, check out the Fully Sharded Data Parallel blog.
We have integrated the latest PyTorch’s Fully Sharded Data Parallel (FSDP) training feature.
All you need to do is enable it through the config.
Required PyTorch version for FSDP support: PyTorch Nightly (or 1.12.0 if you read this after it has been released)
as the model saving with FSDP activated is only available with recent fixes.
Usage:
Make sure you have added the distributed launcher
-m torch.distributed.launch --nproc_per_node=NUMBER_OF_GPUS_YOU_HAVE if you haven’t be


PyTorch Fully Sharded Data parallel

en using it already. Sharding Strategy: FULL_SHARD : Shards optimizer states + gradients + model parameters across data parallel workers/GPUs.
For this, add --fsdp full_shard to the command line arguments. SHARD_GRAD_OP : Shards optimizer states + gradients across data parallel workers/GPUs.
For this, add --fsdp shard_grad_op to the command line arguments. NO_SHARD : No sharding. For this, add --fsdp no_shard to the command line arguments. To offload the parameters and gradients to the CPU,
add --fsdp "full_shard offload" or --fsdp "shard_grad_op offload" to the command line arguments. To automatically recursively wrap layers with FSDP using default_auto_wrap_policy,
add --fsdp "full_shard auto_wrap" or --fsdp "shard_grad_op auto_wrap" to the command line arguments. To enable both CPU offloading and aut


PyTorch Fully Sharded Data parallel

o wrapping,
add --fsdp "full_shard offload auto_wrap" or --fsdp "shard_grad_op offload auto_wrap" to the command line arguments. Remaining FSDP config is passed via --fsdp_config <path_to_fsdp_config.json>. It is either a location of
FSDP json config file (e.g., fsdp_config.json) or an already loaded json file as dict. If auto wrapping is enabled, you can either use transformer based auto wrap policy or size based auto wrap policy.For transformer based auto wrap policy, it is recommended to specify fsdp_transformer_layer_cls_to_wrap in the config file. If not specified, the default value is model._no_split_modules when available.
This specifies the list of transformer layer class name (case-sensitive) to wrap ,e.g, BertLayer, GPTJBlock, T5Block …
This is important because submodules that share weights (


PyTorch Fully Sharded Data parallel

e.g., embedding layer) should not end up in different FSDP wrapped units.
Using this policy, wrapping happens for each block containing Multi-Head Attention followed by couple of MLP layers.
Remaining layers including the shared embeddings are conveniently wrapped in same outermost FSDP unit.
Therefore, use this for transformer based models. For size based auto wrap policy, please add fsdp_min_num_params in the config file.
It specifies FSDP’s minimum number of parameters for auto wrapping. fsdp_backward_prefetch can be specified in the config file. It controls when to prefetch next set of parameters.
backward_pre and backward_pos are available options.
For more information refer torch.distributed.fsdp.fully_sharded_data_parallel.BackwardPrefetch fsdp_forward_prefetch can be specified in the config file


PyTorch Fully Sharded Data parallel

. It controls when to prefetch next set of parameters.
If "True", FSDP explicitly prefetches the next upcoming all-gather while executing in the forward pass. limit_all_gathers can be specified in the config file.
If "True", FSDP explicitly synchronizes the CPU thread to prevent too many in-flight all-gathers. activation_checkpointing can be specified in the config file.
If "True", FSDP activation checkpointing is a technique to reduce memory usage by clearing activations of
certain layers and recomputing them during a backward pass. Effectively, this trades extra computation time
for reduced memory usage.
Few caveats to be aware of
it is incompatible with generate, thus is incompatible with --predict_with_generate
in all seq2seq/clm scripts (translation/summarization/clm etc.).
Please refer issue #2166


PyTorch/XLA Fully Sharded Data parallel

For all the TPU users, great news! PyTorch/XLA now supports FSDP.
All the latest Fully Sharded Data Parallel (FSDP) training are supported.
For more information refer to the Scaling PyTorch models on Cloud TPUs with FSDP and PyTorch/XLA implementation of FSDP
All you need to do is enable it through the config.
Required PyTorch/XLA version for FSDP support: >=2.0
Usage:
Pass --fsdp "full shard" along with following changes to be made in --fsdp_config <path_to_fsdp_config.json>:
xla should be set to True to enable PyTorch/XLA FSDP. xla_fsdp_settings The value is a dictionary which stores the XLA FSDP wrapping parameters.
For a complete list of options, please see here. xla_fsdp_grad_ckpt. When True, uses gradient checkpointing over each nested XLA FSDP wrapped layer.
This setting can only be used when the xla flag is set to true, and an auto wrapping policy is specified through
fsdp_min_num_params or fsdp_transformer_layer_cls_to_wrap. You c


PyTorch/XLA Fully Sharded Data parallel

an either use transformer based auto wrap policy or size based auto wrap policy.For transformer based auto wrap policy, it is recommended to specify fsdp_transformer_layer_cls_to_wrap in the config file. If not specified, the default value is model._no_split_modules when available.
This specifies the list of transformer layer class name (case-sensitive) to wrap ,e.g, BertLayer, GPTJBlock, T5Block …
This is important because submodules that share weights (e.g., embedding layer) should not end up in different FSDP wrapped units.
Using this policy, wrapping happens for each block containing Multi-Head Attention followed by couple of MLP layers.
Remaining layers including the shared embeddings are conveniently wrapped in same outermost FSDP unit.
Therefore, use this for transformer based models. For size based auto wrap policy, please add fsdp_min_num_params in the config file.
It specifies FSDP’s minimum number of parameters for auto wrapping


Using Trainer for accelerated PyTorch Training on Mac

With PyTorch v1.12 release, developers and researchers can take advantage of Apple silicon GPUs for significantly faster model training.
This unlocks the ability to perform machine learning workflows like prototyping and fine-tuning locally, right on Mac.
Apple’s Metal Performance Shaders (MPS) as a backend for PyTorch enables this and can be used via the new "mps" device.
This will map computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
For more information please refer official documents Introducing Accelerated PyTorch Training on Mac
and MPS BACKEND.
We strongly recommend to install PyTorch >= 1.13 (nightly version at the time of writing) on your MacOS machine.
It has major fixes related to model correctness and performance improvements for transformer based models.
Please refer to https://github.com/pytorch/pytorch/issues/82707 for more


details.
Benefits of Training and Inference using Apple Silicon Chips
Enables users to train larger networks or batch sizes locally Reduces data retrieval latency and provides the GPU with direct access to the full memory store due to unified memory architecture.
Therefore, improving end-to-end performance. Reduces costs associated with cloud-based development or the need for additional local GPUs.
Pre-requisites: To install torch with mps support,
please follow this nice medium article GPU-Acceleration Comes to PyTorch on M1 Macs.
Usage:
mps device will be used by default if available similar to the way cuda device is used.
Therefore, no action from user is required.
For example, you can run the official Glue text classififcation task (from the root folder) using Apple Silicon GPU with below command:
  Copied export TASK_NAME=mrpc
python examples/pytorch/text-classification/run_glue.py


 \
  --model_name_or_path bert-base-cased \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir /tmp/$TASK_NAME/ \
  --overwrite_output_dir
A few caveats to be aware of
Some PyTorch operations have not been implemented in mps and will throw an error.
One way to get around that is to set the environment variable PYTORCH_ENABLE_MPS_FALLBACK=1,
which will fallback to CPU for these operations. It still throws a UserWarning however. Distributed setups gloo and nccl are not working with mps device.
This means that currently only single GPU of mps device type can be used.
Finally, please, remember that, 🤗 Trainer only integrates MPS backend, therefore if you
have any problems or questions with regards to MPS backend usage, please,
file an issue with PyTorch GitHub.


Using Accelerate Launcher with Trainer

Accelerate now powers Trainer. In terms of what users should expect:
They can keep using the Trainer ingterations such as FSDP, DeepSpeed vis trainer arguments without any changes on their part. They can now use Accelerate Launcher with Trainer (recommended).
Steps to use Accelerate Launcher with Trainer:
Make sure 🤗 Accelerate is installed, you can’t use the Trainer without it anyway. If not pip install accelerate. You may also need to update your version of Accelerate: pip install accelerate --upgrade Run accelerate config and fill the questionnaire. Below are example accelerate configs:
a. DDP Multi-node Multi-GPU config:   Copied compute_environment: LOCAL_MACHINE
distributed_type: MULTI_GPU
downcast_bf16: 'no'
gpu_ids: all
machine_rank: 0 #change rank as


Using Accelerate Launcher with Trainer

per the node
main_process_ip: 192.168.20.1
main_process_port: 9898
main_training_function: main
mixed_precision: fp16
num_machines: 2
num_processes: 8
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false b. FSDP config:   Copied compute_environment: LOCAL_MACHINE
distributed_type: FSDP
downcast_bf16: 'no'
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch_policy: BACKWARD_PRE
  fsdp_forward_prefetch: true
  fsdp_offload_params: false
  fsdp_sharding_strategy: 1
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_sync_module_states: true
  fsdp_transformer_layer_cls_to_wrap: BertLayer
  fsdp_use_orig_params: true
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false c. DeepSpeed config pointing to a file:   Copi


Using Accelerate Launcher with Trainer

ed compute_environment: LOCAL_MACHINE
deepspeed_config:
  deepspeed_config_file: /home/user/configs/ds_zero3_config.json
  zero3_init_flag: true
distributed_type: DEEPSPEED
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
num_machines: 1
num_processes: 4
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false d. DeepSpeed config using accelerate plugin:   Copied compute_environment: LOCAL_MACHINE
deepspeed_config:
  gradient_accumulation_steps: 1
  gradient_clipping: 0.7
  offload_optimizer_device: cpu
  offload_param_device: cpu
  zero3_init_flag: true
  zero_stage: 2
distributed_type: DEEPSPEED
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
mixed_precision: bf16


Using Accelerate Launcher with Trainer

num_machines: 1
num_processes: 4
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false Run the Trainer script with args other than the ones handled above by accelerate config or launcher args.
Below is an example to run run_glue.py using accelerate launcher with FSDP config from above.
  Copied cd transformers
accelerate launch \
./examples/pytorch/text-classification/run_glue.py \
--model_name_or_path bert-base-cased \
--task_name $TASK_NAME \
--do_train \
--do_eval \
--max_seq_length 128 \
--per_device_train_batch_size 16 \
--learning_rate 5e-5 \
--num_train_epochs 3 \
--output_dir /tmp/$TASK_NAME/ \
--overwrite_output_dir
You can also directly use the cmd args for accelerate launch. Above example would map to:
  Copied cd transformers
accelerate launch --num_processes=2 \
--use_fsdp \
--mixed_precision=bf16 \
--fsdp_auto_wrap_policy=TRANSFORMER_BASED_WRAP  \
--fsdp_transformer_layer_cls_to_


Using Accelerate Launcher with Trainer

wrap="BertLayer" \
--fsdp_sharding_strategy=1 \
--fsdp_state_dict_type=FULL_STATE_DICT \
./examples/pytorch/text-classification/run_glue.py
--model_name_or_path bert-base-cased \
--task_name $TASK_NAME \
--do_train \
--do_eval \
--max_seq_length 128 \
--per_device_train_batch_size 16 \
--learning_rate 5e-5 \
--num_train_epochs 3 \
--output_dir /tmp/$TASK_NAME/ \
--overwrite_output_dir
For more information, please refer the 🤗 Accelerate CLI guide: Launching your 🤗 Accelerate scripts.
Sections that were moved:
[ DeepSpeed
| Installation
| Deployment with multiple GPUs
| Deployment with one GPU
| Deployment in Notebooks
| Configuration
| Passing Configuration
| Shared Configuration
| ZeRO
| ZeRO-2 Config
| ZeRO-3 Config
| NVMe Support
| ZeRO-2 vs ZeRO-3 Performance
| ZeRO-2 Example
| ZeRO-3 Example
| Optimizer
| Scheduler
| fp32 Precision
| Automatic Mixed Precision
| Batch Size
| Gradient Accumulation
| Gradient Clipping
| Getting The Model Weights Ou


Agents & Tools

Transformers Agents is an experimental API which is subject to change at any time. Results returned by the agents
can vary as the APIs or underlying models are prone to change.
To learn more about agents and tools make sure to read the introductory guide. This page
contains the API docs for the underlying classes.


Agents

We provide three types of agents: HfAgent uses inference endpoints for opensource models, LocalAgent uses a model of your choice locally and OpenAiAgent uses OpenAI closed models.


HfAgent

class transformers.HfAgent  < source > ( url_endpoint token = None chat_prompt_template = None run_prompt_template = None additional_tools = None  )   Parameters   url_endpoint (str) —
The name of the url endpoint to use.   token (str, optional) —
The token to use as HTTP bearer authorization for remote files. If unset, will use the token generated when
running huggingface-cli login (stored in ~/.huggingface).   chat_prompt_template (str, optional) —
Pass along your own prompt if you want to override the default template for the chat method. Can be the
actual prompt template or a repo ID (on the Hugging Face Hub). The prompt should be in a file named
chat_prompt_template.txt in this repo in this case.   run_prompt_template (str, optional) —
Pass along your own prompt if


HfAgent

you want to override the default template for the run method. Can be the
actual prompt template or a repo ID (on the Hugging Face Hub). The prompt should be in a file named
run_prompt_template.txt in this repo in this case.   additional_tools (Tool, list of tools or dictionary with tool values, optional) —
Any additional tools to include on top of the default ones. If you pass along a tool with the same name as
one of the default tools, that default tool will be overridden.    Agent that uses an inference endpoint to generate code.  Example:   Copied from transformers import HfAgent
agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder")
agent.run("Is the following `text` (in Spanish) positive or negative?", text="¡Este es un API muy agradable!")


LocalAgent

class transformers.LocalAgent  < source > ( model tokenizer chat_prompt_template = None run_prompt_template = None additional_tools = None  )   Parameters   model (PreTrainedModel) —
The model to use for the agent.   tokenizer (PreTrainedTokenizer) —
The tokenizer to use for the agent.   chat_prompt_template (str, optional) —
Pass along your own prompt if you want to override the default template for the chat method. Can be the
actual prompt template or a repo ID (on the Hugging Face Hub). The prompt should be in a file named
chat_prompt_template.txt in this repo in this case.   run_prompt_template (str, optional) —
Pass along your own prompt if you want to override the default template for the run method. Can be the
actual p


LocalAgent

rompt template or a repo ID (on the Hugging Face Hub). The prompt should be in a file named
run_prompt_template.txt in this repo in this case.   additional_tools (Tool, list of tools or dictionary with tool values, optional) —
Any additional tools to include on top of the default ones. If you pass along a tool with the same name as
one of the default tools, that default tool will be overridden.    Agent that uses a local model and tokenizer to generate code.  Example:   Copied import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, LocalAgent
checkpoint = "bigcode/starcoder"
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto", torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from


LocalAgent

_pretrained(checkpoint)
agent = LocalAgent(model, tokenizer)
agent.run("Draw me a picture of rivers and lakes.")  from_pretrained  < source > ( pretrained_model_name_or_path **kwargs  )   Parameters   pretrained_model_name_or_path (str or os.PathLike) —
The name of a repo on the Hub or a local path to a folder containing both model and tokenizer.   kwargs (Dict[str, Any], optional) —
Keyword arguments passed along to from_pretrained().    Convenience method to build a LocalAgent from a pretrained checkpoint.  Example:   Copied import torch
from transformers import LocalAgent
agent = LocalAgent.from_pretrained("bigcode/starcoder", device_map="auto", torch_dtype=torch.bfloat16)
agent.run("Draw me a picture of rivers and lakes."


OpenAiAgent

class transformers.OpenAiAgent  < source > ( model = 'text-davinci-003' api_key = None chat_prompt_template = None run_prompt_template = None additional_tools = None  )   Parameters   model (str, optional, defaults to "text-davinci-003") —
The name of the OpenAI model to use.   api_key (str, optional) —
The API key to use. If unset, will look for the environment variable "OPENAI_API_KEY".   chat_prompt_template (str, optional) —
Pass along your own prompt if you want to override the default template for the chat method. Can be the
actual prompt template or a repo ID (on the Hugging Face Hub). The prompt should be in a file named
chat_prompt_template.txt in this repo in this case.   run_prompt_template (str, optional) —
Pass along your own prompt if you want to override the default template for the run method. Can be the
actual prompt template or a repo


OpenAiAgent

ID (on the Hugging Face Hub). The prompt should be in a file named
run_prompt_template.txt in this repo in this case.   additional_tools (Tool, list of tools or dictionary with tool values, optional) —
Any additional tools to include on top of the default ones. If you pass along a tool with the same name as
one of the default tools, that default tool will be overridden.    Agent that uses the openai API to generate code. The openAI models are used in generation mode, so even for the chat() API, it’s better to use models like
"text-davinci-003" over the chat-GPT variant. Proper support for chat-GPT models will come in a next version.  Example:   Copied from transformers import OpenAiAgent
agent = OpenAiAgent(model="text-davinci-003", api_key=xxx)
agent.run("Is the following `text` (in Spanish) positive or negative?", text="¡Este es un API muy agradable!"


AzureOpenAiAgent

class transformers.AzureOpenAiAgent  < source > ( deployment_id api_key = None resource_name = None api_version = '2022-12-01' is_chat_model = None chat_prompt_template = None run_prompt_template = None additional_tools = None  )   Parameters   deployment_id (str) —
The name of the deployed Azure openAI model to use.   api_key (str, optional) —
The API key to use. If unset, will look for the environment variable "AZURE_OPENAI_API_KEY".   resource_name (str, optional) —
The name of your Azure OpenAI Resource. If unset, will look for the environment variable
"AZURE_OPENAI_RESOURCE_NAME".   api_version (str, optional, default to "2022-12-01") —
The API version to use for this agent.   is_chat_mode (bool, optional) —
Whether you are using a completion model or a chat mod


AzureOpenAiAgent

el (see note above, chat models won’t be as
efficient). Will default to gpt being in the deployment_id or not.   chat_prompt_template (str, optional) —
Pass along your own prompt if you want to override the default template for the chat method. Can be the
actual prompt template or a repo ID (on the Hugging Face Hub). The prompt should be in a file named
chat_prompt_template.txt in this repo in this case.   run_prompt_template (str, optional) —
Pass along your own prompt if you want to override the default template for the run method. Can be the
actual prompt template or a repo ID (on the Hugging Face Hub). The prompt should be in a file named
run_prompt_template.txt in this repo in this case.   additional_tools (Tool, list of tools or dictionary with tool values, opt


AzureOpenAiAgent

ional) —
Any additional tools to include on top of the default ones. If you pass along a tool with the same name as
one of the default tools, that default tool will be overridden.    Agent that uses Azure OpenAI to generate code. See the official
documentation to learn how to deploy an openAI
model on Azure The openAI models are used in generation mode, so even for the chat() API, it’s better to use models like
"text-davinci-003" over the chat-GPT variant. Proper support for chat-GPT models will come in a next version.  Example:   Copied from transformers import AzureOpenAiAgent
agent = AzureAiAgent(deployment_id="Davinci-003", api_key=xxx, resource_name=yyy)
agent.run("Is the following `text` (in Spanish) positive or negative?", text="¡Este es un API muy agradable!"


Agent

class transformers.Agent  < source > ( chat_prompt_template = None run_prompt_template = None additional_tools = None  )   Parameters   chat_prompt_template (str, optional) —
Pass along your own prompt if you want to override the default template for the chat method. Can be the
actual prompt template or a repo ID (on the Hugging Face Hub). The prompt should be in a file named
chat_prompt_template.txt in this repo in this case.   run_prompt_template (str, optional) —
Pass along your own prompt if you want to override the default template for the run method. Can be the
actual prompt template or a repo ID (on the Hugging Face Hub). The prompt should be in a file named
run_prompt_template.txt in this repo in this case.   additional_tools (Tool, list of tools or dictionary with tool values, optional) —
Any additional tools to include on top of the default ones. If you


Agent

 pass along a tool with the same name as
one of the default tools, that default tool will be overridden.    Base class for all agents which contains the main API methods.  chat  < source > ( task return_code = False remote = False **kwargs  )   Parameters   task (str) — The task to perform   return_code (bool, optional, defaults to False) —
Whether to just return code and not evaluate it.   remote (bool, optional, defaults to False) —
Whether or not to use remote tools (inference endpoints) instead of local ones.   kwargs (additional keyword arguments, optional) —
Any keyword argument to send to the agent when evaluating the code.    Sends a new request to the agent in a chat. Will use the previous ones in its history.  Example:   Copied from transformers import HfAgent
agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder")
agent.chat("D


Agent

raw me a picture of rivers and lakes")
agent.chat("Transform the picture so that there is a rock in there")  run  < source > ( task return_code = False remote = False **kwargs  )   Parameters   task (str) — The task to perform   return_code (bool, optional, defaults to False) —
Whether to just return code and not evaluate it.   remote (bool, optional, defaults to False) —
Whether or not to use remote tools (inference endpoints) instead of local ones.   kwargs (additional keyword arguments, optional) —
Any keyword argument to send to the agent when evaluating the code.    Sends a request to the agent.  Example:   Copied from transformers import HfAgent
agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder")
agent.run("Draw me a picture of rivers and lakes")  prepare_for_new_chat  < source > ( )    Clears the history of prior calls to chat(


load_tool

transformers.load_tool  < source > ( task_or_repo_id model_repo_id = None remote = False token = None **kwargs  )   Parameters   task_or_repo_id (str) —
The task for which to load the tool or a repo ID of a tool on the Hub. Tasks implemented in Transformers
are:
"document-question-answering"
"image-captioning"
"image-question-answering"
"image-segmentation"
"speech-to-text"
"summarization"
"text-classification"
"text-question-answering"
"text-to-speech"
"translation"
   model_repo_id (str, optional) —
Use this argument to use a different model than the default one for the tool you selected.   remote (bool, optional, defaults to False)


load_tool

—
Whether to use your tool by downloading the model or (if it is available) with an inference endpoint.   token (str, optional) —
The token to identify you on hf.co. If unset, will use the token generated when running huggingface-cli login (stored in ~/.huggingface).   kwargs (additional keyword arguments, optional) —
Additional keyword arguments that will be split in two: all arguments relevant to the Hub (such as
cache_dir, revision, subfolder) will be used when downloading the files for your tool, and the others
will be passed along to its init.    Main function to quickly load a tool, be it on the Hub or in the Transformers library


Tool

class transformers.Tool  < source > ( *args **kwargs  )    A base class for the functions used by the agent. Subclass this and implement the __call__ method as well as the
following class attributes: description (str) — A short description of what your tool does, the inputs it expects and the output(s) it
will return. For instance ‘This is a tool that downloads a file from a url. It takes the url as input, and
returns the text contained in the file’. name (str) — A performative name that will be used for your tool in the prompt to the agent. For instance
"text-classifier" or "image_generator". inputs (List[str]) — The list of modalities expected for the inputs (in the same order as in the call).
Modalitiies should be "text", "image" or "audio". This is only used by launch_gradio_demo or to make a
nice space


Tool

 from your tool. outputs (List[str]) — The list of modalities returned but the tool (in the same order as the return of the
call method). Modalitiies should be "text", "image" or "audio". This is only used by launch_gradio_demo
or to make a nice space from your tool. You can also override the method setup() if your tool as an expensive operation to perform before being
usable (such as loading a model). setup() will be called the first time you use your tool, but not at
instantiation.  from_gradio  < source > ( gradio_tool  )    Creates a Tool from a gradio tool.  from_hub  < source > ( repo_id: str model_repo_id: typing.Optional[str] = None token: typing.Optional[str] = None remote: bool = False **kwargs  )   Parameters   repo_id (str) —
The name of the repo on the Hub where your tool is defined.   model_re


Tool

po_id (str, optional) —
If your tool uses a model and you want to use a different model than the default, you can pass a second
repo ID or an endpoint url to this argument.   token (str, optional) —
The token to identify you on hf.co. If unset, will use the token generated when running
huggingface-cli login (stored in ~/.huggingface).   remote (bool, optional, defaults to False) —
Whether to use your tool by downloading the model or (if it is available) with an inference endpoint.   kwargs (additional keyword arguments, optional) —
Additional keyword arguments that will be split in two: all arguments relevant to the Hub (such as
cache_dir, revision, subfolder) will be used when downloading the files for your tool, and the
others will be passed along to its init.    Loads a tool defined on the Hub.  push_to_


Tool

hub  < source > ( repo_id: str commit_message: str = 'Upload tool' private: typing.Optional[bool] = None token: typing.Union[bool, str, NoneType] = None create_pr: bool = False  )   Parameters   repo_id (str) —
The name of the repository you want to push your tool to. It should contain your organization name when
pushing to a given organization.   commit_message (str, optional, defaults to "Upload tool") —
Message to commit while pushing.   private (bool, optional) —
Whether or not the repository created should be private.   token (bool or str, optional) —
The token to use as HTTP bearer authorization for remote files. If unset, will use the token generated
when running huggingface-cli login (stored in ~/.huggingface).   create_pr (bool, optional, defaults to False) —
Whether or not to create a PR with the


Tool

uploaded files or directly commit.    Upload the tool to the Hub.  save  < source > ( output_dir  )   Parameters   output_dir (str) — The folder in which you want to save your tool.    Saves the relevant code files for your tool so it can be pushed to the Hub. This will copy the code of your
tool in output_dir as well as autogenerate: a config file named tool_config.json an app.py file so that your tool can be converted to a space a requirements.txt containing the names of the module used by your tool (as detected when inspecting its
code) You should only use this method to save tools that are defined in a separate module (not __main__).  setup  < source > ( )    Overwrite this method here for any operation that is expensive and needs to be executed before you start using
your tool. Such as loading a big mo


PipelineTool

class transformers.PipelineTool  < source > ( model = None pre_processor = None post_processor = None device = None device_map = None model_kwargs = None token = None **hub_kwargs  )   Parameters   model (str or PreTrainedModel, optional) —
The name of the checkpoint to use for the model, or the instantiated model. If unset, will default to the
value of the class attribute default_checkpoint.   pre_processor (str or Any, optional) —
The name of the checkpoint to use for the pre-processor, or the instantiated pre-processor (can be a
tokenizer, an image processor, a feature extractor or a processor). Will default to the value of model if
unset.   post_processor (str or Any, optional) —
The name of the checkpoint to use for the post-processor, or the instantiated pre-processor (can be a
tokenizer, an image processor, a feature extr


PipelineTool

actor or a processor). Will default to the pre_processor if
unset.   device (int, str or torch.device, optional) —
The device on which to execute the model. Will default to any accelerator available (GPU, MPS etc…), the
CPU otherwise.   device_map (str or dict, optional) —
If passed along, will be used to instantiate the model.   model_kwargs (dict, optional) —
Any keyword argument to send to the model instantiation.   token (str, optional) —
The token to use as HTTP bearer authorization for remote files. If unset, will use the token generated when
running huggingface-cli login (stored in ~/.huggingface).   hub_kwargs (additional keyword arguments, optional) —
Any additional keyword argument to send to the methods that will load the data from the Hub.    A Tool tailored towards Transformer models. On top of the class attributes


PipelineTool

of the base class Tool, you will
need to specify: model_class (type) — The class to use to load the model in this tool. default_checkpoint (str) — The default checkpoint that should be used when the user doesn’t specify one. pre_processor_class (type, optional, defaults to AutoProcessor) — The class to use to load the
pre-processor post_processor_class (type, optional, defaults to AutoProcessor) — The class to use to load the
post-processor (when different from the pre-processor).  decode  < source > ( outputs  )    Uses the post_processor to decode the model output.  encode  < source > ( raw_inputs  )    Uses the pre_processor to prepare the inputs for the model.  forward  < source > ( inputs  )    Sends the inputs through the model.  setup  < source > ( )    Instantiates the pre_processor, model and post_processor if necessary


RemoteTool

class transformers.RemoteTool  < source > ( endpoint_url = None token = None tool_class = None  )   Parameters   endpoint_url (str, optional) —
The url of the endpoint to use.   token (str, optional) —
The token to use as HTTP bearer authorization for remote files. If unset, will use the token generated when
running huggingface-cli login (stored in ~/.huggingface).   tool_class (type, optional) —
The corresponding tool_class if this is a remote version of an existing tool. Will help determine when
the output should be converted to another type (like images).    A


RemoteTool

Tool that will make requests to an inference endpoint.  extract_outputs  < source > ( outputs  )    You can override this method in your custom class of RemoteTool to apply some custom post-processing of the
outputs of the endpoint.  prepare_inputs  < source > ( *args **kwargs  )    Prepare the inputs received for the HTTP client sending data to the endpoint. Positional arguments will be
matched with the signature of the tool_class if it was provided at instantation. Images will be encoded into
bytes. You can override this method in your custom class of RemoteTool


launch_gradio_demo

transformers.launch_gradio_demo  < source > ( tool_class: Tool  )   Parameters   tool_class (type) — The class of the tool for which to launch the demo.    Launches a gradio demo for a tool. The corresponding tool class needs to properly implement the class attributes
inputs and outputs.


Agent Types

Agents can handle any type of object in-between tools; tools, being completely multimodal, can accept and return
text, image, audio, video, among other types. In order to increase compatibility between tools, as well as to
correctly render these returns in ipython (jupyter, colab, ipython notebooks, …), we implement wrapper classes
around these types.
The wrapped objects should continue behaving as initially; a text object should still behave as a string, an image
object should still behave as a PIL.Image.
These types have three specific purposes:
Calling to_raw on the type should return the underlying object Calling to_string on the type should return the object as a string: that can be the string in case of an AgentText
but will be the path of the serialized version of the object in other instances Displaying it in an ipython kernel should display the object correctly


AgentText

class transformers.tools.agent_types.AgentText  < source > ( value  )    Text type returned by the agent. Behaves as a string.


AgentImage

class transformers.tools.agent_types.AgentImage  < source > ( value  )    Image type returned by the agent. Behaves as a PIL.Image.  to_raw  < source > ( )    Returns the “raw” version of that object. In the case of an AgentImage, it is a PIL.Image.  to_string  < source > ( )    Returns the stringified version of that object. In the case of an AgentImage, it is a path to the serialized
version of the image.


AgentAudio

class transformers.tools.agent_types.AgentAudio  < source > ( value samplerate = 16000  )    Audio type returned by the agent.  to_raw  < source > ( )    Returns the “raw” version of that object. It is a torch.Tensor object.  to_string  < source > ( )    Returns the stringified version of that object. In the case of an AgentAudio, it is a path to the serialized
version of the audio.


Callbacks

Callbacks are objects that can customize the behavior of the training loop in the PyTorch
Trainer (this feature is not yet implemented in TensorFlow) that can inspect the training loop
state (for progress reporting, logging on TensorBoard or other ML platforms…) and take decisions (like early
stopping).
Callbacks are “read only” pieces of code, apart from the TrainerControl object they return, they
cannot change anything in the training loop. For customizations that require changes in the training loop, you should
subclass Trainer and override the methods you need (see trainer for examples).
By default a Trainer will use the following callbacks:
DefaultFlowCallback which handles the default behavior for logging, saving and evaluation. PrinterCallback or ProgressCallback to display progress and print the
logs (t


Callbacks

he first one is used if you deactivate tqdm through the TrainingArguments, otherwise
it’s the second one). TensorBoardCallback if tensorboard is accessible (either through PyTorch >= 1.4
or tensorboardX). WandbCallback if wandb is installed. CometCallback if comet_ml is installed. MLflowCallback if mlflow is installed. NeptuneCallback if neptune is installed. AzureMLCallback if azureml-sdk is
installed. CodeCarbonCallback if codecarbon is
installed. ClearMLCallback if clearml is installed. DagsHubCallback if dagshub is installed. FlyteCallback if flyte is installed.
The main class that implements callbacks is TrainerCallback. It gets the
TrainingArguments used to instantiate the Trainer, can access that
Trainer’s internal state via TrainerState, and can take some actions on the training loop via
TrainerControl.


Available Callbacks

Here is the list of the available TrainerCallback in the library:
 class transformers.integrations.CometCallback  < source > ( )    A TrainerCallback that sends the logs to Comet ML.  setup  < source > ( args state model  )    Setup the optional Comet.ml integration. Environment: COMET_MODE (str, optional, defaults to ONLINE):
Whether to create an online, offline experiment or disable Comet logging. Can be OFFLINE, ONLINE, or
DISABLED. COMET_PROJECT_NAME (str, optional):
Comet project name for experiments. COMET_OFFLINE_DIRECTORY (str, optional):
Folder to use for saving offline experiments when COMET_MODE is OFFLINE. COMET_LOG_ASSETS (str, optional, defaults to TRUE):
Whether or not to log training assets (tf event logs, checkpoints, etc), to Comet. Can be TRUE, or
FALSE. For a number of configurable items in the environment, see
here.
 class transformers.DefaultFlowCallback  < source > ( )    A TrainerCallback tha


Available Callbacks

t handles the default flow of the training loop for logs, evaluation and checkpoints.
 class transformers.PrinterCallback  < source > ( )    A bare TrainerCallback that just prints the logs.
 class transformers.ProgressCallback  < source > ( )    A TrainerCallback that displays the progress of training or evaluation.
 class transformers.EarlyStoppingCallback  < source > ( early_stopping_patience: int = 1 early_stopping_threshold: typing.Optional[float] = 0.0  )   Parameters   early_stopping_patience (int) —
Use with metric_for_best_model to stop training when the specified metric worsens for
early_stopping_patience evaluation calls.   early_stopping_threshold(float, optional) —
Use with TrainingArguments metric_for_best_model and early_stopping_patience to denote how much the
specified metric must improve to satisfy early stopping conditions. `    A TrainerCallback that handles early stopping. This callback depends


Available Callbacks

on TrainingArguments argument load_best_model_at_end functionality to set best_metric
in TrainerState. Note that if the TrainingArguments argument save_steps differs from eval_steps, the
early stopping will not occur until the next save step.
 class transformers.integrations.TensorBoardCallback  < source > ( tb_writer = None  )   Parameters   tb_writer (SummaryWriter, optional) —
The writer to use. Will instantiate one if not set.    A TrainerCallback that sends the logs to TensorBoard.
 class transformers.integrations.WandbCallback  < source > ( )    A TrainerCallback that logs metrics, media, model checkpoints to Weight and Biases.  setup  < source > ( args state model **kwargs  )    Setup the optional Weights & Biases (wandb) integration. One can subclass and override this method to customize the setup if needed. Find more information
here. You can also override the following environment
variables: Environment: W


Available Callbacks

ANDB_LOG_MODEL (str, optional, defaults to "false"):
Whether to log model and checkpoints during training. Can be "end", "checkpoint" or "false". If set
to "end", the model will be uploaded at the end of training. If set to "checkpoint", the checkpoint
will be uploaded every args.save_steps . If set to "false", the model will not be uploaded. Use along
with load_best_model_at_end() to upload best model. Deprecated in 5.0 Setting WANDB_LOG_MODEL as bool will be deprecated in version 5 of 🤗 Transformers. WANDB_WATCH (str, optional defaults to "false"):
Can be "gradients", "all", "parameters", or "false". Set to "all" to log gradients and
parameters. WANDB_PROJECT (str, optional, defaults to "huggingface"):
Set this to a custom string to store results in a different project. WANDB_DISABLED (bool, optional, defaults to False):
Whether to disable wandb entirely. Set WANDB_DISABLED=true to disable.
 class transformers.int


Available Callbacks

egrations.MLflowCallback  < source > ( )    A TrainerCallback that sends the logs to MLflow. Can be disabled by setting
environment variable DISABLE_MLFLOW_INTEGRATION = TRUE.  setup  < source > ( args state model  )    Setup the optional MLflow integration. Environment: HF_MLFLOW_LOG_ARTIFACTS (str, optional):
Whether to use MLflow .log_artifact() facility to log artifacts. This only makes sense if logging to a
remote server, e.g. s3 or GCS. If set to True or 1, will copy each saved checkpoint on each save in
TrainingArguments’s output_dir to the local or remote artifact storage. Using it without a remote
storage will just copy the files to your artifact location. MLFLOW_EXPERIMENT_NAME (str, optional, defaults to None):
Whether to use an MLflow experiment_name under which to launch the run. Default to None which will point
to the Default experiment in MLflow. Otherwise, it is a case sensitive name of the experimen


Available Callbacks

t to be
activated. If an experiment with this name does not exist, a new experiment with this name is created. MLFLOW_TAGS (str, optional):
A string dump of a dictionary of key/value pair to be added to the MLflow run as tags. Example:
os.environ['MLFLOW_TAGS']='{"release.candidate": "RC1", "release.version": "2.2.0"}'. MLFLOW_NESTED_RUN (str, optional):
Whether to use MLflow nested runs. If set to True or 1, will create a nested run inside the current
run. MLFLOW_RUN_ID (str, optional):
Allow to reattach to an existing run which can be usefull when resuming training from a checkpoint. When
MLFLOW_RUN_ID environment variable is set, start_run attempts to resume a run with the specified run ID
and other parameters are ignored. MLFLOW_FLATTEN_PARAMS (str, optional, defaults to False):
Whether to flatten the parameters dictionary before logging.
 class transformers.integrations.AzureMLCallback  < source > ( azureml_run


Available Callbacks

 = None  )    A TrainerCallback that sends the logs to AzureML.
 class transformers.integrations.CodeCarbonCallback  < source > ( )    A TrainerCallback that tracks the CO2 emission of training.
 class transformers.integrations.NeptuneCallback  < source > ( api_token: typing.Optional[str] = None project: typing.Optional[str] = None name: typing.Optional[str] = None base_namespace: str = 'finetuning' run = None log_parameters: bool = True log_checkpoints: typing.Optional[str] = None **neptune_run_kwargs  )   Parameters   api_token (str, optional) — Neptune API token obtained upon registration.
You can leave this argument out if you have saved your token to the NEPTUNE_API_TOKEN environment
variable (strongly recommended). See full setup instructions in the
docs.   project (str, optional) — Name of an existing Neptune project, in the form “workspace-name/project-name”.
You can find and copy the name in Neptune from th


Available Callbacks

e project settings -> Properties. If None (default), the
value of the NEPTUNE_PROJECT environment variable is used.   name (str, optional) — Custom name for the run.   base_namespace (str, optional, defaults to “finetuning”) — In the Neptune run, the root namespace
that will contain all of the metadata logged by the callback.   log_parameters (bool, optional, defaults to True) —
If True, logs all Trainer arguments and model parameters provided by the Trainer.   log_checkpoints (str, optional) — If “same”, uploads checkpoints whenever they are saved by the Trainer.
If “last”, uploads only the most recently saved checkpoint. If “best”, uploads the best checkpoint (among
the ones saved by the Trainer). If None, does not upload checkpoints.   run (Run, optional) — Pass a Neptune run object if you want to continue logging to an existing run.
Read more about resuming runs in the docs.   **neptune_run_kwargs (optional) —
A


Available Callbacks

dditional keyword arguments to be passed directly to the
neptune.init_run() function when a new run is created.    TrainerCallback that sends the logs to Neptune. For instructions and examples, see the Transformers integration
guide in the Neptune documentation.
 class transformers.integrations.ClearMLCallback  < source > ( )    A TrainerCallback that sends the logs to ClearML. Environment: CLEARML_PROJECT (str, optional, defaults to HuggingFace Transformers):
ClearML project name. CLEARML_TASK (str, optional, defaults to Trainer):
ClearML task name. CLEARML_LOG_MODEL (bool, optional, defaults to False):
Whether to log models as artifacts during training.
 class transformers.integrations.DagsHubCallback  < source > ( )    A TrainerCallback that logs to DagsHub. Extends MLflowCallback  setup  < source > ( *args **kwargs  )    Setup the DagsHub’s Logging integration. Environment: HF_DAGSHUB_LOG_ARTIFACTS (str, optiona


Available Callbacks

l):
Whether to save the data and model artifacts for the experiment. Default to False.
 class transformers.integrations.FlyteCallback  < source > ( save_log_history: bool = True sync_checkpoints: bool = True  )   Parameters   save_log_history (bool, optional, defaults to True) —
When set to True, the training logs are saved as a Flyte Deck.   sync_checkpoints (bool, optional, defaults to True) —
When set to True, checkpoints are synced with Flyte and can be used to resume training in the case of an
interruption.    A TrainerCallback that sends the logs to Flyte.
NOTE: This callback only works within a Flyte task.  Example:   Copied # Note: This example skips over some setup steps for brevity.
from flytekit import current_context, task
@task
def train_hf_transformer():
    cp = current_context().checkpoint
    trainer = Trainer(..., callbacks=[FlyteCallback()])
    output = trainer.train(resume_from_checkpoint=cp.res