FlaxAutoModel

etModel (RegNet model)
ResNetConfig configuration class: FlaxResNetModel (ResNet model)
RoFormerConfig configuration class: FlaxRoFormerModel (RoFormer model)
RobertaConfig configuration class: FlaxRobertaModel (RoBERTa model)
RobertaPreLayerNormConfig configuration class: FlaxRobertaPreLayerNormModel (RoBERTa-PreLayerNorm model)
T5Config configuration class: FlaxT5Model (T5 model)
ViTConfig configuration class: FlaxViTModel (ViT model)
VisionTextDualEncoderConfig configuration class: FlaxVisionTextDualEncoderModel (VisionTextDualEncoder model)
Wav2Vec2Config configuration class: FlaxWav2Vec2Model (Wav2Vec2 model)
WhisperConfig configuration class: FlaxWhisperModel (Whisper model)
XGLMConfig configuration class: FlaxXGLMModel (XGLM model)
XLMRobertaConfig configuration class: FlaxXLMRobertaModel (XLM-RoBERTa model)
    Instantiates one of the base model classes of the library from a configuration. Note:
Loading


FlaxAutoModel

a model from its configuration file does not load the model weights. It only affects the
model’s configuration. Use from_pretrained() to load the model weights.  Examples:   Copied >>> from transformers import AutoConfig, FlaxAutoModel
>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("bert-base-cased")
>>> model = FlaxAutoModel.from_config(config)  from_pretrained  < source > ( *model_args **kwargs  )   Parameters   pretrained_model_name_or_path (str or os.PathLike) —
Can be either:
A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like bert-base-uncased, or namespaced under a
user or organization name, like dbmdz/bert-base-german-cased.
A path to a directory containing model weights saved using
save_pretrained(), e.g., ./my_model_directory/.
A path or url to a PyTorch sta


FlaxAutoModel

te_dict save file (e.g, ./pt_model/pytorch_model.bin). In this
case, from_pt should be set to True and a configuration object should be provided as config
argument. This loading path is slower than converting the PyTorch model in a TensorFlow model
using the provided conversion scripts and loading the TensorFlow model afterwards.
   model_args (additional positional arguments, optional) —
Will be passed along to the underlying model __init__() method.   config (PretrainedConfig, optional) —
Configuration for the model to use instead of an automatically loaded configuration. Configuration can
be automatically loaded when:
The model is a model provided by the library (loaded with the model id string of a pretrained
model).
The model was saved using save_pretrained() and is reloaded by supplying the
save directory.
The model is loaded by supplying a local directory as pretrained_model_name_or_path and a
configurati


FlaxAutoModel

on JSON file named config.json is found in the directory.
   cache_dir (str or os.PathLike, optional) —
Path to a directory in which a downloaded pretrained model configuration should be cached if the
standard cache should not be used.   from_pt (bool, optional, defaults to False) —
Load the model weights from a PyTorch checkpoint save file (see docstring of
pretrained_model_name_or_path argument).   force_download (bool, optional, defaults to False) —
Whether or not to force the (re-)download of the model weights and configuration files, overriding the
cached versions if they exist.   resume_download (bool, optional, defaults to False) —
Whether or not to delete incompletely received files. Will attempt to resume the download if such a
file exists.   proxies (Dict[str, str], optional) —
A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'


FlaxAutoModel

}. The proxies are used on each request.   output_loading_info(bool, optional, defaults to False) —
Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.   local_files_only(bool, optional, defaults to False) —
Whether or not to only look at local files (e.g., not try downloading the model).   revision (str, optional, defaults to "main") —
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so revision can be any
identifier allowed by git.   trust_remote_code (bool, optional, defaults to False) —
Whether or not to allow for custom models defined on the Hub in their own modeling files. This option
should only be set to True for repositories you trust and in which you have read the code, as it will
execute code present on the Hub on your loca


FlaxAutoModel

l machine.   code_revision (str, optional, defaults to "main") —
The specific revision to use for the code on the Hub, if the code leaves in a different repository than
the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based
system for storing models and other artifacts on huggingface.co, so revision can be any identifier
allowed by git.   kwargs (additional keyword arguments, optional) —
Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
output_attentions=True). Behaves differently depending on whether a config is provided or
automatically loaded:
If a configuration is provided with config, **kwargs will be directly passed to the
underlying model’s __init__ method (we assume all relevant updates to the configuration have
already been done)
If a configuration is not provided, kwargs will be first passed to the configur


FlaxAutoModel

ation class
initialization function (from_pretrained()). Each key of kwargs that
corresponds to a configuration attribute will be used to override said attribute with the
supplied kwargs value. Remaining keys that do not correspond to any configuration attribute
will be passed to the underlying model’s __init__ function.
    Instantiate one of the base model classes of the library from a pretrained model. The model class to instantiate is selected based on the model_type property of the config object (either
passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by
falling back to using pattern matching on pretrained_model_name_or_path: albert — FlaxAlbertModel (ALBERT model) bart — FlaxBartModel (BART model) beit — FlaxBeitModel (BEiT model) bert — FlaxBertModel (BERT model) big_bird — FlaxBigBirdModel (BigBird model) blenderbot — FlaxBlenderbotModel (Blenderbot m


FlaxAutoModel

odel) blenderbot-small — FlaxBlenderbotSmallModel (BlenderbotSmall model) bloom — FlaxBloomModel (BLOOM model) clip — FlaxCLIPModel (CLIP model) distilbert — FlaxDistilBertModel (DistilBERT model) electra — FlaxElectraModel (ELECTRA model) gpt-sw3 — FlaxGPT2Model (GPT-Sw3 model) gpt2 — FlaxGPT2Model (OpenAI GPT-2 model) gpt_neo — FlaxGPTNeoModel (GPT Neo model) gptj — FlaxGPTJModel (GPT-J model) longt5 — FlaxLongT5Model (LongT5 model) marian — FlaxMarianModel (Marian model) mbart — FlaxMBartModel (mBART model) mt5 — FlaxMT5Model (MT5 model) opt — FlaxOPTModel (OPT model) pegasus — FlaxPegasusModel (Pegasus model) regnet — FlaxRegNetModel (RegNet model) resnet — FlaxResNetModel (ResNet model) roberta — FlaxRobertaModel (RoBERTa model) roberta-prelayernorm — FlaxRobertaPreLayerNormModel (RoBERTa-PreLayerNorm model) roformer — FlaxRoFormerModel (RoFormer model) t5 — FlaxT5Model (T5 model) vision-text-dual-encoder —


FlaxAutoModel

 FlaxVisionTextDualEncoderModel (VisionTextDualEncoder model) vit — FlaxViTModel (ViT model) wav2vec2 — FlaxWav2Vec2Model (Wav2Vec2 model) whisper — FlaxWhisperModel (Whisper model) xglm — FlaxXGLMModel (XGLM model) xlm-roberta — FlaxXLMRobertaModel (XLM-RoBERTa model)  Examples:   Copied >>> from transformers import AutoConfig, FlaxAutoModel
>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxAutoModel.from_pretrained("bert-base-cased")
>>> # Update configuration during loading
>>> model = FlaxAutoModel.from_pretrained("bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True
>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = FlaxAutoModel.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
..


AutoModelForPreTraining

class transformers.AutoModelForPreTraining  < source > ( *args **kwargs  )    This is a generic model class that will be instantiated as one of the model classes of the library (with a pretraining head) when created
with the from_pretrained() class method or the from_config() class
method. This class cannot be instantiated directly using __init__() (throws an error).  from_config  < source > ( **kwargs  )   Parameters   config (PretrainedConfig) —
The model class to instantiate is selected based on the configuration class:
AlbertConfig configuration class: AlbertForPreTraining (ALBERT model)
BartConfig configuration class: BartForConditionalGeneration (BART model)
BertConfig configuration class: BertForPreTraining (BERT model)
BigBirdConfig configuration class: BigBirdForPreTraining (BigBird model)
BloomConfig configuration class: BloomForCausalLM (BLOOM model)
CTRLConfig configuration class: CTRLLMHeadModel (CTRL model)
CamembertConfig configuration class: CamembertForMaskedLM (Cam


AutoModelForPreTraining

emBERT model)
Data2VecTextConfig configuration class: Data2VecTextForMaskedLM (Data2VecText model)
DebertaConfig configuration class: DebertaForMaskedLM (DeBERTa model)
DebertaV2Config configuration class: DebertaV2ForMaskedLM (DeBERTa-v2 model)
DistilBertConfig configuration class: DistilBertForMaskedLM (DistilBERT model)
ElectraConfig configuration class: ElectraForPreTraining (ELECTRA model)
ErnieConfig configuration class: ErnieForPreTraining (ERNIE model)
FNetConfig configuration class: FNetForPreTraining (FNet model)
FSMTConfig configuration class: FSMTForConditionalGeneration (FairSeq Machine-Translation model)
FlaubertConfig configuration class: FlaubertWithLMHeadModel (FlauBERT model)
FlavaConfig configuration class: FlavaForPreTraining (FLAVA model)
FunnelConfig configuration class: FunnelForPreTraining (Funnel Transformer model)
GPT2Config configuration class: GPT2LMHeadModel (OpenAI GPT-2 model)
GPTBigCodeConfig configuration class: GPTBigCodeForCausalLM (GPTBigCode mode


AutoModelForPreTraining

l)
GPTSanJapaneseConfig configuration class: GPTSanJapaneseForConditionalGeneration (GPTSAN-japanese model)
IBertConfig configuration class: IBertForMaskedLM (I-BERT model)
IdeficsConfig configuration class: IdeficsForVisionText2Text (IDEFICS model)
LayoutLMConfig configuration class: LayoutLMForMaskedLM (LayoutLM model)
LongformerConfig configuration class: LongformerForMaskedLM (Longformer model)
LukeConfig configuration class: LukeForMaskedLM (LUKE model)
LxmertConfig configuration class: LxmertForPreTraining (LXMERT model)
MPNetConfig configuration class: MPNetForMaskedLM (MPNet model)
MegaConfig configuration class: MegaForMaskedLM (MEGA model)
MegatronBertConfig configuration class: MegatronBertForPreTraining (Megatron-BERT model)
MobileBertConfig configuration class: MobileBertForPreTraining (MobileBERT model)
MptConfig configuration class: MptForCausalLM (MPT model)
MraConfig configuration class: MraForMaskedLM (MRA model)
MvpConfig configuration class: MvpForConditionalGene


AutoModelForPreTraining

ration (MVP model)
NezhaConfig configuration class: NezhaForPreTraining (Nezha model)
NllbMoeConfig configuration class: NllbMoeForConditionalGeneration (NLLB-MOE model)
OpenAIGPTConfig configuration class: OpenAIGPTLMHeadModel (OpenAI GPT model)
RetriBertConfig configuration class: RetriBertModel (RetriBERT model)
RoCBertConfig configuration class: RoCBertForPreTraining (RoCBert model)
RobertaConfig configuration class: RobertaForMaskedLM (RoBERTa model)
RobertaPreLayerNormConfig configuration class: RobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model)
RwkvConfig configuration class: RwkvForCausalLM (RWKV model)
SplinterConfig configuration class: SplinterForPreTraining (Splinter model)
SqueezeBertConfig configuration class: SqueezeBertForMaskedLM (SqueezeBERT model)
SwitchTransformersConfig configuration class: SwitchTransformersForConditionalGeneration (SwitchTransformers model)
T5Config configuration class: T5ForConditionalGeneration (T5 model)
TapasConfig configuration c


AutoModelForPreTraining

lass: TapasForMaskedLM (TAPAS model)
TransfoXLConfig configuration class: TransfoXLLMHeadModel (Transformer-XL model)
TvltConfig configuration class: TvltForPreTraining (TVLT model)
UniSpeechConfig configuration class: UniSpeechForPreTraining (UniSpeech model)
UniSpeechSatConfig configuration class: UniSpeechSatForPreTraining (UniSpeechSat model)
ViTMAEConfig configuration class: ViTMAEForPreTraining (ViTMAE model)
VideoMAEConfig configuration class: VideoMAEForPreTraining (VideoMAE model)
VisualBertConfig configuration class: VisualBertForPreTraining (VisualBERT model)
Wav2Vec2Config configuration class: Wav2Vec2ForPreTraining (Wav2Vec2 model)
Wav2Vec2ConformerConfig configuration class: Wav2Vec2ConformerForPreTraining (Wav2Vec2-Conformer model)
XLMConfig configuration class: XLMWithLMHeadModel (XLM model)
XLMRobertaConfig configuration class: XLMRobertaForMaskedLM (XLM-RoBERTa model)
XLMRobertaXLConfig configuration class: XLMRobertaXLForMaskedLM (XLM-RoBERTa-XL model)
XLNetConfig


AutoModelForPreTraining

 configuration class: XLNetLMHeadModel (XLNet model)
XmodConfig configuration class: XmodForMaskedLM (X-MOD model)
    Instantiates one of the model classes of the library (with a pretraining head) from a configuration. Note:
Loading a model from its configuration file does not load the model weights. It only affects the
model’s configuration. Use from_pretrained() to load the model weights.  Examples:   Copied >>> from transformers import AutoConfig, AutoModelForPreTraining
>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("bert-base-cased")
>>> model = AutoModelForPreTraining.from_config(config)  from_pretrained  < source > ( *model_args **kwargs  )   Parameters   pretrained_model_name_or_path (str or os.PathLike) —
Can be either:
A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like bert-base-uncased, or namespaced under a
user or organizat


AutoModelForPreTraining

ion name, like dbmdz/bert-base-german-cased.
A path to a directory containing model weights saved using
save_pretrained(), e.g., ./my_model_directory/.
A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In
this case, from_tf should be set to True and a configuration object should be provided as
config argument. This loading path is slower than converting the TensorFlow checkpoint in a
PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
   model_args (additional positional arguments, optional) —
Will be passed along to the underlying model __init__() method.   config (PretrainedConfig, optional) —
Configuration for the model to use instead of an automatically loaded configuration. Configuration can
be automatically loaded when:
The model is a model provided by the library (loaded with the model id string of a pretrained
model).
The model was saved using save_pretrained() and is reloaded by supplying the
sa


AutoModelForPreTraining

ve directory.
The model is loaded by supplying a local directory as pretrained_model_name_or_path and a
configuration JSON file named config.json is found in the directory.
   state_dict (Dict[str, torch.Tensor], optional) —
A state dictionary to use instead of a state dictionary loaded from saved weights file.
This option can be used if you want to create a model from a pretrained configuration but load your own
weights. In this case though, you should check if using save_pretrained() and
from_pretrained() is not a simpler option.   cache_dir (str or os.PathLike, optional) —
Path to a directory in which a downloaded pretrained model configuration should be cached if the
standard cache should not be used.   from_tf (bool, optional, defaults to False) —
Load the model weights from a TensorFlow checkpoint save file (see docstring of
pretrained_model_name_or_path argument).   force_download (bool, optional, defaults to False) —
Whether or not to force the (re-)download of the model wei


AutoModelForPreTraining

ghts and configuration files, overriding the
cached versions if they exist.   resume_download (bool, optional, defaults to False) —
Whether or not to delete incompletely received files. Will attempt to resume the download if such a
file exists.   proxies (Dict[str, str], optional) —
A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.   output_loading_info(bool, optional, defaults to False) —
Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.   local_files_only(bool, optional, defaults to False) —
Whether or not to only look at local files (e.g., not try downloading the model).   revision (str, optional, defaults to "main") —
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so revision can be


AutoModelForPreTraining

 any
identifier allowed by git.   trust_remote_code (bool, optional, defaults to False) —
Whether or not to allow for custom models defined on the Hub in their own modeling files. This option
should only be set to True for repositories you trust and in which you have read the code, as it will
execute code present on the Hub on your local machine.   code_revision (str, optional, defaults to "main") —
The specific revision to use for the code on the Hub, if the code leaves in a different repository than
the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based
system for storing models and other artifacts on huggingface.co, so revision can be any identifier
allowed by git.   kwargs (additional keyword arguments, optional) —
Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
output_attentions=True). Behaves differently depending on whether a config is provided or
automatically loaded:
If a conf


AutoModelForPreTraining

iguration is provided with config, **kwargs will be directly passed to the
underlying model’s __init__ method (we assume all relevant updates to the configuration have
already been done)
If a configuration is not provided, kwargs will be first passed to the configuration class
initialization function (from_pretrained()). Each key of kwargs that
corresponds to a configuration attribute will be used to override said attribute with the
supplied kwargs value. Remaining keys that do not correspond to any configuration attribute
will be passed to the underlying model’s __init__ function.
    Instantiate one of the model classes of the library (with a pretraining head) from a pretrained model. The model class to instantiate is selected based on the model_type property of the config object (either
passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by
falling back to using pattern matching on pretrained_model_name_or_path: albert — AlbertFo


AutoModelForPreTraining

rPreTraining (ALBERT model) bart — BartForConditionalGeneration (BART model) bert — BertForPreTraining (BERT model) big_bird — BigBirdForPreTraining (BigBird model) bloom — BloomForCausalLM (BLOOM model) camembert — CamembertForMaskedLM (CamemBERT model) ctrl — CTRLLMHeadModel (CTRL model) data2vec-text — Data2VecTextForMaskedLM (Data2VecText model) deberta — DebertaForMaskedLM (DeBERTa model) deberta-v2 — DebertaV2ForMaskedLM (DeBERTa-v2 model) distilbert — DistilBertForMaskedLM (DistilBERT model) electra — ElectraForPreTraining (ELECTRA model) ernie — ErnieForPreTraining (ERNIE model) flaubert — FlaubertWithLMHeadModel (FlauBERT model) flava — FlavaForPreTraining (FLAVA model) fnet — FNetForPreTraining (FNet model) fsmt — FSMTForConditionalGeneration (FairSeq Machine-Translation model) funnel — FunnelForPreTraining (Funnel Transformer model) gpt-sw3 — GPT2LMHeadModel (GPT-Sw3 model) gpt2 — GPT2LMHeadModel (OpenAI GPT-2 model) gpt_bigcode — GPTBigCodeForCausalLM (GPTBigCode model)


AutoModelForPreTraining

gptsan-japanese — GPTSanJapaneseForConditionalGeneration (GPTSAN-japanese model) ibert — IBertForMaskedLM (I-BERT model) idefics — IdeficsForVisionText2Text (IDEFICS model) layoutlm — LayoutLMForMaskedLM (LayoutLM model) longformer — LongformerForMaskedLM (Longformer model) luke — LukeForMaskedLM (LUKE model) lxmert — LxmertForPreTraining (LXMERT model) mega — MegaForMaskedLM (MEGA model) megatron-bert — MegatronBertForPreTraining (Megatron-BERT model) mobilebert — MobileBertForPreTraining (MobileBERT model) mpnet — MPNetForMaskedLM (MPNet model) mpt — MptForCausalLM (MPT model) mra — MraForMaskedLM (MRA model) mvp — MvpForConditionalGeneration (MVP model) nezha — NezhaForPreTraining (Nezha model) nllb-moe — NllbMoeForConditionalGeneration (NLLB-MOE model) openai-gpt — OpenAIGPTLMHeadModel (OpenAI GPT model) retribert — RetriBertModel (RetriBERT model) roberta — RobertaForMaskedLM (RoBERTa model) roberta-prelayernorm — RobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model) roc_


AutoModelForPreTraining

bert — RoCBertForPreTraining (RoCBert model) rwkv — RwkvForCausalLM (RWKV model) splinter — SplinterForPreTraining (Splinter model) squeezebert — SqueezeBertForMaskedLM (SqueezeBERT model) switch_transformers — SwitchTransformersForConditionalGeneration (SwitchTransformers model) t5 — T5ForConditionalGeneration (T5 model) tapas — TapasForMaskedLM (TAPAS model) transfo-xl — TransfoXLLMHeadModel (Transformer-XL model) tvlt — TvltForPreTraining (TVLT model) unispeech — UniSpeechForPreTraining (UniSpeech model) unispeech-sat — UniSpeechSatForPreTraining (UniSpeechSat model) videomae — VideoMAEForPreTraining (VideoMAE model) visual_bert — VisualBertForPreTraining (VisualBERT model) vit_mae — ViTMAEForPreTraining (ViTMAE model) wav2vec2 — Wav2Vec2ForPreTraining (Wav2Vec2 model) wav2vec2-conformer — Wav2Vec2ConformerForPreTraining (Wav2Vec2-Conformer model) xlm — XLMWithLMHeadModel (XLM model) xlm-roberta — XLMRobertaForMaskedLM (XLM-RoBERTa model) xlm-roberta-xl — XLMRobertaXLForMaskedLM


AutoModelForPreTraining

(XLM-RoBERTa-XL model) xlnet — XLNetLMHeadModel (XLNet model) xmod — XmodForMaskedLM (X-MOD model) The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are
deactivated). To train the model, you should first set it back in training mode with model.train()  Examples:   Copied >>> from transformers import AutoConfig, AutoModelForPreTraining
>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForPreTraining.from_pretrained("bert-base-cased")
>>> # Update configuration during loading
>>> model = AutoModelForPreTraining.from_pretrained("bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True
>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForPreTraining.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
...


TFAutoModelForPreTraining

class transformers.TFAutoModelForPreTraining  < source > ( *args **kwargs  )    This is a generic model class that will be instantiated as one of the model classes of the library (with a pretraining head) when created
with the from_pretrained() class method or the from_config() class
method. This class cannot be instantiated directly using __init__() (throws an error).  from_config  < source > ( **kwargs  )   Parameters   config (PretrainedConfig) —
The model class to instantiate is selected based on the configuration class:
AlbertConfig configuration class: TFAlbertForPreTraining (ALBERT model)
BartConfig configuration class: TFBartForConditionalGeneration (BART model)
BertConfig configuration class: TFBertForPreTraining (BERT model)
CTRLConfig configuration class: TFCTRLLMHeadModel (CTRL model)
CamembertConfig configuration class: TFCamembertForMaskedLM (CamemBERT model)
DistilBertConfig configuration class: TFDistilBertForMaskedLM (DistilBERT model)
ElectraCon


TFAutoModelForPreTraining

fig configuration class: TFElectraForPreTraining (ELECTRA model)
FlaubertConfig configuration class: TFFlaubertWithLMHeadModel (FlauBERT model)
FunnelConfig configuration class: TFFunnelForPreTraining (Funnel Transformer model)
GPT2Config configuration class: TFGPT2LMHeadModel (OpenAI GPT-2 model)
LayoutLMConfig configuration class: TFLayoutLMForMaskedLM (LayoutLM model)
LxmertConfig configuration class: TFLxmertForPreTraining (LXMERT model)
MPNetConfig configuration class: TFMPNetForMaskedLM (MPNet model)
MobileBertConfig configuration class: TFMobileBertForPreTraining (MobileBERT model)
OpenAIGPTConfig configuration class: TFOpenAIGPTLMHeadModel (OpenAI GPT model)
RobertaConfig configuration class: TFRobertaForMaskedLM (RoBERTa model)
RobertaPreLayerNormConfig configuration class: TFRobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model)
T5Config configuration class: TFT5ForConditionalGeneration (T5 model)
TapasConfig configuration class: TFTapasForMaskedLM


TFAutoModelForPreTraining

 (TAPAS model)
TransfoXLConfig configuration class: TFTransfoXLLMHeadModel (Transformer-XL model)
ViTMAEConfig configuration class: TFViTMAEForPreTraining (ViTMAE model)
XLMConfig configuration class: TFXLMWithLMHeadModel (XLM model)
XLMRobertaConfig configuration class: TFXLMRobertaForMaskedLM (XLM-RoBERTa model)
XLNetConfig configuration class: TFXLNetLMHeadModel (XLNet model)
    Instantiates one of the model classes of the library (with a pretraining head) from a configuration. Note:
Loading a model from its configuration file does not load the model weights. It only affects the
model’s configuration. Use from_pretrained() to load the model weights.  Examples:   Copied >>> from transformers import AutoConfig, TFAutoModelForPreTraining
>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("bert-base-cased")
>>> model = TFAutoModelForPreTraining.from_config(config)  from_pretrained  < source > ( *model_args **kwargs


TFAutoModelForPreTraining

  )   Parameters   pretrained_model_name_or_path (str or os.PathLike) —
Can be either:
A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like bert-base-uncased, or namespaced under a
user or organization name, like dbmdz/bert-base-german-cased.
A path to a directory containing model weights saved using
save_pretrained(), e.g., ./my_model_directory/.
A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this
case, from_pt should be set to True and a configuration object should be provided as config
argument. This loading path is slower than converting the PyTorch model in a TensorFlow model
using the provided conversion scripts and loading the TensorFlow model afterwards.
   model_args (additional positional arguments, optional) —
Will be passed along to the underlying model __init__() method.   config (PretrainedConfig, optional) —
Configur


TFAutoModelForPreTraining

ation for the model to use instead of an automatically loaded configuration. Configuration can
be automatically loaded when:
The model is a model provided by the library (loaded with the model id string of a pretrained
model).
The model was saved using save_pretrained() and is reloaded by supplying the
save directory.
The model is loaded by supplying a local directory as pretrained_model_name_or_path and a
configuration JSON file named config.json is found in the directory.
   cache_dir (str or os.PathLike, optional) —
Path to a directory in which a downloaded pretrained model configuration should be cached if the
standard cache should not be used.   from_pt (bool, optional, defaults to False) —
Load the model weights from a PyTorch checkpoint save file (see docstring of
pretrained_model_name_or_path argument).   force_download (bool, optional, defaults to False) —
Whether or not to force the (re-)download of the model weights and configuration files, overriding


TFAutoModelForPreTraining

the
cached versions if they exist.   resume_download (bool, optional, defaults to False) —
Whether or not to delete incompletely received files. Will attempt to resume the download if such a
file exists.   proxies (Dict[str, str], optional) —
A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.   output_loading_info(bool, optional, defaults to False) —
Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.   local_files_only(bool, optional, defaults to False) —
Whether or not to only look at local files (e.g., not try downloading the model).   revision (str, optional, defaults to "main") —
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so revision can be any
identifier allow


TFAutoModelForPreTraining

ed by git.   trust_remote_code (bool, optional, defaults to False) —
Whether or not to allow for custom models defined on the Hub in their own modeling files. This option
should only be set to True for repositories you trust and in which you have read the code, as it will
execute code present on the Hub on your local machine.   code_revision (str, optional, defaults to "main") —
The specific revision to use for the code on the Hub, if the code leaves in a different repository than
the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based
system for storing models and other artifacts on huggingface.co, so revision can be any identifier
allowed by git.   kwargs (additional keyword arguments, optional) —
Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
output_attentions=True). Behaves differently depending on whether a config is provided or
automatically loaded:
If a confi


TFAutoModelForPreTraining

guration is provided with config, **kwargs will be directly passed to the
underlying model’s __init__ method (we assume all relevant updates to the configuration have
already been done)
If a configuration is not provided, kwargs will be first passed to the configuration class
initialization function (from_pretrained()). Each key of kwargs that
corresponds to a configuration attribute will be used to override said attribute with the
supplied kwargs value. Remaining keys that do not correspond to any configuration attribute
will be passed to the underlying model’s __init__ function.
    Instantiate one of the model classes of the library (with a pretraining head) from a pretrained model. The model class to instantiate is selected based on the model_type property of the config object (either
passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by
falling back to using pattern matching on pretrained_model_name_or_path


TFAutoModelForPreTraining

: albert — TFAlbertForPreTraining (ALBERT model) bart — TFBartForConditionalGeneration (BART model) bert — TFBertForPreTraining (BERT model) camembert — TFCamembertForMaskedLM (CamemBERT model) ctrl — TFCTRLLMHeadModel (CTRL model) distilbert — TFDistilBertForMaskedLM (DistilBERT model) electra — TFElectraForPreTraining (ELECTRA model) flaubert — TFFlaubertWithLMHeadModel (FlauBERT model) funnel — TFFunnelForPreTraining (Funnel Transformer model) gpt-sw3 — TFGPT2LMHeadModel (GPT-Sw3 model) gpt2 — TFGPT2LMHeadModel (OpenAI GPT-2 model) layoutlm — TFLayoutLMForMaskedLM (LayoutLM model) lxmert — TFLxmertForPreTraining (LXMERT model) mobilebert — TFMobileBertForPreTraining (MobileBERT model) mpnet — TFMPNetForMaskedLM (MPNet model) openai-gpt — TFOpenAIGPTLMHeadModel (OpenAI GPT model) roberta — TFRobertaForMaskedLM (RoBERTa model) roberta-prelayernorm — TFRobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model) t5 — TFT5ForConditionalGeneration (T5 model) tapas —


TFAutoModelForPreTraining

 TFTapasForMaskedLM (TAPAS model) transfo-xl — TFTransfoXLLMHeadModel (Transformer-XL model) vit_mae — TFViTMAEForPreTraining (ViTMAE model) xlm — TFXLMWithLMHeadModel (XLM model) xlm-roberta — TFXLMRobertaForMaskedLM (XLM-RoBERTa model) xlnet — TFXLNetLMHeadModel (XLNet model)  Examples:   Copied >>> from transformers import AutoConfig, TFAutoModelForPreTraining
>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForPreTraining.from_pretrained("bert-base-cased")
>>> # Update configuration during loading
>>> model = TFAutoModelForPreTraining.from_pretrained("bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True
>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = TFAutoModelForPreTraining.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=conf


FlaxAutoModelForPreTraining

class transformers.FlaxAutoModelForPreTraining  < source > ( *args **kwargs  )    This is a generic model class that will be instantiated as one of the model classes of the library (with a pretraining head) when created
with the from_pretrained() class method or the from_config() class
method. This class cannot be instantiated directly using __init__() (throws an error).  from_config  < source > ( **kwargs  )   Parameters   config (PretrainedConfig) —
The model class to instantiate is selected based on the configuration class:
AlbertConfig configuration class: FlaxAlbertForPreTraining (ALBERT model)
BartConfig configuration class: FlaxBartForConditionalGeneration (BART model)
BertConfig configuration class: FlaxBertForPreTraining (BERT model)
BigBirdConfig configuration class: FlaxBigBirdForPreTraining (BigBird model)
ElectraConfig configuration class: FlaxElectraForPreTraining (ELECTRA model)
LongT5Config configuration class: FlaxLongT5ForConditionalGenerati


FlaxAutoModelForPreTraining

on (LongT5 model)
MBartConfig configuration class: FlaxMBartForConditionalGeneration (mBART model)
MT5Config configuration class: FlaxMT5ForConditionalGeneration (MT5 model)
RoFormerConfig configuration class: FlaxRoFormerForMaskedLM (RoFormer model)
RobertaConfig configuration class: FlaxRobertaForMaskedLM (RoBERTa model)
RobertaPreLayerNormConfig configuration class: FlaxRobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model)
T5Config configuration class: FlaxT5ForConditionalGeneration (T5 model)
Wav2Vec2Config configuration class: FlaxWav2Vec2ForPreTraining (Wav2Vec2 model)
WhisperConfig configuration class: FlaxWhisperForConditionalGeneration (Whisper model)
XLMRobertaConfig configuration class: FlaxXLMRobertaForMaskedLM (XLM-RoBERTa model)
    Instantiates one of the model classes of the library (with a pretraining head) from a configuration. Note:
Loading a model from its configuration file does not load the model weights. It only affects the
model’


FlaxAutoModelForPreTraining

s configuration. Use from_pretrained() to load the model weights.  Examples:   Copied >>> from transformers import AutoConfig, FlaxAutoModelForPreTraining
>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("bert-base-cased")
>>> model = FlaxAutoModelForPreTraining.from_config(config)  from_pretrained  < source > ( *model_args **kwargs  )   Parameters   pretrained_model_name_or_path (str or os.PathLike) —
Can be either:
A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like bert-base-uncased, or namespaced under a
user or organization name, like dbmdz/bert-base-german-cased.
A path to a directory containing model weights saved using
save_pretrained(), e.g., ./my_model_directory/.
A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this
case, from_pt should be set to True and a configurat


FlaxAutoModelForPreTraining

ion object should be provided as config
argument. This loading path is slower than converting the PyTorch model in a TensorFlow model
using the provided conversion scripts and loading the TensorFlow model afterwards.
   model_args (additional positional arguments, optional) —
Will be passed along to the underlying model __init__() method.   config (PretrainedConfig, optional) —
Configuration for the model to use instead of an automatically loaded configuration. Configuration can
be automatically loaded when:
The model is a model provided by the library (loaded with the model id string of a pretrained
model).
The model was saved using save_pretrained() and is reloaded by supplying the
save directory.
The model is loaded by supplying a local directory as pretrained_model_name_or_path and a
configuration JSON file named config.json is found in the directory.
   cache_dir (str or os.PathLike, optional) —
Path to a directory in which a downloaded pretrained model


FlaxAutoModelForPreTraining

configuration should be cached if the
standard cache should not be used.   from_pt (bool, optional, defaults to False) —
Load the model weights from a PyTorch checkpoint save file (see docstring of
pretrained_model_name_or_path argument).   force_download (bool, optional, defaults to False) —
Whether or not to force the (re-)download of the model weights and configuration files, overriding the
cached versions if they exist.   resume_download (bool, optional, defaults to False) —
Whether or not to delete incompletely received files. Will attempt to resume the download if such a
file exists.   proxies (Dict[str, str], optional) —
A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.   output_loading_info(bool, optional, defaults to False) —
Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.   local


FlaxAutoModelForPreTraining

_files_only(bool, optional, defaults to False) —
Whether or not to only look at local files (e.g., not try downloading the model).   revision (str, optional, defaults to "main") —
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so revision can be any
identifier allowed by git.   trust_remote_code (bool, optional, defaults to False) —
Whether or not to allow for custom models defined on the Hub in their own modeling files. This option
should only be set to True for repositories you trust and in which you have read the code, as it will
execute code present on the Hub on your local machine.   code_revision (str, optional, defaults to "main") —
The specific revision to use for the code on the Hub, if the code leaves in a different repository than
the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a


FlaxAutoModelForPreTraining

 git-based
system for storing models and other artifacts on huggingface.co, so revision can be any identifier
allowed by git.   kwargs (additional keyword arguments, optional) —
Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
output_attentions=True). Behaves differently depending on whether a config is provided or
automatically loaded:
If a configuration is provided with config, **kwargs will be directly passed to the
underlying model’s __init__ method (we assume all relevant updates to the configuration have
already been done)
If a configuration is not provided, kwargs will be first passed to the configuration class
initialization function (from_pretrained()). Each key of kwargs that
corresponds to a configuration attribute will be used to override said attribute with the
supplied kwargs value. Remaining keys that do not correspond to any configuration attribute
will be passed to the underlying model’s __i


FlaxAutoModelForPreTraining

nit__ function.
    Instantiate one of the model classes of the library (with a pretraining head) from a pretrained model. The model class to instantiate is selected based on the model_type property of the config object (either
passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by
falling back to using pattern matching on pretrained_model_name_or_path: albert — FlaxAlbertForPreTraining (ALBERT model) bart — FlaxBartForConditionalGeneration (BART model) bert — FlaxBertForPreTraining (BERT model) big_bird — FlaxBigBirdForPreTraining (BigBird model) electra — FlaxElectraForPreTraining (ELECTRA model) longt5 — FlaxLongT5ForConditionalGeneration (LongT5 model) mbart — FlaxMBartForConditionalGeneration (mBART model) mt5 — FlaxMT5ForConditionalGeneration (MT5 model) roberta — FlaxRobertaForMaskedLM (RoBERTa model) roberta-prelayernorm — FlaxRobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model) roformer — Flax


FlaxAutoModelForPreTraining

RoFormerForMaskedLM (RoFormer model) t5 — FlaxT5ForConditionalGeneration (T5 model) wav2vec2 — FlaxWav2Vec2ForPreTraining (Wav2Vec2 model) whisper — FlaxWhisperForConditionalGeneration (Whisper model) xlm-roberta — FlaxXLMRobertaForMaskedLM (XLM-RoBERTa model)  Examples:   Copied >>> from transformers import AutoConfig, FlaxAutoModelForPreTraining
>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxAutoModelForPreTraining.from_pretrained("bert-base-cased")
>>> # Update configuration during loading
>>> model = FlaxAutoModelForPreTraining.from_pretrained("bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True
>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = FlaxAutoModelForPreTraining.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
...


AutoModelForCausalLM

class transformers.AutoModelForCausalLM  < source > ( *args **kwargs  )    This is a generic model class that will be instantiated as one of the model classes of the library (with a causal language modeling head) when created
with the from_pretrained() class method or the from_config() class
method. This class cannot be instantiated directly using __init__() (throws an error).  from_config  < source > ( **kwargs  )   Parameters   config (PretrainedConfig) —
The model class to instantiate is selected based on the configuration class:
BartConfig configuration class: BartForCausalLM (BART model)
BertConfig configuration class: BertLMHeadModel (BERT model)
BertGenerationConfig configuration class: BertGenerationDecoder (Bert Generation model)
BigBirdConfig configuration class: BigBirdForCausalLM (BigBird model)
BigBirdPegasusConfig configuration class: BigBirdPegasusForCausalLM (BigBird-Pegasus model)
BioGptConfig configuration class: BioGptFor


AutoModelForCausalLM

CausalLM (BioGpt model)
BlenderbotConfig configuration class: BlenderbotForCausalLM (Blenderbot model)
BlenderbotSmallConfig configuration class: BlenderbotSmallForCausalLM (BlenderbotSmall model)
BloomConfig configuration class: BloomForCausalLM (BLOOM model)
CTRLConfig configuration class: CTRLLMHeadModel (CTRL model)
CamembertConfig configuration class: CamembertForCausalLM (CamemBERT model)
CodeGenConfig configuration class: CodeGenForCausalLM (CodeGen model)
CpmAntConfig configuration class: CpmAntForCausalLM (CPM-Ant model)
Data2VecTextConfig configuration class: Data2VecTextForCausalLM (Data2VecText model)
ElectraConfig configuration class: ElectraForCausalLM (ELECTRA model)
ErnieConfig configuration class: ErnieForCausalLM (ERNIE model)
FalconConfig configuration class: FalconForCausalLM (Falcon model)
GPT2Config configuration class: GPT2LMHeadModel (OpenAI GPT-2 model)
GPTBigCodeConfig configuration class: GPTBigCodeForCausalLM (GP


AutoModelForCausalLM

TBigCode model)
GPTJConfig configuration class: GPTJForCausalLM (GPT-J model)
GPTNeoConfig configuration class: GPTNeoForCausalLM (GPT Neo model)
GPTNeoXConfig configuration class: GPTNeoXForCausalLM (GPT NeoX model)
GPTNeoXJapaneseConfig configuration class: GPTNeoXJapaneseForCausalLM (GPT NeoX Japanese model)
GitConfig configuration class: GitForCausalLM (GIT model)
LlamaConfig configuration class: LlamaForCausalLM (LLaMA model)
MBartConfig configuration class: MBartForCausalLM (mBART model)
MarianConfig configuration class: MarianForCausalLM (Marian model)
MegaConfig configuration class: MegaForCausalLM (MEGA model)
MegatronBertConfig configuration class: MegatronBertForCausalLM (Megatron-BERT model)
MistralConfig configuration class: MistralForCausalLM (Mistral model)
MptConfig configuration class: MptForCausalLM (MPT model)
MusicgenConfig configuration class: MusicgenForCausalLM (MusicGen model)
MvpConfig configuration class: MvpForCau


AutoModelForCausalLM

salLM (MVP model)
OPTConfig configuration class: OPTForCausalLM (OPT model)
OpenAIGPTConfig configuration class: OpenAIGPTLMHeadModel (OpenAI GPT model)
OpenLlamaConfig configuration class: OpenLlamaForCausalLM (OpenLlama model)
PLBartConfig configuration class: PLBartForCausalLM (PLBart model)
PegasusConfig configuration class: PegasusForCausalLM (Pegasus model)
PersimmonConfig configuration class: PersimmonForCausalLM (Persimmon model)
ProphetNetConfig configuration class: ProphetNetForCausalLM (ProphetNet model)
QDQBertConfig configuration class: QDQBertLMHeadModel (QDQBert model)
ReformerConfig configuration class: ReformerModelWithLMHead (Reformer model)
RemBertConfig configuration class: RemBertForCausalLM (RemBERT model)
RoCBertConfig configuration class: RoCBertForCausalLM (RoCBert model)
RoFormerConfig configuration class: RoFormerForCausalLM (RoFormer model)
RobertaConfig configuration class: RobertaForCausalLM (RoBERTa model)
Rob


AutoModelForCausalLM

ertaPreLayerNormConfig configuration class: RobertaPreLayerNormForCausalLM (RoBERTa-PreLayerNorm model)
RwkvConfig configuration class: RwkvForCausalLM (RWKV model)
Speech2Text2Config configuration class: Speech2Text2ForCausalLM (Speech2Text2 model)
TrOCRConfig configuration class: TrOCRForCausalLM (TrOCR model)
TransfoXLConfig configuration class: TransfoXLLMHeadModel (Transformer-XL model)
XGLMConfig configuration class: XGLMForCausalLM (XGLM model)
XLMConfig configuration class: XLMWithLMHeadModel (XLM model)
XLMProphetNetConfig configuration class: XLMProphetNetForCausalLM (XLM-ProphetNet model)
XLMRobertaConfig configuration class: XLMRobertaForCausalLM (XLM-RoBERTa model)
XLMRobertaXLConfig configuration class: XLMRobertaXLForCausalLM (XLM-RoBERTa-XL model)
XLNetConfig configuration class: XLNetLMHeadModel (XLNet model)
XmodConfig configuration class: XmodForCausalLM (X-MOD model)
    Instantiates one of the model classes of the libra


AutoModelForCausalLM

ry (with a causal language modeling head) from a configuration. Note:
Loading a model from its configuration file does not load the model weights. It only affects the
model’s configuration. Use from_pretrained() to load the model weights.  Examples:   Copied >>> from transformers import AutoConfig, AutoModelForCausalLM
>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("bert-base-cased")
>>> model = AutoModelForCausalLM.from_config(config)  from_pretrained  < source > ( *model_args **kwargs  )   Parameters   pretrained_model_name_or_path (str or os.PathLike) —
Can be either:
A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like bert-base-uncased, or namespaced under a
user or organization name, like dbmdz/bert-base-german-cased.
A path to a directory containing model weights saved using
save_pretrained


AutoModelForCausalLM

(), e.g., ./my_model_directory/.
A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In
this case, from_tf should be set to True and a configuration object should be provided as
config argument. This loading path is slower than converting the TensorFlow checkpoint in a
PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
   model_args (additional positional arguments, optional) —
Will be passed along to the underlying model __init__() method.   config (PretrainedConfig, optional) —
Configuration for the model to use instead of an automatically loaded configuration. Configuration can
be automatically loaded when:
The model is a model provided by the library (loaded with the model id string of a pretrained
model).
The model was saved using save_pretrained() and is reloaded by supplying the
save directory.
The model is loaded by supplying a local directory as pretrain


AutoModelForCausalLM

ed_model_name_or_path and a
configuration JSON file named config.json is found in the directory.
   state_dict (Dict[str, torch.Tensor], optional) —
A state dictionary to use instead of a state dictionary loaded from saved weights file.
This option can be used if you want to create a model from a pretrained configuration but load your own
weights. In this case though, you should check if using save_pretrained() and
from_pretrained() is not a simpler option.   cache_dir (str or os.PathLike, optional) —
Path to a directory in which a downloaded pretrained model configuration should be cached if the
standard cache should not be used.   from_tf (bool, optional, defaults to False) —
Load the model weights from a TensorFlow checkpoint save file (see docstring of
pretrained_model_name_or_path argument).   force_download (bool, optional, defaults to False) —
Whether or not to force the (re-)download of the model weights and configuration files, ove


AutoModelForCausalLM

rriding the
cached versions if they exist.   resume_download (bool, optional, defaults to False) —
Whether or not to delete incompletely received files. Will attempt to resume the download if such a
file exists.   proxies (Dict[str, str], optional) —
A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.   output_loading_info(bool, optional, defaults to False) —
Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.   local_files_only(bool, optional, defaults to False) —
Whether or not to only look at local files (e.g., not try downloading the model).   revision (str, optional, defaults to "main") —
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so revis


AutoModelForCausalLM

ion can be any
identifier allowed by git.   trust_remote_code (bool, optional, defaults to False) —
Whether or not to allow for custom models defined on the Hub in their own modeling files. This option
should only be set to True for repositories you trust and in which you have read the code, as it will
execute code present on the Hub on your local machine.   code_revision (str, optional, defaults to "main") —
The specific revision to use for the code on the Hub, if the code leaves in a different repository than
the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based
system for storing models and other artifacts on huggingface.co, so revision can be any identifier
allowed by git.   kwargs (additional keyword arguments, optional) —
Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
output_attentions=True). Behaves differently depending on whether a


AutoModelForCausalLM

config is provided or
automatically loaded:
If a configuration is provided with config, **kwargs will be directly passed to the
underlying model’s __init__ method (we assume all relevant updates to the configuration have
already been done)
If a configuration is not provided, kwargs will be first passed to the configuration class
initialization function (from_pretrained()). Each key of kwargs that
corresponds to a configuration attribute will be used to override said attribute with the
supplied kwargs value. Remaining keys that do not correspond to any configuration attribute
will be passed to the underlying model’s __init__ function.
    Instantiate one of the model classes of the library (with a causal language modeling head) from a pretrained model. The model class to instantiate is selected based on the model_type property of the config object (either
passed as an argument or loaded from pretrained_model_name_or_path if possible), or whe


AutoModelForCausalLM

n it’s missing, by
falling back to using pattern matching on pretrained_model_name_or_path: bart — BartForCausalLM (BART model) bert — BertLMHeadModel (BERT model) bert-generation — BertGenerationDecoder (Bert Generation model) big_bird — BigBirdForCausalLM (BigBird model) bigbird_pegasus — BigBirdPegasusForCausalLM (BigBird-Pegasus model) biogpt — BioGptForCausalLM (BioGpt model) blenderbot — BlenderbotForCausalLM (Blenderbot model) blenderbot-small — BlenderbotSmallForCausalLM (BlenderbotSmall model) bloom — BloomForCausalLM (BLOOM model) camembert — CamembertForCausalLM (CamemBERT model) code_llama — LlamaForCausalLM (CodeLlama model) codegen — CodeGenForCausalLM (CodeGen model) cpmant — CpmAntForCausalLM (CPM-Ant model) ctrl — CTRLLMHeadModel (CTRL model) data2vec-text — Data2VecTextForCausalLM (Data2VecText model) electra — ElectraForCausalLM (ELECTRA model) ernie — ErnieForCausalLM (ERNIE model) falcon — FalconForCausalLM (Falcon mode


AutoModelForCausalLM

l) git — GitForCausalLM (GIT model) gpt-sw3 — GPT2LMHeadModel (GPT-Sw3 model) gpt2 — GPT2LMHeadModel (OpenAI GPT-2 model) gpt_bigcode — GPTBigCodeForCausalLM (GPTBigCode model) gpt_neo — GPTNeoForCausalLM (GPT Neo model) gpt_neox — GPTNeoXForCausalLM (GPT NeoX model) gpt_neox_japanese — GPTNeoXJapaneseForCausalLM (GPT NeoX Japanese model) gptj — GPTJForCausalLM (GPT-J model) llama — LlamaForCausalLM (LLaMA model) marian — MarianForCausalLM (Marian model) mbart — MBartForCausalLM (mBART model) mega — MegaForCausalLM (MEGA model) megatron-bert — MegatronBertForCausalLM (Megatron-BERT model) mistral — MistralForCausalLM (Mistral model) mpt — MptForCausalLM (MPT model) musicgen — MusicgenForCausalLM (MusicGen model) mvp — MvpForCausalLM (MVP model) open-llama — OpenLlamaForCausalLM (OpenLlama model) openai-gpt — OpenAIGPTLMHeadModel (OpenAI GPT model) opt — OPTForCausalLM (OPT model) pegasus — PegasusForCausalLM (Pegasus model) persimmon — Pers


AutoModelForCausalLM

immonForCausalLM (Persimmon model) plbart — PLBartForCausalLM (PLBart model) prophetnet — ProphetNetForCausalLM (ProphetNet model) qdqbert — QDQBertLMHeadModel (QDQBert model) reformer — ReformerModelWithLMHead (Reformer model) rembert — RemBertForCausalLM (RemBERT model) roberta — RobertaForCausalLM (RoBERTa model) roberta-prelayernorm — RobertaPreLayerNormForCausalLM (RoBERTa-PreLayerNorm model) roc_bert — RoCBertForCausalLM (RoCBert model) roformer — RoFormerForCausalLM (RoFormer model) rwkv — RwkvForCausalLM (RWKV model) speech_to_text_2 — Speech2Text2ForCausalLM (Speech2Text2 model) transfo-xl — TransfoXLLMHeadModel (Transformer-XL model) trocr — TrOCRForCausalLM (TrOCR model) xglm — XGLMForCausalLM (XGLM model) xlm — XLMWithLMHeadModel (XLM model) xlm-prophetnet — XLMProphetNetForCausalLM (XLM-ProphetNet model) xlm-roberta — XLMRobertaForCausalLM (XLM-RoBERTa model) xlm-roberta-xl — XLMRobertaXLForCausalLM (XLM-RoBERTa-XL model) xlnet


AutoModelForCausalLM

 — XLNetLMHeadModel (XLNet model) xmod — XmodForCausalLM (X-MOD model) The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are
deactivated). To train the model, you should first set it back in training mode with model.train()  Examples:   Copied >>> from transformers import AutoConfig, AutoModelForCausalLM
>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForCausalLM.from_pretrained("bert-base-cased")
>>> # Update configuration during loading
>>> model = AutoModelForCausalLM.from_pretrained("bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True
>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForCausalLM.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config


TFAutoModelForCausalLM

class transformers.TFAutoModelForCausalLM  < source > ( *args **kwargs  )    This is a generic model class that will be instantiated as one of the model classes of the library (with a causal language modeling head) when created
with the from_pretrained() class method or the from_config() class
method. This class cannot be instantiated directly using __init__() (throws an error).  from_config  < source > ( **kwargs  )   Parameters   config (PretrainedConfig) —
The model class to instantiate is selected based on the configuration class:
BertConfig configuration class: TFBertLMHeadModel (BERT model)
CTRLConfig configuration class: TFCTRLLMHeadModel (CTRL model)
CamembertConfig configuration class: TFCamembertForCausalLM (CamemBERT model)
GPT2Config configuration class: TFGPT2LMHeadModel (OpenAI GPT-2 model)
GPTJConfig configuration class: TFGPTJForCausalLM (GPT-J model)
OPTConfig configuration class: TFOPTForCausalLM (OPT model)
OpenAIGPTConfig configuratio


TFAutoModelForCausalLM

n class: TFOpenAIGPTLMHeadModel (OpenAI GPT model)
RemBertConfig configuration class: TFRemBertForCausalLM (RemBERT model)
RoFormerConfig configuration class: TFRoFormerForCausalLM (RoFormer model)
RobertaConfig configuration class: TFRobertaForCausalLM (RoBERTa model)
RobertaPreLayerNormConfig configuration class: TFRobertaPreLayerNormForCausalLM (RoBERTa-PreLayerNorm model)
TransfoXLConfig configuration class: TFTransfoXLLMHeadModel (Transformer-XL model)
XGLMConfig configuration class: TFXGLMForCausalLM (XGLM model)
XLMConfig configuration class: TFXLMWithLMHeadModel (XLM model)
XLMRobertaConfig configuration class: TFXLMRobertaForCausalLM (XLM-RoBERTa model)
XLNetConfig configuration class: TFXLNetLMHeadModel (XLNet model)
    Instantiates one of the model classes of the library (with a causal language modeling head) from a configuration. Note:
Loading a model from its configuration file does not load the model weights. It only affects the
model’s co


TFAutoModelForCausalLM

nfiguration. Use from_pretrained() to load the model weights.  Examples:   Copied >>> from transformers import AutoConfig, TFAutoModelForCausalLM
>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("bert-base-cased")
>>> model = TFAutoModelForCausalLM.from_config(config)  from_pretrained  < source > ( *model_args **kwargs  )   Parameters   pretrained_model_name_or_path (str or os.PathLike) —
Can be either:
A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like bert-base-uncased, or namespaced under a
user or organization name, like dbmdz/bert-base-german-cased.
A path to a directory containing model weights saved using
save_pretrained(), e.g., ./my_model_directory/.
A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this
case, from_pt should be set to True and a configuration objec


TFAutoModelForCausalLM

t should be provided as config
argument. This loading path is slower than converting the PyTorch model in a TensorFlow model
using the provided conversion scripts and loading the TensorFlow model afterwards.
   model_args (additional positional arguments, optional) —
Will be passed along to the underlying model __init__() method.   config (PretrainedConfig, optional) —
Configuration for the model to use instead of an automatically loaded configuration. Configuration can
be automatically loaded when:
The model is a model provided by the library (loaded with the model id string of a pretrained
model).
The model was saved using save_pretrained() and is reloaded by supplying the
save directory.
The model is loaded by supplying a local directory as pretrained_model_name_or_path and a
configuration JSON file named config.json is found in the directory.
   cache_dir (str or os.PathLike, optional) —
Path to a directory in which a downloaded pretrained model conf


TFAutoModelForCausalLM

iguration should be cached if the
standard cache should not be used.   from_pt (bool, optional, defaults to False) —
Load the model weights from a PyTorch checkpoint save file (see docstring of
pretrained_model_name_or_path argument).   force_download (bool, optional, defaults to False) —
Whether or not to force the (re-)download of the model weights and configuration files, overriding the
cached versions if they exist.   resume_download (bool, optional, defaults to False) —
Whether or not to delete incompletely received files. Will attempt to resume the download if such a
file exists.   proxies (Dict[str, str], optional) —
A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.   output_loading_info(bool, optional, defaults to False) —
Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.   loca


TFAutoModelForCausalLM

l_files_only(bool, optional, defaults to False) —
Whether or not to only look at local files (e.g., not try downloading the model).   revision (str, optional, defaults to "main") —
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so revision can be any
identifier allowed by git.   trust_remote_code (bool, optional, defaults to False) —
Whether or not to allow for custom models defined on the Hub in their own modeling files. This option
should only be set to True for repositories you trust and in which you have read the code, as it will
execute code present on the Hub on your local machine.   code_revision (str, optional, defaults to "main") —
The specific revision to use for the code on the Hub, if the code leaves in a different repository than
the rest of the model. It can be a branch name, a tag name, or a commit id, since we


TFAutoModelForCausalLM

 use a git-based
system for storing models and other artifacts on huggingface.co, so revision can be any identifier
allowed by git.   kwargs (additional keyword arguments, optional) —
Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
output_attentions=True). Behaves differently depending on whether a config is provided or
automatically loaded:
If a configuration is provided with config, **kwargs will be directly passed to the
underlying model’s __init__ method (we assume all relevant updates to the configuration have
already been done)
If a configuration is not provided, kwargs will be first passed to the configuration class
initialization function (from_pretrained()). Each key of kwargs that
corresponds to a configuration attribute will be used to override said attribute with the
supplied kwargs value. Remaining keys that do not correspond to any configuration attribute
will be passed to the underlying


TFAutoModelForCausalLM

model’s __init__ function.
    Instantiate one of the model classes of the library (with a causal language modeling head) from a pretrained model. The model class to instantiate is selected based on the model_type property of the config object (either
passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by
falling back to using pattern matching on pretrained_model_name_or_path: bert — TFBertLMHeadModel (BERT model) camembert — TFCamembertForCausalLM (CamemBERT model) ctrl — TFCTRLLMHeadModel (CTRL model) gpt-sw3 — TFGPT2LMHeadModel (GPT-Sw3 model) gpt2 — TFGPT2LMHeadModel (OpenAI GPT-2 model) gptj — TFGPTJForCausalLM (GPT-J model) openai-gpt — TFOpenAIGPTLMHeadModel (OpenAI GPT model) opt — TFOPTForCausalLM (OPT model) rembert — TFRemBertForCausalLM (RemBERT model) roberta — TFRobertaForCausalLM (RoBERTa model) roberta-prelayernorm — TFRobertaPreLayerNormForCausalLM (RoBERTa-PreLayerNorm model) roformer —


TFAutoModelForCausalLM

 TFRoFormerForCausalLM (RoFormer model) transfo-xl — TFTransfoXLLMHeadModel (Transformer-XL model) xglm — TFXGLMForCausalLM (XGLM model) xlm — TFXLMWithLMHeadModel (XLM model) xlm-roberta — TFXLMRobertaForCausalLM (XLM-RoBERTa model) xlnet — TFXLNetLMHeadModel (XLNet model)  Examples:   Copied >>> from transformers import AutoConfig, TFAutoModelForCausalLM
>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForCausalLM.from_pretrained("bert-base-cased")
>>> # Update configuration during loading
>>> model = TFAutoModelForCausalLM.from_pretrained("bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True
>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = TFAutoModelForCausalLM.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
...


FlaxAutoModelForCausalLM

class transformers.FlaxAutoModelForCausalLM  < source > ( *args **kwargs  )    This is a generic model class that will be instantiated as one of the model classes of the library (with a causal language modeling head) when created
with the from_pretrained() class method or the from_config() class
method. This class cannot be instantiated directly using __init__() (throws an error).  from_config  < source > ( **kwargs  )   Parameters   config (PretrainedConfig) —
The model class to instantiate is selected based on the configuration class:
BartConfig configuration class: FlaxBartForCausalLM (BART model)
BertConfig configuration class: FlaxBertForCausalLM (BERT model)
BigBirdConfig configuration class: FlaxBigBirdForCausalLM (BigBird model)
BloomConfig configuration class: FlaxBloomForCausalLM (BLOOM model)
ElectraConfig configuration class: FlaxElectraForCausalLM (ELECTRA model)
GPT2Config configuration class: FlaxGPT2LM


FlaxAutoModelForCausalLM

HeadModel (OpenAI GPT-2 model)
GPTJConfig configuration class: FlaxGPTJForCausalLM (GPT-J model)
GPTNeoConfig configuration class: FlaxGPTNeoForCausalLM (GPT Neo model)
OPTConfig configuration class: FlaxOPTForCausalLM (OPT model)
RobertaConfig configuration class: FlaxRobertaForCausalLM (RoBERTa model)
RobertaPreLayerNormConfig configuration class: FlaxRobertaPreLayerNormForCausalLM (RoBERTa-PreLayerNorm model)
XGLMConfig configuration class: FlaxXGLMForCausalLM (XGLM model)
XLMRobertaConfig configuration class: FlaxXLMRobertaForCausalLM (XLM-RoBERTa model)
    Instantiates one of the model classes of the library (with a causal language modeling head) from a configuration. Note:
Loading a model from its configuration file does not load the model weights. It only affects the
model’s configuration. Use from_pretrained() to load the model weights.  Examples:   Copied >>> from transformers import AutoConfig, FlaxAutoMode


FlaxAutoModelForCausalLM

lForCausalLM
>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("bert-base-cased")
>>> model = FlaxAutoModelForCausalLM.from_config(config)  from_pretrained  < source > ( *model_args **kwargs  )   Parameters   pretrained_model_name_or_path (str or os.PathLike) —
Can be either:
A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like bert-base-uncased, or namespaced under a
user or organization name, like dbmdz/bert-base-german-cased.
A path to a directory containing model weights saved using
save_pretrained(), e.g., ./my_model_directory/.
A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this
case, from_pt should be set to True and a configuration object should be provided as config
argument. This loading path is slower than converting the PyTor


FlaxAutoModelForCausalLM

ch model in a TensorFlow model
using the provided conversion scripts and loading the TensorFlow model afterwards.
   model_args (additional positional arguments, optional) —
Will be passed along to the underlying model __init__() method.   config (PretrainedConfig, optional) —
Configuration for the model to use instead of an automatically loaded configuration. Configuration can
be automatically loaded when:
The model is a model provided by the library (loaded with the model id string of a pretrained
model).
The model was saved using save_pretrained() and is reloaded by supplying the
save directory.
The model is loaded by supplying a local directory as pretrained_model_name_or_path and a
configuration JSON file named config.json is found in the directory.
   cache_dir (str or os.PathLike, optional) —
Path to a directory in which a downloaded pretrained model configuration should be cached if the
standard cache should n


FlaxAutoModelForCausalLM

ot be used.   from_pt (bool, optional, defaults to False) —
Load the model weights from a PyTorch checkpoint save file (see docstring of
pretrained_model_name_or_path argument).   force_download (bool, optional, defaults to False) —
Whether or not to force the (re-)download of the model weights and configuration files, overriding the
cached versions if they exist.   resume_download (bool, optional, defaults to False) —
Whether or not to delete incompletely received files. Will attempt to resume the download if such a
file exists.   proxies (Dict[str, str], optional) —
A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.   output_loading_info(bool, optional, defaults to False) —
Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.   local_files_only(bool, o


FlaxAutoModelForCausalLM

ptional, defaults to False) —
Whether or not to only look at local files (e.g., not try downloading the model).   revision (str, optional, defaults to "main") —
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so revision can be any
identifier allowed by git.   trust_remote_code (bool, optional, defaults to False) —
Whether or not to allow for custom models defined on the Hub in their own modeling files. This option
should only be set to True for repositories you trust and in which you have read the code, as it will
execute code present on the Hub on your local machine.   code_revision (str, optional, defaults to "main") —
The specific revision to use for the code on the Hub, if the code leaves in a different repository than
the rest of the model. It can be a branch name, a tag name, or a co


FlaxAutoModelForCausalLM

mmit id, since we use a git-based
system for storing models and other artifacts on huggingface.co, so revision can be any identifier
allowed by git.   kwargs (additional keyword arguments, optional) —
Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
output_attentions=True). Behaves differently depending on whether a config is provided or
automatically loaded:
If a configuration is provided with config, **kwargs will be directly passed to the
underlying model’s __init__ method (we assume all relevant updates to the configuration have
already been done)
If a configuration is not provided, kwargs will be first passed to the configuration class
initialization function (from_pretrained()). Each key of kwargs that
corresponds to a configuration attribute will be used to override said attribute with the
supplied kwargs value. Remaining keys that do not correspond to any con


FlaxAutoModelForCausalLM

figuration attribute
will be passed to the underlying model’s __init__ function.
    Instantiate one of the model classes of the library (with a causal language modeling head) from a pretrained model. The model class to instantiate is selected based on the model_type property of the config object (either
passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by
falling back to using pattern matching on pretrained_model_name_or_path: bart — FlaxBartForCausalLM (BART model) bert — FlaxBertForCausalLM (BERT model) big_bird — FlaxBigBirdForCausalLM (BigBird model) bloom — FlaxBloomForCausalLM (BLOOM model) electra — FlaxElectraForCausalLM (ELECTRA model) gpt-sw3 — FlaxGPT2LMHeadModel (GPT-Sw3 model) gpt2 — FlaxGPT2LMHeadModel (OpenAI GPT-2 model) gpt_neo — FlaxGPTNeoForCausalLM (GPT Neo model) gptj — FlaxGPTJForCausalLM (GPT-J model) opt — FlaxOPTForCausalLM (OPT model) robe


FlaxAutoModelForCausalLM

rta — FlaxRobertaForCausalLM (RoBERTa model) roberta-prelayernorm — FlaxRobertaPreLayerNormForCausalLM (RoBERTa-PreLayerNorm model) xglm — FlaxXGLMForCausalLM (XGLM model) xlm-roberta — FlaxXLMRobertaForCausalLM (XLM-RoBERTa model)  Examples:   Copied >>> from transformers import AutoConfig, FlaxAutoModelForCausalLM
>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxAutoModelForCausalLM.from_pretrained("bert-base-cased")
>>> # Update configuration during loading
>>> model = FlaxAutoModelForCausalLM.from_pretrained("bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True
>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = FlaxAutoModelForCausalLM.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
..


AutoModelForMaskedLM

class transformers.AutoModelForMaskedLM  < source > ( *args **kwargs  )    This is a generic model class that will be instantiated as one of the model classes of the library (with a masked language modeling head) when created
with the from_pretrained() class method or the from_config() class
method. This class cannot be instantiated directly using __init__() (throws an error).  from_config  < source > ( **kwargs  )   Parameters   config (PretrainedConfig) —
The model class to instantiate is selected based on the configuration class:
AlbertConfig configuration class: AlbertForMaskedLM (ALBERT model)
BartConfig configuration class: BartForConditionalGeneration (BART model)
BertConfig configuration class: BertForMaskedLM (BERT model)
BigBirdConfig configuration class: BigBirdForMaskedLM (BigBird model)
CamembertConfig configuration class: CamembertForMaskedLM (CamemBERT model)
ConvBertConfig configuration class: ConvBertForMaskedLM (ConvBERT model)
Data2VecTextCon


AutoModelForMaskedLM

fig configuration class: Data2VecTextForMaskedLM (Data2VecText model)
DebertaConfig configuration class: DebertaForMaskedLM (DeBERTa model)
DebertaV2Config configuration class: DebertaV2ForMaskedLM (DeBERTa-v2 model)
DistilBertConfig configuration class: DistilBertForMaskedLM (DistilBERT model)
ElectraConfig configuration class: ElectraForMaskedLM (ELECTRA model)
ErnieConfig configuration class: ErnieForMaskedLM (ERNIE model)
EsmConfig configuration class: EsmForMaskedLM (ESM model)
FNetConfig configuration class: FNetForMaskedLM (FNet model)
FlaubertConfig configuration class: FlaubertWithLMHeadModel (FlauBERT model)
FunnelConfig configuration class: FunnelForMaskedLM (Funnel Transformer model)
IBertConfig configuration class: IBertForMaskedLM (I-BERT model)
LayoutLMConfig configuration class: LayoutLMForMaskedLM (LayoutLM model)
LongformerConfig configuration class: LongformerForMaskedLM (Longformer model)
LukeConfig configuration class: LukeForMaskedLM (LUKE


AutoModelForMaskedLM

 model)
MBartConfig configuration class: MBartForConditionalGeneration (mBART model)
MPNetConfig configuration class: MPNetForMaskedLM (MPNet model)
MegaConfig configuration class: MegaForMaskedLM (MEGA model)
MegatronBertConfig configuration class: MegatronBertForMaskedLM (Megatron-BERT model)
MobileBertConfig configuration class: MobileBertForMaskedLM (MobileBERT model)
MraConfig configuration class: MraForMaskedLM (MRA model)
MvpConfig configuration class: MvpForConditionalGeneration (MVP model)
NezhaConfig configuration class: NezhaForMaskedLM (Nezha model)
NystromformerConfig configuration class: NystromformerForMaskedLM (Nyströmformer model)
PerceiverConfig configuration class: PerceiverForMaskedLM (Perceiver model)
QDQBertConfig configuration class: QDQBertForMaskedLM (QDQBert model)
ReformerConfig configuration class: ReformerForMaskedLM (Reformer model)
RemBertConfig configuration class: RemBertForMaskedLM (RemBERT model)
RoCBertConfig configuration cl


AutoModelForMaskedLM

ass: RoCBertForMaskedLM (RoCBert model)
RoFormerConfig configuration class: RoFormerForMaskedLM (RoFormer model)
RobertaConfig configuration class: RobertaForMaskedLM (RoBERTa model)
RobertaPreLayerNormConfig configuration class: RobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model)
SqueezeBertConfig configuration class: SqueezeBertForMaskedLM (SqueezeBERT model)
TapasConfig configuration class: TapasForMaskedLM (TAPAS model)
Wav2Vec2Config configuration class: Wav2Vec2ForMaskedLM (Wav2Vec2 model)
XLMConfig configuration class: XLMWithLMHeadModel (XLM model)
XLMRobertaConfig configuration class: XLMRobertaForMaskedLM (XLM-RoBERTa model)
XLMRobertaXLConfig configuration class: XLMRobertaXLForMaskedLM (XLM-RoBERTa-XL model)
XmodConfig configuration class: XmodForMaskedLM (X-MOD model)
YosoConfig configuration class: YosoForMaskedLM (YOSO model)
    Instantiates one of the model classes of the library (with a masked language modeling head) from a configurati


AutoModelForMaskedLM

on. Note:
Loading a model from its configuration file does not load the model weights. It only affects the
model’s configuration. Use from_pretrained() to load the model weights.  Examples:   Copied >>> from transformers import AutoConfig, AutoModelForMaskedLM
>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("bert-base-cased")
>>> model = AutoModelForMaskedLM.from_config(config)  from_pretrained  < source > ( *model_args **kwargs  )   Parameters   pretrained_model_name_or_path (str or os.PathLike) —
Can be either:
A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like bert-base-uncased, or namespaced under a
user or organization name, like dbmdz/bert-base-german-cased.
A path to a directory containing model weights saved using
save_pretrained(), e.g., ./my_model_directory/.
A path or url to a tensorflow index checkpoint f


AutoModelForMaskedLM

ile (e.g, ./tf_model/model.ckpt.index). In
this case, from_tf should be set to True and a configuration object should be provided as
config argument. This loading path is slower than converting the TensorFlow checkpoint in a
PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
   model_args (additional positional arguments, optional) —
Will be passed along to the underlying model __init__() method.   config (PretrainedConfig, optional) —
Configuration for the model to use instead of an automatically loaded configuration. Configuration can
be automatically loaded when:
The model is a model provided by the library (loaded with the model id string of a pretrained
model).
The model was saved using save_pretrained() and is reloaded by supplying the
save directory.
The model is loaded by supplying a local directory as pretrained_model_name_or_path and a
configuration JSON file named config.json is found in the directory.
   st


AutoModelForMaskedLM

ate_dict (Dict[str, torch.Tensor], optional) —
A state dictionary to use instead of a state dictionary loaded from saved weights file.
This option can be used if you want to create a model from a pretrained configuration but load your own
weights. In this case though, you should check if using save_pretrained() and
from_pretrained() is not a simpler option.   cache_dir (str or os.PathLike, optional) —
Path to a directory in which a downloaded pretrained model configuration should be cached if the
standard cache should not be used.   from_tf (bool, optional, defaults to False) —
Load the model weights from a TensorFlow checkpoint save file (see docstring of
pretrained_model_name_or_path argument).   force_download (bool, optional, defaults to False) —
Whether or not to force the (re-)download of the model weights and configuration files, overriding the
cached versions if they exist.   resume_download (bool, optional, defaults to False) —
Whether or not to delete


AutoModelForMaskedLM

 incompletely received files. Will attempt to resume the download if such a
file exists.   proxies (Dict[str, str], optional) —
A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.   output_loading_info(bool, optional, defaults to False) —
Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.   local_files_only(bool, optional, defaults to False) —
Whether or not to only look at local files (e.g., not try downloading the model).   revision (str, optional, defaults to "main") —
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so revision can be any
identifier allowed by git.   trust_remote_code (bool, optional, defaults to False) —
Whether or not to allow for custom models de


AutoModelForMaskedLM

fined on the Hub in their own modeling files. This option
should only be set to True for repositories you trust and in which you have read the code, as it will
execute code present on the Hub on your local machine.   code_revision (str, optional, defaults to "main") —
The specific revision to use for the code on the Hub, if the code leaves in a different repository than
the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based
system for storing models and other artifacts on huggingface.co, so revision can be any identifier
allowed by git.   kwargs (additional keyword arguments, optional) —
Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
output_attentions=True). Behaves differently depending on whether a config is provided or
automatically loaded:
If a configuration is provided with config, **kwargs will be directly passed to the
underlying model’s __init__ method (w


AutoModelForMaskedLM

e assume all relevant updates to the configuration have
already been done)
If a configuration is not provided, kwargs will be first passed to the configuration class
initialization function (from_pretrained()). Each key of kwargs that
corresponds to a configuration attribute will be used to override said attribute with the
supplied kwargs value. Remaining keys that do not correspond to any configuration attribute
will be passed to the underlying model’s __init__ function.
    Instantiate one of the model classes of the library (with a masked language modeling head) from a pretrained model. The model class to instantiate is selected based on the model_type property of the config object (either
passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by
falling back to using pattern matching on pretrained_model_name_or_path: albert — AlbertForMaskedLM (ALBERT model) bart — BartForConditionalGeneration (BART model) ber


AutoModelForMaskedLM

t — BertForMaskedLM (BERT model) big_bird — BigBirdForMaskedLM (BigBird model) camembert — CamembertForMaskedLM (CamemBERT model) convbert — ConvBertForMaskedLM (ConvBERT model) data2vec-text — Data2VecTextForMaskedLM (Data2VecText model) deberta — DebertaForMaskedLM (DeBERTa model) deberta-v2 — DebertaV2ForMaskedLM (DeBERTa-v2 model) distilbert — DistilBertForMaskedLM (DistilBERT model) electra — ElectraForMaskedLM (ELECTRA model) ernie — ErnieForMaskedLM (ERNIE model) esm — EsmForMaskedLM (ESM model) flaubert — FlaubertWithLMHeadModel (FlauBERT model) fnet — FNetForMaskedLM (FNet model) funnel — FunnelForMaskedLM (Funnel Transformer model) ibert — IBertForMaskedLM (I-BERT model) layoutlm — LayoutLMForMaskedLM (LayoutLM model) longformer — LongformerForMaskedLM (Longformer model) luke — LukeForMaskedLM (LUKE model) mbart — MBartForConditionalGeneration (mBART model) mega — MegaForMaskedLM (MEGA model) megatron-bert — MegatronBertForMaskedLM (Megatron-BERT mode


AutoModelForMaskedLM

l) mobilebert — MobileBertForMaskedLM (MobileBERT model) mpnet — MPNetForMaskedLM (MPNet model) mra — MraForMaskedLM (MRA model) mvp — MvpForConditionalGeneration (MVP model) nezha — NezhaForMaskedLM (Nezha model) nystromformer — NystromformerForMaskedLM (Nyströmformer model) perceiver — PerceiverForMaskedLM (Perceiver model) qdqbert — QDQBertForMaskedLM (QDQBert model) reformer — ReformerForMaskedLM (Reformer model) rembert — RemBertForMaskedLM (RemBERT model) roberta — RobertaForMaskedLM (RoBERTa model) roberta-prelayernorm — RobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model) roc_bert — RoCBertForMaskedLM (RoCBert model) roformer — RoFormerForMaskedLM (RoFormer model) squeezebert — SqueezeBertForMaskedLM (SqueezeBERT model) tapas — TapasForMaskedLM (TAPAS model) wav2vec2 — Wav2Vec2ForMaskedLM (Wav2Vec2 model) xlm — XLMWithLMHeadModel (XLM model) xlm-roberta — XLMRobertaForMaskedLM (XLM-RoBERTa model) xlm-roberta-xl — XLMRobertaXLForMaskedLM (XLM-RoBE


AutoModelForMaskedLM

RTa-XL model) xmod — XmodForMaskedLM (X-MOD model) yoso — YosoForMaskedLM (YOSO model) The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are
deactivated). To train the model, you should first set it back in training mode with model.train()  Examples:   Copied >>> from transformers import AutoConfig, AutoModelForMaskedLM
>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForMaskedLM.from_pretrained("bert-base-cased")
>>> # Update configuration during loading
>>> model = AutoModelForMaskedLM.from_pretrained("bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True
>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForMaskedLM.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )


TFAutoModelForMaskedLM

class transformers.TFAutoModelForMaskedLM  < source > ( *args **kwargs  )    This is a generic model class that will be instantiated as one of the model classes of the library (with a masked language modeling head) when created
with the from_pretrained() class method or the from_config() class
method. This class cannot be instantiated directly using __init__() (throws an error).  from_config  < source > ( **kwargs  )   Parameters   config (PretrainedConfig) —
The model class to instantiate is selected based on the configuration class:
AlbertConfig configuration class: TFAlbertForMaskedLM (ALBERT model)
BertConfig configuration class: TFBertForMaskedLM (BERT model)
CamembertConfig configuration class: TFCamembertForMaskedLM (CamemBERT model)
ConvBertConfig configuration class: TFConvBertForMaskedLM (ConvBERT model)
DebertaConfig configuration class: TFDebertaForMaskedLM (DeBERTa model)
DebertaV2Config configuration class: TFDebertaV2For


TFAutoModelForMaskedLM

MaskedLM (DeBERTa-v2 model)
DistilBertConfig configuration class: TFDistilBertForMaskedLM (DistilBERT model)
ElectraConfig configuration class: TFElectraForMaskedLM (ELECTRA model)
EsmConfig configuration class: TFEsmForMaskedLM (ESM model)
FlaubertConfig configuration class: TFFlaubertWithLMHeadModel (FlauBERT model)
FunnelConfig configuration class: TFFunnelForMaskedLM (Funnel Transformer model)
LayoutLMConfig configuration class: TFLayoutLMForMaskedLM (LayoutLM model)
LongformerConfig configuration class: TFLongformerForMaskedLM (Longformer model)
MPNetConfig configuration class: TFMPNetForMaskedLM (MPNet model)
MobileBertConfig configuration class: TFMobileBertForMaskedLM (MobileBERT model)
RemBertConfig configuration class: TFRemBertForMaskedLM (RemBERT model)
RoFormerConfig configuration class: TFRoFormerForMaskedLM (RoFormer model)
RobertaConfig configuration class: TFRobertaForMaskedLM (RoBERTa model)
RobertaPreLayerNormConfig


TFAutoModelForMaskedLM

configuration class: TFRobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model)
TapasConfig configuration class: TFTapasForMaskedLM (TAPAS model)
XLMConfig configuration class: TFXLMWithLMHeadModel (XLM model)
XLMRobertaConfig configuration class: TFXLMRobertaForMaskedLM (XLM-RoBERTa model)
    Instantiates one of the model classes of the library (with a masked language modeling head) from a configuration. Note:
Loading a model from its configuration file does not load the model weights. It only affects the
model’s configuration. Use from_pretrained() to load the model weights.  Examples:   Copied >>> from transformers import AutoConfig, TFAutoModelForMaskedLM
>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("bert-base-cased")
>>> model = TFAutoModelForMaskedLM.from_config(config)  from_pretrained  < source > ( *model_args **kwargs  )   Parameters   pretrained_model_name_or_path (st


TFAutoModelForMaskedLM

r or os.PathLike) —
Can be either:
A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like bert-base-uncased, or namespaced under a
user or organization name, like dbmdz/bert-base-german-cased.
A path to a directory containing model weights saved using
save_pretrained(), e.g., ./my_model_directory/.
A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this
case, from_pt should be set to True and a configuration object should be provided as config
argument. This loading path is slower than converting the PyTorch model in a TensorFlow model
using the provided conversion scripts and loading the TensorFlow model afterwards.
   model_args (additional positional arguments, optional) —
Will be passed along to the underlying model __init__() method.   config (PretrainedConfig, optional) —
Configuration for the model to u


TFAutoModelForMaskedLM

se instead of an automatically loaded configuration. Configuration can
be automatically loaded when:
The model is a model provided by the library (loaded with the model id string of a pretrained
model).
The model was saved using save_pretrained() and is reloaded by supplying the
save directory.
The model is loaded by supplying a local directory as pretrained_model_name_or_path and a
configuration JSON file named config.json is found in the directory.
   cache_dir (str or os.PathLike, optional) —
Path to a directory in which a downloaded pretrained model configuration should be cached if the
standard cache should not be used.   from_pt (bool, optional, defaults to False) —
Load the model weights from a PyTorch checkpoint save file (see docstring of
pretrained_model_name_or_path argument).   force_download (bool, optional, defaults to False) —
Whether or not to force the (re-)download of the model weights and configuration files, overrid


TFAutoModelForMaskedLM

ing the
cached versions if they exist.   resume_download (bool, optional, defaults to False) —
Whether or not to delete incompletely received files. Will attempt to resume the download if such a
file exists.   proxies (Dict[str, str], optional) —
A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.   output_loading_info(bool, optional, defaults to False) —
Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.   local_files_only(bool, optional, defaults to False) —
Whether or not to only look at local files (e.g., not try downloading the model).   revision (str, optional, defaults to "main") —
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so revi


TFAutoModelForMaskedLM

sion can be any
identifier allowed by git.   trust_remote_code (bool, optional, defaults to False) —
Whether or not to allow for custom models defined on the Hub in their own modeling files. This option
should only be set to True for repositories you trust and in which you have read the code, as it will
execute code present on the Hub on your local machine.   code_revision (str, optional, defaults to "main") —
The specific revision to use for the code on the Hub, if the code leaves in a different repository than
the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based
system for storing models and other artifacts on huggingface.co, so revision can be any identifier
allowed by git.   kwargs (additional keyword arguments, optional) —
Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
output_attentions=True). Behaves differently depending on whet


TFAutoModelForMaskedLM

her a config is provided or
automatically loaded:
If a configuration is provided with config, **kwargs will be directly passed to the
underlying model’s __init__ method (we assume all relevant updates to the configuration have
already been done)
If a configuration is not provided, kwargs will be first passed to the configuration class
initialization function (from_pretrained()). Each key of kwargs that
corresponds to a configuration attribute will be used to override said attribute with the
supplied kwargs value. Remaining keys that do not correspond to any configuration attribute
will be passed to the underlying model’s __init__ function.
    Instantiate one of the model classes of the library (with a masked language modeling head) from a pretrained model. The model class to instantiate is selected based on the model_type property of the config object (either
passed as an argument or loaded from pretrained_model_name_or_path if possib


TFAutoModelForMaskedLM

le), or when it’s missing, by
falling back to using pattern matching on pretrained_model_name_or_path: albert — TFAlbertForMaskedLM (ALBERT model) bert — TFBertForMaskedLM (BERT model) camembert — TFCamembertForMaskedLM (CamemBERT model) convbert — TFConvBertForMaskedLM (ConvBERT model) deberta — TFDebertaForMaskedLM (DeBERTa model) deberta-v2 — TFDebertaV2ForMaskedLM (DeBERTa-v2 model) distilbert — TFDistilBertForMaskedLM (DistilBERT model) electra — TFElectraForMaskedLM (ELECTRA model) esm — TFEsmForMaskedLM (ESM model) flaubert — TFFlaubertWithLMHeadModel (FlauBERT model) funnel — TFFunnelForMaskedLM (Funnel Transformer model) layoutlm — TFLayoutLMForMaskedLM (LayoutLM model) longformer — TFLongformerForMaskedLM (Longformer model) mobilebert — TFMobileBertForMaskedLM (MobileBERT model) mpnet — TFMPNetForMaskedLM (MPNet model) rembert — TFRemBertForMaskedLM (RemBERT model) roberta — TFRobertaForMaskedLM (RoBERTa model) roberta-prelay


TFAutoModelForMaskedLM

ernorm — TFRobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model) roformer — TFRoFormerForMaskedLM (RoFormer model) tapas — TFTapasForMaskedLM (TAPAS model) xlm — TFXLMWithLMHeadModel (XLM model) xlm-roberta — TFXLMRobertaForMaskedLM (XLM-RoBERTa model)  Examples:   Copied >>> from transformers import AutoConfig, TFAutoModelForMaskedLM
>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForMaskedLM.from_pretrained("bert-base-cased")
>>> # Update configuration during loading
>>> model = TFAutoModelForMaskedLM.from_pretrained("bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True
>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = TFAutoModelForMaskedLM.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
.


FlaxAutoModelForMaskedLM

class transformers.FlaxAutoModelForMaskedLM  < source > ( *args **kwargs  )    This is a generic model class that will be instantiated as one of the model classes of the library (with a masked language modeling head) when created
with the from_pretrained() class method or the from_config() class
method. This class cannot be instantiated directly using __init__() (throws an error).  from_config  < source > ( **kwargs  )   Parameters   config (PretrainedConfig) —
The model class to instantiate is selected based on the configuration class:
AlbertConfig configuration class: FlaxAlbertForMaskedLM (ALBERT model)
BartConfig configuration class: FlaxBartForConditionalGeneration (BART model)
BertConfig configuration class: FlaxBertForMaskedLM (BERT model)
BigBirdConfig configuration class: FlaxBigBirdForMaskedLM (BigBird model)
DistilBertConfig configuration class: FlaxDistilBertForMaskedLM (DistilBERT model)


FlaxAutoModelForMaskedLM

ElectraConfig configuration class: FlaxElectraForMaskedLM (ELECTRA model)
MBartConfig configuration class: FlaxMBartForConditionalGeneration (mBART model)
RoFormerConfig configuration class: FlaxRoFormerForMaskedLM (RoFormer model)
RobertaConfig configuration class: FlaxRobertaForMaskedLM (RoBERTa model)
RobertaPreLayerNormConfig configuration class: FlaxRobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model)
XLMRobertaConfig configuration class: FlaxXLMRobertaForMaskedLM (XLM-RoBERTa model)
    Instantiates one of the model classes of the library (with a masked language modeling head) from a configuration. Note:
Loading a model from its configuration file does not load the model weights. It only affects the
model’s configuration. Use from_pretrained() to load the model weights.  Examples:   Copied >>> from transformers import AutoConfig, FlaxAutoModelForMaskedLM
>>> # Download configuration fro