sor of shape batch_size, sequence_length, optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape batch_size, sequence_length, optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to direct


ly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed


 (Cross-Entropy).   Returns
transformers.modeling_outputs.SequenceClassifierOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.SequenceClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (LayoutLMv2Config) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of e


ach layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The LayoutLMv2ForSequenceClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import


 AutoProcessor, LayoutLMv2ForSequenceClassification, set_seed
>>> from PIL import Image
>>> import torch
>>> from datasets import load_dataset
>>> set_seed(88)
>>> dataset = load_dataset("rvl_cdip", split="train", streaming=True)
>>> data = next(iter(dataset))
>>> image = data["image"].convert("RGB")
>>> processor = AutoProcessor.from_pretrained("microsoft/layoutlmv2-base-uncased")
>>> model = LayoutLMv2ForSequenceClassification.from_pretrained(
...     "microsoft/layoutlmv2-base-uncased", num_labels=dataset.info.features["label"].num_classes
... )
>>> encoding = processor(image, return_tensors="pt")
>>> sequence_label = torch.tensor([data["label"]])
>>> outputs = model(**encoding, labels=sequence_label)
>>> loss, logits = outputs.loss, outputs.logits
>>> predicted_idx = logits.argmax(dim=-1).item()
>>> predicted_answer = dataset.info.features["label"].names[4]
>>> predicted_idx, predicted_answer
(4, 'advertisement'


class transformers.LayoutLMv2ForTokenClassification  < source > ( config  )   Parameters   config (LayoutLMv2Config) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    LayoutLMv2 Model with a token classification head on top (a linear layer on top of the text part of the hidden
states) e.g. for sequence labeling (information extraction) tasks such as
FUNSD, SROIE,
CORD and Kleister-NDA. This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None bbox: typing.Optional[torch.LongTensor] = None image: typing.Optional[torch.FloatTensor] = None


 attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.LongTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.TokenClassifierOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape batch_size, sequence_length) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   bbox (torch.LongTensor of shape (batch_size, sequence_length, 4), optional) —
Bounding boxes of ea


ch input sequence tokens. Selected in the range [0, config.max_2d_position_embeddings-1]. Each bounding box should be a normalized version in (x0, y0, x1, y1)
format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1,
y1) represents the position of the lower right corner.   image (torch.FloatTensor of shape (batch_size, num_channels, height, width) or detectron.structures.ImageList whose tensors is of shape (batch_size, num_channels, height, width)) —
Batch of document images.   attention_mask (torch.FloatTensor of shape batch_size, sequence_length, optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape batch_size, sequence_length, optional) —
Segment token indices to indicate first and secon


d portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape batch_size, sequence_length, optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids


indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Labels for computing the token classification loss. Indices should be in [0, ..., config.num_labels - 1].   Returns
transformers.modeling_outputs.TokenClassifierOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.TokenClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=Fa


lse) comprising various
elements depending on the configuration (LayoutLMv2Config) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided)  — Classification loss.
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.num_labels)) — Classification scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (on


e for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The LayoutLMv2ForTokenClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoProcessor, LayoutLMv2ForTokenClassification, set_seed
>>> from PIL import Image
>>> from datasets import load_dataset
>>> set_seed(88)
>>> datasets = load_dataset("nielsr/funsd", split="test")
>>> labels = datasets.features["ner_tags"].feature.names
>>> id2label = {v: k for v, k in enumerate(labels)}
>>> processor = AutoProcessor


.from_pretrained("microsoft/layoutlmv2-base-uncased", revision="no_ocr")
>>> model = LayoutLMv2ForTokenClassification.from_pretrained(
...     "microsoft/layoutlmv2-base-uncased", num_labels=len(labels)
... )
>>> data = datasets[0]
>>> image = Image.open(data["image_path"]).convert("RGB")
>>> words = data["words"]
>>> boxes = data["bboxes"]  # make sure to normalize your bounding boxes
>>> word_labels = data["ner_tags"]
>>> encoding = processor(
...     image,
...     words,
...     boxes=boxes,
...     word_labels=word_labels,
...     padding="max_length",
...     truncation=True,
...     return_tensors="pt",
... )
>>> outputs = model(**encoding)
>>> logits, loss = outputs.logits, outputs.loss
>>> predicted_token_class_ids = logits.argmax(-1)
>>> predicted_tokens_classes = [id2label[t.item()] for t in predicted_token_class_ids[0]]
>>> predicted_tokens_classes[:5]
['B-ANSWER', 'B-HEADER', 'B-HEADER', 'B-HEADER', 'B-HEA


class transformers.LayoutLMv2ForQuestionAnswering  < source > ( config has_visual_segment_embedding = True  )   Parameters   config (LayoutLMv2Config) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    LayoutLMv2 Model with a span classification head on top for extractive question-answering tasks such as
DocVQA (a linear layer on top of the text part of the hidden-states output to
compute span start logits and span end logits). This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( input_ids: typing.Optional[torch.LongTensor] = None bbox: typing.Optional[torch.LongTensor] = None image: typing.Optional[torch.FloatTensor] = N


one attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None start_positions: typing.Optional[torch.LongTensor] = None end_positions: typing.Optional[torch.LongTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.QuestionAnsweringModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape batch_size, sequence_length) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   bbox (torch.LongTensor of shape (batch_size, sequence_length,


4), optional) —
Bounding boxes of each input sequence tokens. Selected in the range [0, config.max_2d_position_embeddings-1]. Each bounding box should be a normalized version in (x0, y0, x1, y1)
format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1,
y1) represents the position of the lower right corner.   image (torch.FloatTensor of shape (batch_size, num_channels, height, width) or detectron.structures.ImageList whose tensors is of shape (batch_size, num_channels, height, width)) —
Batch of document images.   attention_mask (torch.FloatTensor of shape batch_size, sequence_length, optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape batch_size, sequence_length, optional) —
Segment token indices to indicate first and second p


ortions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   position_ids (torch.LongTensor of shape batch_size, sequence_length, optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
m


odel’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   start_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the start of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
are not taken into account for computing the loss.   end_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the end of the labelled span for computing the token classification loss


.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
are not taken into account for computing the loss.   Returns
transformers.modeling_outputs.QuestionAnsweringModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.QuestionAnsweringModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (LayoutLMv2Config) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Total span extraction loss is the sum of a Cross-Entropy for the start and end positions.
start_logits (torch.FloatTensor of shape (batch_size, sequence_length)) — Span-start scores (before SoftMax).
end_logits (torch.FloatTensor of shape (batch_size, sequence_length)) — Span-end scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_h


idden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The LayoutLMv2ForQuestionAnswering forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since t


he former takes care of running the pre and post processing steps while
the latter silently ignores them. Example: In this example below, we give the LayoutLMv2 model an image (of texts) and ask it a question. It will give us
a prediction of what it thinks the answer is (the span of the answer within the texts parsed from the image).    Copied >>> from transformers import AutoProcessor, LayoutLMv2ForQuestionAnswering, set_seed
>>> import torch
>>> from PIL import Image
>>> from datasets import load_dataset
>>> set_seed(88)
>>> processor = AutoProcessor.from_pretrained("microsoft/layoutlmv2-base-uncased")
>>> model = LayoutLMv2ForQuestionAnswering.from_pretrained("microsoft/layoutlmv2-base-uncased")
>>> dataset = load_dataset("hf-internal-testing/fixtures_docvqa")
>>> image_path = dataset["test"][0]["file"]
>>> image = Image.open(image_path).convert("RGB")
>>> question = "When is coffee break?"
>>> encoding = processor(image, question, return_tensors="pt")
>


>> outputs = model(**encoding)
>>> predicted_start_idx = outputs.start_logits.argmax(-1).item()
>>> predicted_end_idx = outputs.end_logits.argmax(-1).item()
>>> predicted_start_idx, predicted_end_idx
(154, 287)
>>> predicted_answer_tokens = encoding.input_ids.squeeze()[predicted_start_idx : predicted_end_idx + 1]
>>> predicted_answer = processor.tokenizer.decode(predicted_answer_tokens)
>>> predicted_answer  # results are not very good without further fine-tuning
'council mem - bers conducted by trrf treasurer philip g. kuehn to get answers which the public ...    Copied >>> target_start_index = torch.tensor([7])
>>> target_end_index = torch.tensor([14])
>>> outputs = model(**encoding, start_positions=target_start_index, end_positions=target_end_index)
>>> predicted_answer_span_start = outputs.start_logits.argmax(-1).item()
>>> predicted_answer_span_end = outputs.end_logits.argmax(-1).item()
>>> predicted_answer_span_start, predicted_answer_span_end
(154, 2


Overview

The Nougat model was proposed in Nougat: Neural Optical Understanding for Academic Documents by
Lukas Blecher, Guillem Cucurull, Thomas Scialom, Robert Stojnic. Nougat uses the same architecture as Donut, meaning an image Transformer
encoder and an autoregressive text Transformer decoder to translate scientific PDFs to markdown, enabling easier access to them.
The abstract from the paper is the following:
Scientific knowledge is predominantly stored in books and scientific journals, often in the form of PDFs. However, the PDF format leads to a loss of semantic information, particularly for mathematical expressions. We propose Nougat (Neural Optical Understanding for Academic Documents), a Visual Transformer model that performs an Optical Character Recognition (OCR) task for processing scientific documents into a mark


Overview

up language, and demonstrate the effectiveness of our model on a new dataset of scientific documents. The proposed approach offers a promising solution to enhance the accessibility of scientific knowledge in the digital age, by bridging the gap between human-readable documents and machine-readable text. We release the models and code to accelerate future work on scientific text recognition.
Nougat high-level overview. Taken from the original paper.
This model was contributed by nielsr. The original code can be found
here.
Tips:
The quickest way to get started with Nougat is by checking the tutorial
notebooks, which show how to use the model
at inference time as well as fine-tuning on custom data. Nougat is always used within the VisionEncoderDecoder framework. The model is identical to Donut in terms of architecture.


Inference

Nougat’s VisionEncoderDecoder model accepts images as input and makes use of
generate() to autoregressively generate text given the input image.
The NougatImageProcessor class is responsible for preprocessing the input image and
NougatTokenizerFast decodes the generated target tokens to the target string. The
NougatProcessor wraps NougatImageProcessor and NougatTokenizerFast classes
into a single instance to both extract the input features and decode the predicted token ids.
Step-by-step PDF transcription
  Copied >>> from huggingface_hub import hf_hub_download
>>> import re
>>> from PIL import Image
>>> from transformers import NougatProcessor, VisionEncoderDecoderModel
>>> from datasets import load_dataset
>>> import torch
>>> processor = NougatProcessor.from_pretrained("facebook/nougat-base")
>>> model = VisionEncoderDecoderModel.from_pretrained("facebook/nougat-base")
>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> model.to(device)
>>> #


Inference

prepare PDF image for the model
>>> filepath = hf_hub_download(repo_id="hf-internal-testing/fixtures_docvqa", filename="nougat_paper.png", repo_type="dataset")
>>> image = Image.open(filepath)
>>> pixel_values = processor(image, return_tensors="pt").pixel_values
>>> # generate transcription (here we only generate 30 tokens)
>>> outputs = model.generate(
...     pixel_values.to(device),
...     min_length=1,
...     max_new_tokens=30,
...     bad_words_ids=[[processor.tokenizer.unk_token_id]],
... )
>>> sequence = processor.batch_decode(outputs, skip_special_tokens=True)[0]
>>> sequence = processor.post_process_generation(sequence, fix_markdown=False)
>>> # note: we're using repr here such for the sake of printing the \n characters, feel free to just print the sequence
>>> print(repr(sequence))
'\n\n# Nougat: Neural Optical Understanding for Academic Documents\n\n Lukas Blecher\n\nCorrespondence to: lblecher@'
See the model hub to look for Nougat checkpoints.


NougatImageProcessor

class transformers.NougatImageProcessor  < source > ( do_crop_margin: bool = True do_resize: bool = True size: typing.Dict[str, int] = None resample: Resampling = <Resampling.BILINEAR: 2> do_thumbnail: bool = True do_align_long_axis: bool = False do_pad: bool = True do_rescale: bool = True rescale_factor: typing.Union[int, float] = 0.00392156862745098 do_normalize: bool = True image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float], NoneType] = None **kwargs  )   Parameters   do_crop_margin (bool, optional, defaults to True) —
Whether to crop the image margins.   do_resize (bool, optional, defaults to True) —
Whether to resize the image’s (height, width) dimensions to the specified size. Can be overridden by
do_resize in the preprocess method.   size (Dict[str, int] optional, defaults to {"height" -- 896, "width": 672}):
Size of the image after r


NougatImageProcessor

esizing. Can be overridden by size in the preprocess method.   resample (PILImageResampling, optional, defaults to Resampling.BILINEAR) —
Resampling filter to use if resizing the image. Can be overridden by resample in the preprocess method.   do_thumbnail (bool, optional, defaults to True) —
Whether to resize the image using thumbnail method.   do_align_long_axis (bool, optional, defaults to False) —
Whether to align the long axis of the image with the long axis of size by rotating by 90 degrees.   do_pad (bool, optional, defaults to True) —
Whether to pad the images to the largest image size in the batch.   do_rescale (bool, optional, defaults to True) —
Whether to rescale the image by the specified scale rescale_factor. Can be overridden by the do_rescale
parameter in the preprocess method.   rescale_factor (int or float, optional, defaults to 1/255) —
Scale factor to use if rescaling the image. Can be ov


NougatImageProcessor

erridden by the rescale_factor parameter in the
preprocess method.   do_normalize (bool, optional, defaults to True) —
Whether to normalize the image. Can be overridden by do_normalize in the preprocess method.   image_mean (float or List[float], optional, defaults to IMAGENET_DEFAULT_MEAN) —
Mean to use if normalizing the image. This is a float or list of floats the length of the number of
channels in the image. Can be overridden by the image_mean parameter in the preprocess method.   image_std (float or List[float], optional, defaults to IMAGENET_DEFAULT_STD) —
Image standard deviation.    Constructs a Nougat image processor.  preprocess  < source > ( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]] do_crop_margin: bool = None do_resize: bool = None


NougatImageProcessor

size: typing.Dict[str, int] = None resample: Resampling = None do_thumbnail: bool = None do_align_long_axis: bool = None do_pad: bool = None do_rescale: bool = None rescale_factor: typing.Union[int, float] = None do_normalize: bool = None image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float], NoneType] = None return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None data_format: typing.Optional[transformers.image_utils.ChannelDimension] = <ChannelDimension.FIRST: 'channels_first'> input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None **kwargs  )   Parameters   images (ImageInput) —
Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255.   do_crop_margin (bool, optional, defaults to self.do_crop_margin) —
Whether to crop the image


NougatImageProcessor

 margins.   do_resize (bool, optional, defaults to self.do_resize) —
Whether to resize the image.   size (Dict[str, int], optional, defaults to self.size) —
Size of the image after resizing. Shortest edge of the image is resized to min(size[“height”],
size[“width”]) with the longest edge resized to keep the input aspect ratio.   resample (int, optional, defaults to self.resample) —
Resampling filter to use if resizing the image. This can be one of the enum PILImageResampling. Only
has an effect if do_resize is set to True.   do_thumbnail (bool, optional, defaults to self.do_thumbnail) —
Whether to resize the image using thumbnail method.   do_align_long_axis (bool, optional, defaults to self.do_align_long_axis) —
Whether to align the long axis of the image with the long axis of size by rotating by 90 degrees.   do_pad (bool, optional, defaults to self.do_pad) —
Whether to pad the images to the largest image


NougatImageProcessor

size in the batch.   do_rescale (bool, optional, defaults to self.do_rescale) —
Whether to rescale the image by the specified scale rescale_factor.   rescale_factor (int or float, optional, defaults to self.rescale_factor) —
Scale factor to use if rescaling the image.   do_normalize (bool, optional, defaults to self.do_normalize) —
Whether to normalize the image.   image_mean (float or List[float], optional, defaults to self.image_mean) —
Image mean to use for normalization.   image_std (float or List[float], optional, defaults to self.image_std) —
Image standard deviation to use for normalization.   return_tensors (str or TensorType, optional) —
The type of tensors to return. Can be one of:
Unset: Return a list of np.ndarray.
TensorType.TENSORFLOW or 'tf': Return a batch of type tf.Tensor.
TensorType.PYTORCH or 'pt': Return a batch of type torch.Tensor.
TensorType.NUMPY or 'np': Return a batch of type np.nd


NougatImageProcessor

array.
TensorType.JAX or 'jax': Return a batch of type jax.numpy.ndarray.
   data_format (ChannelDimension or str, optional, defaults to ChannelDimension.FIRST) —
The channel dimension format for the output image. Can be one of:
ChannelDimension.FIRST: image in (num_channels, height, width) format.
ChannelDimension.LAST: image in (height, width, num_channels) format.
Unset: defaults to the channel dimension format of the input image.
   input_data_format (ChannelDimension or str, optional) —
The channel dimension format for the input image. If unset, the channel dimension format is inferred
from the input image. Can be one of:
"channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
"channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
"none" or ChannelDimension.NONE: image in (height, width) format.
    Preprocess an image or batch of imag


NougatTokenizerFast

class transformers.NougatTokenizerFast  < source > ( vocab_file = None tokenizer_file = None clean_up_tokenization_spaces = False unk_token = '<unk>' bos_token = '<s>' eos_token = '</s>' pad_token = '<pad>' **kwargs  )   Parameters   vocab_file (str, optional) —
SentencePiece file (generally has a .model extension) that
contains the vocabulary necessary to instantiate a tokenizer.   tokenizer_file (str, optional) —
tokenizers file (generally has a .json extension) that
contains everything needed to load the tokenizer.   clean_up_tokenization_spaces (str, optional, defaults to False) —
Wether to cleanup spaces after decoding, cleanup consists in removing potential artifacts like extra
spaces.   unk_token (str, optional, defaults to "<unk>") —
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.   bos_token (str, optional, defaults to


NougatTokenizerFast

 "<s>") —
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.   eos_token (str, optional, defaults to "</s>") —
The end of sequence token.   pad_token (str, optional, defaults to "<pad>") —
The token used for padding, for example when batching sequences of different lengths.   model_max_length (int, optional) —
The maximum length (in number of tokens) for the inputs to the transformer model. When the tokenizer is
loaded with from_pretrained(), this will be set to the
value stored for the associated model in max_model_input_sizes (see above). If no value is provided, will
default to VERY_LARGE_INTEGER (int(1e30)).   padding_side (str, optional) —
The side on which the model should have padding applied. Should be selected between [‘right’, ‘left’].
Default value is picked from the class attribute of the same name.   truncation_side (str, optional) —


NougatTokenizerFast

The side on which the model should have truncation applied. Should be selected between [‘right’, ‘left’].
Default value is picked from the class attribute of the same name.   chat_template (str, optional) —
A Jinja template string that will be used to format lists of chat messages. See
https://huggingface.co/docs/transformers/chat_templating for a full description.   model_input_names (List[string], optional) —
The list of inputs accepted by the forward pass of the model (like "token_type_ids" or
"attention_mask"). Default value is picked from the class attribute of the same name.   bos_token (str or tokenizers.AddedToken, optional) —
A special token representing the beginning of a sentence. Will be associated to self.bos_token and
self.bos_token_id.   eos_token (str or tokenizers.AddedToken, optional) —
A special token representing the end of a sentence. Will be associated to self.eos_token and
self


NougatTokenizerFast

.eos_token_id.   unk_token (str or tokenizers.AddedToken, optional) —
A special token representing an out-of-vocabulary token. Will be associated to self.unk_token and
self.unk_token_id.   sep_token (str or tokenizers.AddedToken, optional) —
A special token separating two different sentences in the same input (used by BERT for instance). Will be
associated to self.sep_token and self.sep_token_id.   pad_token (str or tokenizers.AddedToken, optional) —
A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by
attention mechanisms or loss computation. Will be associated to self.pad_token and self.pad_token_id.   cls_token (str or tokenizers.AddedToken, optional) —
A special token representing the class of the input (used by BERT for instance). Will be associated to
self.cls_token and self.cls_token_id.   mask_token (str or tokenizers.AddedToken, optional) —


NougatTokenizerFast

A special token representing a masked token (used by masked-language modeling pretraining objectives, like
BERT). Will be associated to self.mask_token and self.mask_token_id.   additional_special_tokens (tuple or list of str or tokenizers.AddedToken, optional) —
A tuple or a list of additional special tokens. Add them here to ensure they are skipped when decoding with
skip_special_tokens is set to True. If they are not part of the vocabulary, they will be added at the end
of the vocabulary.   clean_up_tokenization_spaces (bool, optional, defaults to True) —
Whether or not the model should cleanup the spaces that were added when splitting the input text during the
tokenization process.   split_special_tokens (bool, optional, defaults to False) —
Whether or not the special tokens should be split during the tokenization process. The default behavior is
to not split special tokens. This means that if <s


NougatTokenizerFast

> is the bos_token, then tokenizer.tokenize("<s>") = ['<s>]. Otherwise, if split_special_tokens=True, then tokenizer.tokenize("<s>") will be give ['<', 's', '>']. This argument is only supported for slow tokenizers for the moment.   tokenizer_object (tokenizers.Tokenizer) —
A tokenizers.Tokenizer object from 🤗 tokenizers to instantiate from. See Using tokenizers from 🤗
tokenizers for more information.   tokenizer_file (str) —
A path to a local JSON file representing a previously serialized tokenizers.Tokenizer object from 🤗
tokenizers.    Fast tokenizer for Nougat (backed by HuggingFace tokenizers library). This tokenizer inherits from PreTrainedTokenizerFast which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods. This class mainly adds Nougat-specific
methods for postprocessing the generated text. Class attributes (overridden by der


NougatTokenizerFast

ived classes) vocab_files_names (Dict[str, str]) — A dictionary with, as keys, the __init__ keyword name of each
vocabulary file required by the model, and as associated values, the filename for saving the associated file
(string). pretrained_vocab_files_map (Dict[str, Dict[str, str]]) — A dictionary of dictionaries, with the
high-level keys being the __init__ keyword name of each vocabulary file required by the model, the
low-level being the short-cut-names of the pretrained models with, as associated values, the url to the
associated pretrained vocabulary file. max_model_input_sizes (Dict[str, Optional[int]]) — A dictionary with, as keys, the short-cut-names
of the pretrained models, and as associated values, the maximum length of the sequence inputs of this model,
or None if the model has no maximum input size. pretrained_init_configuration (Dict[str, Dict[str, Any]]) — A dictionary with, as keys,


NougatTokenizerFast

the
short-cut-names of the pretrained models, and as associated values, a dictionary of specific arguments to
pass to the __init__ method of the tokenizer class for this pretrained model when loading the tokenizer
with the from_pretrained() method. model_input_names (List[str]) — A list of inputs expected in the forward pass of the model. padding_side (str) — The default value for the side on which the model should have padding applied.
Should be 'right' or 'left'. truncation_side (str) — The default value for the side on which the model should have truncation
applied. Should be 'right' or 'left'.  correct_tables  < source > ( generation: str  ) → str  Parameters   generation (str) — The generated text to be postprocessed.   Returns
str

The postprocessed text.
  Takes a generated string and fixes tables/tabulars to make them match the markdown format needed.  Example:   Copied correct_tables("\beg


NougatTokenizerFast

in{table} \begin{tabular}{l l} & \ \end{tabular} \end{table}")
"\begin{table}
abular}{l l} & \ \end{tabular}
le}"  post_process_generation  < source > ( generation: typing.Union[str, typing.List[str]] fix_markdown: bool = True num_workers: int = None  ) → Union[str, List[str]]  Parameters   generation (Union[str, List[str]]) —
The generated text or a list of generated texts.   fix_markdown (bool, optional, defaults to True) —
Whether to perform Markdown formatting fixes.   num_workers (int, optional) —
Optional number of workers to pass to leverage multiprocessing (postprocessing several texts in
parallel).   Returns
Union[str, List[str]]

The postprocessed text or list of postprocessed texts.
  Postprocess a generated text or a list of generated texts. This function can be used to perform postprocessing on generated text, such as fixing Markdown formatting. Postprocessing is quite slow so it is re


NougatTokenizerFast

commended to use multiprocessing to speed up the process.  post_process_single  < source > ( generation: str fix_markdown: bool = True  ) → str  Parameters   generation (str) — The generated text to be postprocessed.   fix_markdown (bool, optional) — Whether to perform Markdown formatting fixes. Default is True.   Returns
str

The postprocessed text.
  Postprocess a single generated text. Regular expressions used here are taken directly from the Nougat article
authors. These expressions are commented for clarity and tested end-to-end in most cases.  remove_hallucinated_references  < source > ( text: str  ) → str  Parameters   text (str) —
The input text containing references.   Returns
str

The text with hallucinated references removed.
  Remove hallucinated or missing references from the text. This function identifies and removes references that are marked as missing or hallucinated from the in


NougatProcessor

class transformers.NougatProcessor  < source > ( image_processor tokenizer  )   Parameters   image_processor (NougatImageProcessor) —
An instance of NougatImageProcessor. The image processor is a required input.   tokenizer (NougatTokenizerFast) —
An instance of NougatTokenizerFast. The tokenizer is a required input.    Constructs a Nougat processor which wraps a Nougat image processor and a Nougat tokenizer into a single processor. NougatProcessor offers all the functionalities of NougatImageProcessor and NougatTokenizerFast. See the
call() and decode() for more information.  __call__  < source > ( images = None text = None do_crop_margin: bool = None do_resize: bool = None size: typing.Dict[str, int] = None resample: PILImageResampling = None do_thumbnail: bool = None do_align_long_axis: bool = None do_pad: bool = None do_rescale: bool = None resc


NougatProcessor

ale_factor: typing.Union[int, float] = None do_normalize: bool = None image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float], NoneType] = None data_format: typing.Optional[ForwardRef('ChannelDimension')] = 'channels_first' input_data_format: typing.Union[str, ForwardRef('ChannelDimension'), NoneType] = None text_pair: typing.Union[str, typing.List[str], typing.List[typing.List[str]], NoneType] = None text_target: typing.Union[str, typing.List[str], typing.List[typing.List[str]]] = None text_pair_target: typing.Union[str, typing.List[str], typing.List[typing.List[str]], NoneType] = None add_special_tokens: bool = True padding: typing.Union[bool, str, transformers.utils.generic.PaddingStrategy] = False truncation: typing.Union[bool, str, transformers.tokenization_utils_base.TruncationStra


NougatProcessor

tegy] = None max_length: typing.Optional[int] = None stride: int = 0 is_split_into_words: bool = False pad_to_multiple_of: typing.Optional[int] = None return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None return_token_type_ids: typing.Optional[bool] = None return_attention_mask: typing.Optional[bool] = None return_overflowing_tokens: bool = False return_special_tokens_mask: bool = False return_offsets_mapping: bool = False return_length: bool = False verbose: bool = True  )     from_pretrained  < source > ( pretrained_model_name_or_path: typing.Union[str, os.PathLike] cache_dir: typing.Union[str, os.PathLike, NoneType] = None force_download: bool = False local_files_only: bool = False token: typing.Union[bool, str, NoneType] = None revision: str = 'main' **kwargs  )   Parameters   pretrained_model_name_or_path (st


NougatProcessor

r or os.PathLike) —
This can be either:
a string, the model id of a pretrained feature_extractor hosted inside a model repo on
huggingface.co. Valid model ids can be located at the root-level, like bert-base-uncased, or
namespaced under a user or organization name, like dbmdz/bert-base-german-cased.
a path to a directory containing a feature extractor file saved using the
save_pretrained() method, e.g., ./my_model_directory/.
a path or url to a saved feature extractor JSON file, e.g.,
./my_model_directory/preprocessor_config.json.
**kwargs —
Additional keyword arguments passed along to both
from_pretrained() and
~tokenization_utils_base.PreTrainedTokenizer.from_pretrained.
    Instantiate a processor associated with a pretrained model. This class method is simply calling the feature extractor
from_pretrained(), image processor
ImageProcessingMixin a


NougatProcessor

nd the tokenizer
~tokenization_utils_base.PreTrainedTokenizer.from_pretrained methods. Please refer to the docstrings of the
methods above for more information.  save_pretrained  < source > ( save_directory push_to_hub: bool = False **kwargs  )   Parameters   save_directory (str or os.PathLike) —
Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will
be created if it does not exist).   push_to_hub (bool, optional, defaults to False) —
Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the
repository you want to push to with repo_id (will default to the name of save_directory in your
namespace).   kwargs (Dict[str, Any], optional) —
Additional key word arguments passed along to the push_to_hub() method.    Saves the attributes of this processor (feature extra


NougatProcessor

ctor, tokenizer…) in the specified directory so that it
can be reloaded using the from_pretrained() method. This class method is simply calling save_pretrained() and
save_pretrained(). Please refer to the docstrings of the
methods above for more information.  batch_decode  < source > ( *args **kwargs  )    This method forwards all its arguments to NougatTokenizer’s batch_decode(). Please refer
to the docstring of this method for more information.  decode  < source > ( *args **kwargs  )    This method forwards all its arguments to NougatTokenizer’s decode(). Please refer to
the docstring of this method for more information.  post_process_generation  < source > ( *args **kwargs  )    This method forwards all its arguments to NougatTokenizer’s ~PreTrainedTokenizer.post_process_generation.
Please refer to the docstring of this method for more informatio


Overview

SEW-D (Squeezed and Efficient Wav2Vec with Disentangled attention) was proposed in Performance-Efficiency Trade-offs
in Unsupervised Pre-training for Speech Recognition by Felix Wu, Kwangyoun Kim,
Jing Pan, Kyu Han, Kilian Q. Weinberger, Yoav Artzi.
The abstract from the paper is the following:
This paper is a study of performance-efficiency trade-offs in pre-trained models for automatic speech recognition
(ASR). We focus on wav2vec 2.0, and formalize several architecture designs that influence both the model performance
and its efficiency. Putting together all our observations, we introduce SEW (Squeezed and Efficient Wav2vec), a
pre-trained model architecture with signif


Overview

icant improvements along both performance and efficiency dimensions across a
variety of training setups. For example, under the 100h-960h semi-supervised setup on LibriSpeech, SEW achieves a 1.9x
inference speedup compared to wav2vec 2.0, with a 13.5% relative reduction in word error rate. With a similar inference
time, SEW reduces word error rate by 25-50% across different model sizes.
Tips:
SEW-D is a speech model that accepts a float array corresponding to the raw waveform of the speech signal. SEWDForCTC is fine-tuned using connectionist temporal classification (CTC) so the model output has to be decoded
using Wav2Vec2CTCTokenizer.
This model was contributed by anton-l


SEWDConfig

class transformers.SEWDConfig  < source > ( vocab_size = 32 hidden_size = 768 num_hidden_layers = 12 num_attention_heads = 12 intermediate_size = 3072 squeeze_factor = 2 max_position_embeddings = 512 position_buckets = 256 share_att_key = True relative_attention = True pos_att_type = ('p2c', 'c2p') norm_rel_ebd = 'layer_norm' hidden_act = 'gelu_python' hidden_dropout = 0.1 activation_dropout = 0.1 attention_dropout = 0.1 feat_proj_dropout = 0.0 final_dropout = 0.1 initializer_range = 0.02 layer_norm_eps = 1e-07 feature_layer_norm_eps = 1e-05 feat_extract_norm = 'group' feat_extract_activation = 'gelu' conv_dim = (64, 128, 128, 128, 128, 256, 256, 256, 256, 512, 512, 512, 512) conv_stride = (5, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1) conv_kernel = (10, 3, 1, 3, 1, 3, 1, 3, 1, 2, 1, 2, 1) conv_bias = False num_conv_pos_embeddings = 128 num_conv_pos_embedding_groups = 16 apply_spec_augment = True mask_time_prob =


SEWDConfig

0.05 mask_time_length = 10 mask_time_min_masks = 2 mask_feature_prob = 0.0 mask_feature_length = 10 mask_feature_min_masks = 0 ctc_loss_reduction = 'mean' ctc_zero_infinity = False use_weighted_layer_sum = False classifier_proj_size = 256 pad_token_id = 0 bos_token_id = 1 eos_token_id = 2 **kwargs  )   Parameters   vocab_size (int, optional, defaults to 32) —
Vocabulary size of the SEW-D model. Defines the number of different tokens that can be represented by the
inputs_ids passed when calling SEWD.   hidden_size (int, optional, defaults to 768) —
Dimensionality of the encoder layers and the pooler layer.   num_hidden_layers (int, optional, defaults to 12) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 12) —
Number of attention heads for each attention layer in the Transformer encoder.   intermediate_size (int, optional, defaults to 3072) —
Dimensiona


SEWDConfig

lity of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.   squeeze_factor (int, optional, defaults to 2) —
Sequence length downsampling factor after the encoder and upsampling factor after the transformer.   max_position_embeddings (int, optional, defaults to 512) —
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).   position_buckets (int, optional, defaults to 256) —
The maximum size of relative position embeddings.   share_att_key (bool, optional, defaults to True) —
Whether to share attention key with c2p and p2c.   relative_attention (bool, optional, defaults to True) —
Whether to use relative position encoding.   pos_att_type (Tuple[str], optional, defaults to ("p2c", "c2p")) —
The type of relative position attention, it can be a combination of ("p2c", "c2p"), e.g. ("p2c"),
("p2c"


SEWDConfig

, "c2p"), ("p2c", "c2p").   norm_rel_ebd (str, optional, defaults to "layer_norm") —
Whether to use layer norm in relative embedding ("layer_norm" if yes)   hidden_act (str or function, optional, defaults to "gelu_python") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "selu", "gelu_python" and "gelu_new" are supported.   hidden_dropout (float, optional, defaults to 0.1) —
Deprecated. Not used by the model and will be removed in a future version.   activation_dropout (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.   attention_dropout (float, optional, defaults to 0.1) —
The dropout ratio for the attention probabilities.   final_dropout (float, optional, defaults to 0.1) —
The dropout probability for the final projection layer of SEWDForCTC.   initializer_range


SEWDConfig

(float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   layer_norm_eps (float, optional, defaults to 1e-7) —
The epsilon used by the layer normalization layers in the transformer encoder.   feature_layer_norm_eps (float, optional, defaults to 1e-5) —
The epsilon used by the layer normalization after the feature encoder.   feat_extract_norm (str, optional, defaults to "group") —
The norm to be applied to 1D convolutional layers in feature encoder. One of "group" for group
normalization of only the first 1D convolutional layer or "layer" for layer normalization of all 1D
convolutional layers.   feat_proj_dropout (float, optional, defaults to 0.0) —
The dropout probability for output of the feature encoder.   feat_extract_activation (str, optional, defaults to “gelu”) -- The non-linear activation function (function or string) in t


SEWDConfig

he 1D convolutional layers of the feature extractor. If string, “gelu”, “relu”, “selu”and“gelu_new”` are supported.   conv_dim (Tuple[int] or List[int], optional, defaults to (64, 128, 128, 128, 128, 256, 256, 256, 256, 512, 512, 512, 512)) —
A tuple of integers defining the number of input and output channels of each 1D convolutional layer in the
feature encoder. The length of conv_dim defines the number of 1D convolutional layers.   conv_stride (Tuple[int] or List[int], optional, defaults to (5, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1)) —
A tuple of integers defining the stride of each 1D convolutional layer in the feature encoder. The length
of conv_stride defines the number of convolutional layers and has to match the length of conv_dim.   conv_kernel (Tuple[int] or List[int], optional, defaults to (10, 3, 1, 3, 1, 3, 1, 3, 1, 2, 1, 2, 1)) —
A tuple of integers defining the kernel size of each 1D convolution


SEWDConfig

al layer in the feature encoder. The
length of conv_kernel defines the number of convolutional layers and has to match the length of
conv_dim.   conv_bias (bool, optional, defaults to False) —
Whether the 1D convolutional layers have a bias.   num_conv_pos_embeddings (int, optional, defaults to 128) —
Number of convolutional positional embeddings. Defines the kernel size of 1D convolutional positional
embeddings layer.   num_conv_pos_embedding_groups (int, optional, defaults to 16) —
Number of groups of 1D convolutional positional embeddings layer.   apply_spec_augment (bool, optional, defaults to True) —
Whether to apply SpecAugment data augmentation to the outputs of the feature encoder. For reference see
SpecAugment: A Simple Data Augmentation Method for Automatic Speech
Recognition.   mask_time_prob (float, optional, defaults to 0.05) —
Percentage (between 0 and 1) of all feature vectors along the time


SEWDConfig

axis which will be masked. The masking
procecure generates ”mask_time_problen(time_axis)/mask_time_length” independent masks over the axis. If
reasoning from the propability of each feature vector to be chosen as the start of the vector span to be
masked, mask_time_prob should be `prob_vector_startmask_time_length. Note that overlap may decrease the actual percentage of masked vectors. This is only relevant if apply_spec_augment is True`.   mask_time_length (int, optional, defaults to 10) —
Length of vector span along the time axis.   mask_time_min_masks (int, optional, defaults to 2), —
The minimum number of masks of length mask_feature_length generated along the time axis, each time step,
irrespectively of mask_feature_prob. Only relevant if ”mask_time_prob*len(time_axis)/mask_time_length <
mask_time_min_masks”   mask_feature_prob (float, optional, defaults to 0.0) —
Percentage (between 0 and 1) of all fe


SEWDConfig

ature vectors along the feature axis which will be masked. The
masking procecure generates ”mask_feature_problen(feature_axis)/mask_time_length” independent masks over
the axis. If reasoning from the propability of each feature vector to be chosen as the start of the vector
span to be masked, mask_feature_prob should be `prob_vector_startmask_feature_length. Note that overlap may decrease the actual percentage of masked vectors. This is only relevant if apply_spec_augment is
True`.   mask_feature_length (int, optional, defaults to 10) —
Length of vector span along the feature axis.   mask_feature_min_masks (int, optional, defaults to 0), —
The minimum number of masks of length mask_feature_length generated along the feature axis, each time
step, irrespectively of mask_feature_prob. Only relevant if
”mask_feature_prob*len(feature_axis)/mask_feature_length < mask_feature_min_masks”   diversity_loss_weight (in


SEWDConfig

t, optional, defaults to 0.1) —
The weight of the codebook diversity loss component.   ctc_loss_reduction (str, optional, defaults to "sum") —
Specifies the reduction to apply to the output of torch.nn.CTCLoss. Only relevant when training an
instance of SEWDForCTC.   ctc_zero_infinity (bool, optional, defaults to False) —
Whether to zero infinite losses and the associated gradients of torch.nn.CTCLoss. Infinite losses mainly
occur when the inputs are too short to be aligned to the targets. Only relevant when training an instance
of SEWDForCTC.   use_weighted_layer_sum (bool, optional, defaults to False) —
Whether to use a weighted average of layer outputs with learned weights. Only relevant when using an
instance of Wav2Vec2ForSequenceClassification.   classifier_proj_size (int, optional, defaults to 256) —
Dimensionality of the projection before token mean-pooling for classification.    This is the configu


SEWDConfig

ration class to store the configuration of a SEWDModel. It is used to instantiate a SEW-D
model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
defaults will yield a similar configuration to that of the SEW-D
asapp/sew-d-tiny-100k architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import SEWDConfig, SEWDModel
>>> # Initializing a SEW-D asapp/sew-d-tiny-100k style configuration
>>> configuration = SEWDConfig()
>>> # Initializing a model (with random weights) from the asapp/sew-d-tiny-100k style configuration
>>> model = SEWDModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config  to_dict  < source > ( )    Serializes this instance to a Python d


SEWDModel

class transformers.SEWDModel  < source > ( config: SEWDConfig  )   Parameters   config (SEWDConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare SEW-D Model transformer outputting raw hidden-states without any specific head on top.
SEW-D was proposed in Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech
Recognition by Felix Wu, Kwangyoun Kim, Jing Pan, Kyu Han, Kilian Q. Weinberger,
Yoav Artzi. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving etc.). This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forwa


SEWDModel

rd  < source > ( input_values: typing.Optional[torch.Tensor] attention_mask: typing.Optional[torch.Tensor] = None mask_time_indices: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.BaseModelOutput or tuple(torch.FloatTensor)  Parameters   input_values (torch.FloatTensor of shape (batch_size, sequence_length)) —
Float values of input raw speech waveform. Values can be obtained by loading a .flac or .wav audio file
into an array of type List[float] or a numpy.ndarray, e.g. via the soundfile library (pip install soundfile). To prepare the array into input_values, the AutoProcessor should be used for padding and
conversion into a tensor of type torch.FloatTensor. See Wav2Vec2Processor.call() for details.   attention_mask (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing co


SEWDModel

nvolution and attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.BaseModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (SEWDConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidde


SEWDModel

n_size)) — Sequence of hidden-states at the output of the last layer of the model.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The SEWDModel forward method, overrides the __call__ special method. Although the recipe for forward pass


SEWDModel

needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoProcessor, SEWDModel
>>> import torch
>>> from datasets import load_dataset
>>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
>>> dataset = dataset.sort("id")
>>> sampling_rate = dataset.features["audio"].sampling_rate
>>> processor = AutoProcessor.from_pretrained("asapp/sew-d-tiny-100k-ft-ls100h")
>>> model = SEWDModel.from_pretrained("asapp/sew-d-tiny-100k-ft-ls100h")
>>> # audio file is decoded on the fly
>>> inputs = processor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1, 292, 384]


SEWDForCTC

class transformers.SEWDForCTC  < source > ( config target_lang: typing.Optional[str] = None  )   Parameters   config (SEWDConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    SEW-D Model with a language modeling head on top for Connectionist Temporal Classification (CTC).
SEW-D was proposed in Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech
Recognition by Felix Wu, Kwangyoun Kim, Jing Pan, Kyu Han, Kilian Q. Weinberger,
Yoav Artzi. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving etc.). This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matt


SEWDForCTC

er related to general usage and
behavior.  forward  < source > ( input_values: typing.Optional[torch.Tensor] attention_mask: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None labels: typing.Optional[torch.Tensor] = None  ) → transformers.modeling_outputs.CausalLMOutput or tuple(torch.FloatTensor)  Parameters   input_values (torch.FloatTensor of shape (batch_size, sequence_length)) —
Float values of input raw speech waveform. Values can be obtained by loading a .flac or .wav audio file
into an array of type List[float] or a numpy.ndarray, e.g. via the soundfile library (pip install soundfile). To prepare the array into input_values, the AutoProcessor should be used for padding and
conversion into a tensor of type torch.FloatTensor. See Wav2Vec2Processor.call() for details.   attention_mask (torch.LongTensor of shape (batch_size, sequence_l


SEWDForCTC

ength), optional) —
Mask to avoid performing convolution and attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size, target_length), optional) —
Labels for connectionist temporal classification. Note that target_length has to be smaller or equal to
the sequence length of the output logits. Indices are selected in [-100, 0, ..., config.vocab_size - 1].
All labels set to -100 are ignored (masked), the l


SEWDForCTC

oss is only computed for labels in [0, ..., config.vocab_size - 1].   Returns
transformers.modeling_outputs.CausalLMOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.CausalLMOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (SEWDConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Language modeling loss (for next-token prediction).
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of


SEWDForCTC

 each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The SEWDForCTC forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoProcessor, SEWDForCTC
>>> from datasets import load_dataset
>>> imp


SEWDForCTC

ort torch
>>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
>>> dataset = dataset.sort("id")
>>> sampling_rate = dataset.features["audio"].sampling_rate
>>> processor = AutoProcessor.from_pretrained("asapp/sew-d-tiny-100k-ft-ls100h")
>>> model = SEWDForCTC.from_pretrained("asapp/sew-d-tiny-100k-ft-ls100h")
>>> # audio file is decoded on the fly
>>> inputs = processor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_ids = torch.argmax(logits, dim=-1)
>>> # transcribe speech
>>> transcription = processor.batch_decode(predicted_ids)
>>> transcription[0]
'MISTER QUILTER IS THE APOSTIL OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL'
>>> inputs["labels"] = processor(text=dataset[0]["text"], return_tensors="pt").input_ids
>>> # compute loss
>>> loss = model(**inputs).loss
>>> round(loss.item(), 2)
0.


SEWDForSequenceClassification

class transformers.SEWDForSequenceClassification  < source > ( config  )   Parameters   config (SEWDConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    SEWD Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB
Keyword Spotting. SEW-D was proposed in Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech
Recognition by Felix Wu, Kwangyoun Kim, Jing Pan, Kyu Han, Kilian Q. Weinberger,
Yoav Artzi. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving etc.). This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for


SEWDForSequenceClassification

 all matter related to general usage and
behavior.  forward  < source > ( input_values: typing.Optional[torch.Tensor] attention_mask: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None labels: typing.Optional[torch.Tensor] = None  ) → transformers.modeling_outputs.SequenceClassifierOutput or tuple(torch.FloatTensor)  Parameters   input_values (torch.FloatTensor of shape (batch_size, sequence_length)) —
Float values of input raw speech waveform. Values can be obtained by loading a .flac or .wav audio file
into an array of type List[float] or a numpy.ndarray, e.g. via the soundfile library (pip install soundfile). To prepare the array into input_values, the AutoProcessor should be used for padding and
conversion into a tensor of type torch.FloatTensor. See Wav2Vec2Processor.call() for details.   attention_mask (torch.LongTensor of shape (batch_si


SEWDForSequenceClassification

ze, sequence_length), optional) —
Mask to avoid performing convolution and attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-En


SEWDForSequenceClassification

tropy).   Returns
transformers.modeling_outputs.SequenceClassifierOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.SequenceClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (SEWDConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden


SEWDForSequenceClassification

-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The SEWDForSequenceClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoFeatureExtractor, SEWDForSequenceClassification
>>> from datasets import load_dataset
>>> import torch
>>> dataset = load_dat


SEWDForSequenceClassification

aset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
>>> dataset = dataset.sort("id")
>>> sampling_rate = dataset.features["audio"].sampling_rate
>>> feature_extractor = AutoFeatureExtractor.from_pretrained("anton-l/sew-d-mid-400k-ft-keyword-spotting")
>>> model = SEWDForSequenceClassification.from_pretrained("anton-l/sew-d-mid-400k-ft-keyword-spotting")
>>> # audio file is decoded on the fly
>>> inputs = feature_extractor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_class_ids = torch.argmax(logits, dim=-1).item()
>>> predicted_label = model.config.id2label[predicted_class_ids]
>>> predicted_label
'_unknown_'
>>> # compute loss - target_label is e.g. "down"
>>> target_label = model.config.id2label[0]
>>> inputs["labels"] = torch.tensor([model.config.label2id[target_label]])
>>> loss = model(**inputs).loss
>>> round(loss.item(), 2)


Overview

CPM-Ant is an open-source Chinese pre-trained language model (PLM) with 10B parameters. It is also the first milestone of the live training process of CPM-Live. The training process is cost-effective and environment-friendly. CPM-Ant also achieves promising results with delta tuning on the CUGE benchmark. Besides the full model, we also provide various compressed versions to meet the requirements of different hardware configurations. See more
Tips:
This model was contributed by OpenBMB. The original code can be found here.
⚙ Training & Inference
A tutorial on CPM-Live.


CpmAntConfig

class transformers.CpmAntConfig  < source > ( vocab_size: int = 30720 hidden_size: int = 4096 num_attention_heads: int = 32 dim_head: int = 128 dim_ff: int = 10240 num_hidden_layers: int = 48 dropout_p: int = 0.0 position_bias_num_buckets: int = 512 position_bias_max_distance: int = 2048 eps: int = 1e-06 init_std: float = 1.0 prompt_types: int = 32 prompt_length: int = 32 segment_types: int = 32 use_cache: bool = True **kwargs  )   Parameters   vocab_size (int, optional, defaults to 30720) —
Vocabulary size of the CPMAnt model. Defines the number of different tokens that can be represented by the
input passed when calling CpmAntModel.   hidden_size (int, optional, defaults to 4096) —
Dimension of the encoder layers.   num_attention_heads (int, optional, defaults to 32) —
Number of attention heads in the Transformer encoder.   dim_head (int, optional, defaults to 128) —
Dimension of attention heads for each attention layer in the Transformer encoder.   dim_ff (int, optiona


CpmAntConfig

l, defaults to 10240) —
Dimension of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.   num_hidden_layers (int, optional, defaults to 48) —
Number of layers of the Transformer encoder.   dropout_p (float, optional, defaults to 0.0) —
The dropout probabilitiy for all fully connected layers in the embeddings, encoder.   position_bias_num_buckets (int, optional, defaults to 512) —
The number of position_bias buckets.   position_bias_max_distance (int, optional, defaults to 2048) —
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).   eps (float, optional, defaults to 1e-06) —
The epsilon used by the layer normalization layers.   init_std (float, optional, defaults to 1.0) —
Initialize parameters with std = init_std.   prompt_types (int, optional, defaults to 32) —
The type of prompt.   prompt_length (int, optional, defaults to 32) —
The length of prompt.


CpmAntConfig

  segment_types (int, optional, defaults to 32) —
The type of segment.   use_cache (bool, optional, defaults to True) —
Whether to use cache.    This is the configuration class to store the configuration of a CpmAntModel. It is used to instantiate an
CPMAnt model according to the specified arguments, defining the model architecture. Instantiating a configuration
with the defaults will yield a similar configuration to that of the CPMAnt
openbmb/cpm-ant-10b architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import CpmAntModel, CpmAntConfig
>>> # Initializing a CPMAnt cpm-ant-10b style configuration
>>> configuration = CpmAntConfig()
>>> # Initializing a model from the cpm-ant-10b style configuration
>>> model = CpmAntModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.confi


CpmAntTokenizer

class transformers.CpmAntTokenizer  < source > ( vocab_file bod_token = '<d>' eod_token = '</d>' bos_token = '<s>' eos_token = '</s>' pad_token = '<pad>' unk_token = '<unk>' line_token = '</n>' space_token = '</_>' padding_side = 'left' **kwargs  )   Parameters   vocab_file (str) —
Path to the vocabulary file.   bod_token (str, optional, defaults to "<d>") —
The beginning of document token.   eod_token (str, optional, defaults to "</d>") —
The end of document token.   bos_token (str, optional, defaults to "<s>") —
The beginning of sequence token.   eos_token (str, optional, defaults to "</s>") —
The end of sequence token.   pad_token (str, optional, defaults to "<pad>") —
The token used for padding.   unk_token (str, optional, defaults to "<unk>") —
The unknown token.   li


CpmAntTokenizer

ne_token (str, optional, defaults to "</n>") —
The line token.   space_token (str, optional, defaults to "</_>") —
The space token.    Construct a CPMAnt tokenizer. Based on byte-level Byte-Pair-Encoding.  build_inputs_with_special_tokens  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.List[int] = None  ) → List[int]  Parameters   token_ids_0 (List[int]) — The first tokenized sequence that special tokens will be added.   token_ids_1 (List[int]) — The optional second tokenized sequence that special tokens will be added.   Returns
List[int]

The model input with special tokens.
  Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A CPMAnt sequence has the following format:


CpmAntTokenizer

single sequence: [BOS] Sequence.  get_special_tokens_mask  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None already_has_special_tokens: bool = False  ) → List[int]  Parameters   token_ids_0 (List[int]) — List of IDs.   token_ids_1 (List[int], optional) — Optional second list of IDs for sequence pairs.   already_has_special_tokens (bool, optional, defaults to False) —
Whether or not the token list is already formatted with special tokens for the model.   Returns
List[int]

A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
  Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer prepare_for_model metho


CpmAntModel

class transformers.CpmAntModel  < source > ( config: CpmAntConfig  )    The bare CPMAnt Model outputting raw hidden-states without any specific head on top.
This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior. Parameters
config (~CpmAntConfig): Model configuration class with all the parameters of the
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None past_key_values: typing.Optional[typing.Tuple[typing.Tuple[torch.Tensor]]] = None use_cache: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None **kwargs  ) → transformers.modeling_outputs


CpmAntModel

.BaseModelOutputWithPast or tuple(torch.FloatTensor)  Parameters   input_ids (torch.Tensor of shape (batch_size, seq_len)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using CPMAntTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) —
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers.   output_hidden_states (bool, optional) —
Whether or not to return


CpmAntModel

 the hidden states of all layers.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   Returns
transformers.modeling_outputs.BaseModelOutputWithPast or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutputWithPast or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (CpmAntConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
If past_key_values is used only the last hidden-state of the sequences of shape (batch_size, 1, hidden_size) is output.
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of


CpmAntModel

 shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and optionally if
config.is_encoder_decoder=True 2 additional tensors of shape (batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if
config.is_encoder_decoder=True in the cross-attention blocks) that can be used (see past_key_values
input) to speed up sequential decoding.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True


CpmAntModel

 is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The CpmAntModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, CpmAntModel
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("openbmb/cpm-ant-10b")
>>> model = CpmAntModel.from_pretrained("openbmb/cpm-ant-10b")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_st


CpmAntForCausalLM

class transformers.CpmAntForCausalLM  < source > ( config: CpmAntConfig  )    The CPMAnt Model with a language modeling head on top (linear layer with weights tied to the input embeddings). This model is a PyTorch torch.nn.Module sub-class. Use
it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior. Parameters
config (~CpmAntConfig): Model configuration class with all the parameters of the
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.  forward  < source > ( input_ids: typing.Optional[torch.Tensor] = None past_key_values: typing.Union[typing.List[typing.Tuple[torch.Tensor, torch.Tensor]], NoneType] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output


CpmAntForCausalLM

_hidden_states: typing.Optional[bool] = None labels: typing.Optional[torch.Tensor] = None return_dict: typing.Optional[bool] = None attention_mask: typing.Optional[torch.Tensor] = None **kwargs  ) → transformers.modeling_outputs.CausalLMOutputWithPast or tuple(torch.FloatTensor)  Parameters   input_ids (torch.Tensor of shape (batch_size, seq_len)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using CPMAntTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) —
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.   use_cache (bool, optional


CpmAntForCausalLM

) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.
Args —
input_ids (torch.Tensor of shape (batch_size, seq_len)):
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using CPMAntTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True):
Contains pre-computed hidden-states (key and values in the self-attention bl


CpmAntForCausalLM

ocks and in the
cross-attention blocks) that can be used (see past_key_values input) to speed up sequential decoding.
use_cache (bool, optional):
If set to True, past_key_values key value states are returned and can be used to speed up decoding
(see past_key_values).
output_attentions (bool, optional):
Whether or not to return the attentions tensors of all attention layers.
output_hidden_states (bool, optional):
Whether or not to return the hidden states of all layers.
labels (torch.Tensor of shape (batch_size, sequence_length), optional):
Labels for computing the masked language modeling loss.
return_dict (bool, optional):
Whether or not to return a ModelOutput instead of a plain tuple.
attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional):
CPMAnt will process attention mask automatically, this parameter is a dummy parameter for
text-generation pipeline.
Example —


CpmAntForCausalLM

  Text Generation with CpmAntForCausalLM. —   Returns
transformers.modeling_outputs.CausalLMOutputWithPast or tuple(torch.FloatTensor)

A transformers.modeling_outputs.CausalLMOutputWithPast or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (CpmAntConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Language modeling loss (for next-token prediction).
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each


CpmAntForCausalLM

 tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head))
Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see
past_key_values input) to speed up sequential decoding.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_


CpmAntForCausalLM

heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The CpmAntForCausalLM forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> import torch
>>> from transformers import AutoTokenizer, CpmAntForCausalLM
>>> tokenizer = AutoTokenizer.from_pretrained("openbmb/cpm-ant-10b")
>>> model = CpmAntForCausalLM.from_pretrained("openbmb/cpm-ant-10b")
>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
>>> outputs = model(**inputs, labels=inputs["input_ids"])
>>> loss = outputs.loss
>>> logits = outputs.logit


Overview

The ViTMAE model was proposed in Masked Autoencoders Are Scalable Vision Learners by Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li,
Piotr Dollár, Ross Girshick. The paper shows that, by pre-training a Vision Transformer (ViT) to reconstruct pixel values for masked patches, one can get results after
fine-tuning that outperform supervised pre-training.
The abstract from the paper is the following:
This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the
input image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates
only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask
tokens. Second, we find that masking a high pr


Overview

oportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task. Coupling these two designs
enables us to train large models efficiently and effectively: we accelerate training (by 3x or more) and improve accuracy. Our scalable approach allows for learning high-capacity
models that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data. Transfer performance in downstream
tasks outperforms supervised pre-training and shows promising scaling behavior.
Tips:
MAE (masked auto encoding) is a method for self-supervised pre-training of Vision Transformers (ViTs). The pre-training objective is relatively simple:
by masking a large portion (75%) of the image patches, the model must reconstruct raw pixel values. One can use ViTMAEForPreTraining for this purpose. After pre-training, one “throws away” the decoder used to reconstruct pixels, and one us


Overview

es the encoder for fine-tuning/linear probing. This means that after
fine-tuning, one can directly plug in the weights into a ViTForImageClassification. One can use ViTImageProcessor to prepare images for the model. See the code examples for more info. Note that the encoder of MAE is only used to encode the visual patches. The encoded patches are then concatenated with mask tokens, which the decoder (which also
consists of Transformer blocks) takes as input. Each mask token is a shared, learned vector that indicates the presence of a missing patch to be predicted. Fixed
sin/cos position embeddings are added both to the input of the encoder and the decoder. For a visual understanding of how MAEs work you can check out this post.
MAE architecture. Taken from the original paper.
This model was contributed by nielsr. TensorFlow version of the model was contributed by sayakpaul and
ariG23498 (equal contribution). The original code can be found here


Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ViTMAE.
ViTMAEForPreTraining is supported by this example script, allowing you to pre-train the model from scratch/further pre-train the model on custom data. A notebook that illustrates how to visualize reconstructed pixel values with ViTMAEForPreTraining can be found here.
If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we’ll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


ViTMAEConfig

class transformers.ViTMAEConfig  < source > ( hidden_size = 768 num_hidden_layers = 12 num_attention_heads = 12 intermediate_size = 3072 hidden_act = 'gelu' hidden_dropout_prob = 0.0 attention_probs_dropout_prob = 0.0 initializer_range = 0.02 layer_norm_eps = 1e-12 image_size = 224 patch_size = 16 num_channels = 3 qkv_bias = True decoder_num_attention_heads = 16 decoder_hidden_size = 512 decoder_num_hidden_layers = 8 decoder_intermediate_size = 2048 mask_ratio = 0.75 norm_pix_loss = False **kwargs  )   Parameters   hidden_size (int, optional, defaults to 768) —
Dimensionality of the encoder layers and the pooler layer.   num_hidden_layers (int, optional, defaults to 12) —
Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 12) —
Number of attention heads for each attention layer in the Transformer encoder.   intermediate_size (int, optional, defaults to 3072) —
Dimensionality of the


ViTMAEConfig

 “intermediate” (i.e., feed-forward) layer in the Transformer encoder.   hidden_act (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "selu" and "gelu_new" are supported.   hidden_dropout_prob (float, optional, defaults to 0.0) —
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.   attention_probs_dropout_prob (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   layer_norm_eps (float, optional, defaults to 1e-12) —
The epsilon used by the layer normalization layers.   image_size (int, optional, defaults to 224) —
The size (resolution) of each image.   patch_size (int, optional, defaults to 1


ViTMAEConfig

6) —
The size (resolution) of each patch.   num_channels (int, optional, defaults to 3) —
The number of input channels.   qkv_bias (bool, optional, defaults to True) —
Whether to add a bias to the queries, keys and values.   decoder_num_attention_heads (int, optional, defaults to 16) —
Number of attention heads for each attention layer in the decoder.   decoder_hidden_size (int, optional, defaults to 512) —
Dimensionality of the decoder.   decoder_num_hidden_layers (int, optional, defaults to 8) —
Number of hidden layers in the decoder.   decoder_intermediate_size (int, optional, defaults to 2048) —
Dimensionality of the “intermediate” (i.e., feed-forward) layer in the decoder.   mask_ratio (float, optional, defaults to 0.75) —
The ratio of the number of masked tokens in the input sequence.   norm_pix_loss (bool, optional, defaults to False) —
Whether or not to train with normalized pixels (see Table 3 in the paper). Using normaliz


ViTMAEConfig

ed pixels improved
representation quality in the experiments of the authors.    This is the configuration class to store the configuration of a ViTMAEModel. It is used to instantiate an ViT
MAE model according to the specified arguments, defining the model architecture. Instantiating a configuration with
the defaults will yield a similar configuration to that of the ViT
facebook/vit-mae-base architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import ViTMAEConfig, ViTMAEModel
>>> # Initializing a ViT MAE vit-mae-base style configuration
>>> configuration = ViTMAEConfig()
>>> # Initializing a model (with random weights) from the vit-mae-base style configuration
>>> model = ViTMAEModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.conf