NezhaForTokenClassification

 (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length


NezhaForTokenClassification

, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Labels for computing the token classification loss. Indices should be in [0, ..., config.num


NezhaForTokenClassification

_labels - 1].   Returns
transformers.modeling_outputs.TokenClassifierOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.TokenClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (NezhaConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided)  — Classification loss.
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.num_labels)) — Classification scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (bat


NezhaForTokenClassification

ch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The NezhaForTokenClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> fro


NezhaForTokenClassification

m transformers import AutoTokenizer, NezhaForTokenClassification
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("sijunhe/nezha-cn-base")
>>> model = NezhaForTokenClassification.from_pretrained("sijunhe/nezha-cn-base")
>>> inputs = tokenizer(
...     "HuggingFace is a company based in Paris and New York", add_special_tokens=False, return_tensors="pt"
... )
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> predicted_token_class_ids = logits.argmax(-1)
>>> # Note that tokens are classified rather then input words which means that
>>> # there might be more predicted token classes than words.
>>> # Multiple token classes might account for the same word
>>> predicted_tokens_classes = [model.config.id2label[t.item()] for t in predicted_token_class_ids[0]]
>>> labels = predicted_token_class_ids
>>> loss = model(**inputs, labels=labels).


NezhaForQuestionAnswering

class transformers.NezhaForQuestionAnswering  < source > ( config  )   Parameters   config (NezhaConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    Nezha Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear
layers on top of the hidden-states output to compute span start logits and span end logits). This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source >


NezhaForQuestionAnswering

( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None start_positions: typing.Optional[torch.Tensor] = None end_positions: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.QuestionAnsweringModelOutput or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performi


NezhaForQuestionAnswering

ng attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?   token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:
0 corresponds to a sentence A token,
1 corresponds to a sentence B token.
What are token type IDs?   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids in


NezhaForQuestionAnswering

dices into associated vectors than the
model’s internal embedding lookup matrix.   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   start_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the start of the labelled span for computing the token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
are not taken into account for computing the loss.   end_positions (torch.LongTensor of shape (batch_size,), optional) —
Labels for position (index) of the end of the labelled span for computing t


NezhaForQuestionAnswering

he token classification loss.
Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence
are not taken into account for computing the loss.   Returns
transformers.modeling_outputs.QuestionAnsweringModelOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.QuestionAnsweringModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (NezhaConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Total span extraction loss is the sum of a Cross-Entropy for the start and end positions.
start_logits (torch.FloatTensor of shape (batch_size, sequence_length)) — Span-start scores (before SoftMax).
end_logits (torch.FloatTensor of shape (batch_size, sequence_length)) — Span-end scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned wh


NezhaForQuestionAnswering

en output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The NezhaForQuestionAnswering forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the for


NezhaForQuestionAnswering

mer takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, NezhaForQuestionAnswering
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("sijunhe/nezha-cn-base")
>>> model = NezhaForQuestionAnswering.from_pretrained("sijunhe/nezha-cn-base")
>>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
>>> inputs = tokenizer(question, text, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> answer_start_index = outputs.start_logits.argmax()
>>> answer_end_index = outputs.end_logits.argmax()
>>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
>>> # target is "nice puppet"
>>> target_start_index = torch.tensor([14])
>>> target_end_index = torch.tensor([15])
>>> outputs = model(**inputs, start_positions=target_start_index, end_positions=target_end_index)
>>> loss = outputs.lo


Overview

The Graphormer model was proposed in Do Transformers Really Perform Bad for Graph Representation?  by
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen and Tie-Yan Liu. It is a Graph Transformer model, modified to allow computations on graphs instead of text sequences by generating embeddings and features of interest during preprocessing and collation, then using a modified attention.
The abstract from the paper is the following:
The Transformer architecture has become a dominant choice in many domains, such as natural language processing and computer vision. Yet, it has not achieved competitive performance on popular leaderboards of graph-level pred


Overview

iction compared to mainstream GNN variants. Therefore, it remains a mystery how Transformers could perform well for graph representation learning. In this paper, we solve this mystery by presenting Graphormer, which is built upon the standard Transformer architecture, and could attain excellent results on a broad range of graph representation learning tasks, especially on the recent OGB Large-Scale Challenge. Our key insight to utilizing Transformer in the graph is the necessity of effectively encoding the structural information of a graph into the model. To this end, we propose several simple yet effective structural encoding methods to help Graphormer better model graph-structured dat


Overview

a. Besides, we mathematically characterize the expressive power of Graphormer and exhibit that with our ways of encoding the structural information of graphs, many popular GNN variants could be covered as the special cases of Graphormer.
Tips:
This model will not work well on large graphs (more than 100 nodes/edges), as it will make the memory explode.
You can reduce the batch size, increase your RAM, or decrease the UNREACHABLE_NODE_DISTANCE parameter in algos_graphormer.pyx, but it will be hard to go above 700 nodes/edges.
This model does not use a tokenizer, but instead a special collator during training.
This model was contributed by clefourrier. The original code can be found here.


GraphormerConfig

class transformers.GraphormerConfig  < source > ( num_classes: int = 1 num_atoms: int = 4608 num_edges: int = 1536 num_in_degree: int = 512 num_out_degree: int = 512 num_spatial: int = 512 num_edge_dis: int = 128 multi_hop_max_dist: int = 5 spatial_pos_max: int = 1024 edge_type: str = 'multi_hop' max_nodes: int = 512 share_input_output_embed: bool = False num_hidden_layers: int = 12 embedding_dim: int = 768 ffn_embedding_dim: int = 768 num_attention_heads: int = 32 dropout: float = 0.1 attention_dropout: float = 0.1 activation_dropout: float = 0.1 layerdrop: float = 0.0 encoder_normalize_before: bool = False pre_layernorm: bool = False apply_graphormer_init: bool = False activation_fn: str = 'gelu' embed_scale: float = None freeze_embeddings: bool = False num_trans_layers_to_freeze: int = 0 traceable: bool = False q_noise: float = 0.0 qn_block_size: int = 8 kdim: int = None vdim: int = None bias: bool = True self_attention: bool = True pad_token_id = 0


GraphormerConfig

bos_token_id = 1 eos_token_id = 2 **kwargs  )   Parameters   num_classes (int, optional, defaults to 1) —
Number of target classes or labels, set to n for binary classification of n tasks.   num_atoms (int, optional, defaults to 512*9) —
Number of node types in the graphs.   num_edges (int, optional, defaults to 512*3) —
Number of edges types in the graph.   num_in_degree (int, optional, defaults to 512) —
Number of in degrees types in the input graphs.   num_out_degree (int, optional, defaults to 512) —
Number of out degrees types in the input graphs.   num_edge_dis (int, optional, defaults to 128) —
Number of edge dis in the input graphs.   multi_hop_max_dist (int, optional, defaults to 20) —
Maximum distance of multi hop edges between two nodes.   spatial_pos_max (int, optional, defaults to 1024) —
Maximum distance between nodes in the graph attention bias matrices, used during preprocessing and
collation.   edge_type (str, optional, defaults to mult


GraphormerConfig

ihop) —
Type of edge relation chosen.   max_nodes (int, optional, defaults to 512) —
Maximum number of nodes which can be parsed for the input graphs.   share_input_output_embed (bool, optional, defaults to False) —
Shares the embedding layer between encoder and decoder - careful, True is not implemented.   num_layers (int, optional, defaults to 12) —
Number of layers.   embedding_dim (int, optional, defaults to 768) —
Dimension of the embedding layer in encoder.   ffn_embedding_dim (int, optional, defaults to 768) —
Dimension of the “intermediate” (often named feed-forward) layer in encoder.   num_attention_heads (int, optional, defaults to 32) —
Number of attention heads in the encoder.   self_attention (bool, optional, defaults to True) —
Model is self attentive (False not implemented).   activation_function (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string,


GraphormerConfig

"gelu",
"relu", "silu" and "gelu_new" are supported.   dropout (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.   attention_dropout (float, optional, defaults to 0.1) —
The dropout probability for the attention weights.   activation_dropout (float, optional, defaults to 0.1) —
The dropout probability for the activation of the linear transformer layer.   layerdrop (float, optional, defaults to 0.0) —
The LayerDrop probability for the encoder. See the [LayerDrop paper](see https://arxiv.org/abs/1909.11556)
for more details.   bias (bool, optional, defaults to True) —
Uses bias in the attention module - unsupported at the moment.   embed_scale(float, optional, defaults to None) —
Scaling factor for the node embeddings.   num_trans_layers_to_freeze (int, optional, defaults to 0) —
Number of transformer layers to freeze.   encoder_normalize_before (bool, optional, defaults to


GraphormerConfig

False) —
Normalize features before encoding the graph.   pre_layernorm (bool, optional, defaults to False) —
Apply layernorm before self attention and the feed forward network. Without this, post layernorm will be
used.   apply_graphormer_init (bool, optional, defaults to False) —
Apply a custom graphormer initialisation to the model before training.   freeze_embeddings (bool, optional, defaults to False) —
Freeze the embedding layer, or train it along the model.   encoder_normalize_before (bool, optional, defaults to False) —
Apply the layer norm before each encoder block.   q_noise (float, optional, defaults to 0.0) —
Amount of quantization noise (see “Training with Quantization Noise for Extreme Model Compression”). (For
more detail, see fairseq’s documentation on quant_noise).   qn_block_size (int, optional, defaults to 8) —
Size of the blocks for subsequent quantization with iPQ (see q_noise).   kdim (int, optional, defaults to None) —
Dimension of


GraphormerConfig

 the key in the attention, if different from the other values.   vdim (int, optional, defaults to None) —
Dimension of the value in the attention, if different from the other values.   use_cache (bool, optional, defaults to True) —
Whether or not the model should return the last key/values attentions (not used by all models).   traceable (bool, optional, defaults to False) —
Changes return value of the encoder’s inner_state to stacked tensors.
Example —    This is the configuration class to store the configuration of a ~GraphormerModel. It is used to instantiate an
Graphormer model according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the Graphormer
graphormer-base-pcqm4mv1 architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more informa


GraphormerModel

class transformers.GraphormerModel  < source > ( config: GraphormerConfig  )    The Graphormer model is a graph-encoder model. It goes from a graph to its representation. If you want to use the model for a downstream classification task, use
GraphormerForGraphClassification instead. For any other downstream task, feel free to add a new class, or combine
this model with a downstream model of your choice, following the example in GraphormerForGraphClassification.  forward  < source > ( input_nodes: LongTensor input_edges: LongTensor attn_bias: Tensor in_degree: LongTensor out_degree: LongTensor spatial_pos: LongTensor attn_edge_type: LongTensor perturb: typing.Optional[torch.FloatTensor] = None masked_tokens: None = None return_dict: typing.Optional[bool] = None **unused  )


GraphormerForGraphClassification

class transformers.GraphormerForGraphClassification  < source > ( config: GraphormerConfig  )    This model can be used for graph-level classification or regression tasks. It can be trained on regression (by setting config.num_classes to 1); there should be one float-type label per graph one task classification (by setting config.num_classes to the number of classes); there should be one integer
label per graph binary multi-task classification (by setting config.num_classes to the number of labels); there should be a list
of integer labels for each graph.  forward  < source > ( input_nodes: LongTensor input_edges: LongTensor attn_bias: Tensor in_degree: LongTensor out_degree: LongTensor spatial_pos: LongTensor attn_edge_type: LongTensor labels: typing.Optional[torch.LongTensor] = None return_dict: typing.Optional[bool] = None **unused  )


Overview

The hybrid Vision Transformer (ViT) model was proposed in An Image is Worth 16x16 Words: Transformers for Image Recognition
at Scale by Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk
Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob
Uszkoreit, Neil Houlsby. It’s the first paper that successfully trains a Transformer encoder on ImageNet, attaining
very good results compared to familiar convolutional architectures. ViT hybrid is a slight variant of the plain Vision Transformer,
by leveraging a convolutional backbone (specifically, BiT) whose features are used as initial “tokens” for the Transformer.
The abstract from the paper is the following:
While the Transformer architecture has become the de-facto standard for natural language processing tasks,


Overview

 its
applications to computer vision remain limited. In vision, attention is either applied in conjunction with
convolutional networks, or used to replace certain components of convolutional networks while keeping their overall
structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to
sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of
data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.),
Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring
substantially fewer computational resources to train.
This model was contributed by nielsr. The original code (written in JAX) can be
found here.


Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ViT Hybrid.
 Image Classification
ViTHybridForImageClassification is supported by this example script and notebook. See also: Image classification task guide
If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we’ll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


ViTHybridConfig

class transformers.ViTHybridConfig  < source > ( backbone_config = None hidden_size = 768 num_hidden_layers = 12 num_attention_heads = 12 intermediate_size = 3072 hidden_act = 'gelu' hidden_dropout_prob = 0.0 attention_probs_dropout_prob = 0.0 initializer_range = 0.02 layer_norm_eps = 1e-12 image_size = 224 patch_size = 1 num_channels = 3 backbone_featmap_shape = [1, 1024, 24, 24] qkv_bias = True **kwargs  )   Parameters   backbone_config (Union[Dict[str, Any], PretrainedConfig], optional) —
The configuration of the backbone in a dictionary or the config object of the backbone.   hidden_size (int, optional, defaults to 768) —
Dimensionality of the encoder layers and the pooler layer.   num_hidden_layers (int, optional, defaults to 12) —
Number of hidden layers in the Transformer encoder.   num_attention_hea


ViTHybridConfig

ds (int, optional, defaults to 12) —
Number of attention heads for each attention layer in the Transformer encoder.   intermediate_size (int, optional, defaults to 3072) —
Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.   hidden_act (str or function, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "selu" and "gelu_new" are supported.   hidden_dropout_prob (float, optional, defaults to 0.0) —
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.   attention_probs_dropout_prob (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.   initializer_range (float, optional, defaults to 0.02) —
The standard


ViTHybridConfig

deviation of the truncated_normal_initializer for initializing all weight matrices.   layer_norm_eps (float, optional, defaults to 1e-12) —
The epsilon used by the layer normalization layers.   image_size (int, optional, defaults to 224) —
The size (resolution) of each image.   patch_size (int, optional, defaults to 1) —
The size (resolution) of each patch.   num_channels (int, optional, defaults to 3) —
The number of input channels.   backbone_featmap_shape (List[int], optional, defaults to [1, 1024, 24, 24]) —
Used only for the hybrid embedding type. The shape of the feature maps of the backbone.   qkv_bias (bool, optional, defaults to True) —
Whether to add a bias to the queries, keys and values.    This is the configuration class to store the configuration of a ViTHybridModel. It is used to instantiate


ViTHybridConfig

a ViT
Hybrid model according to the specified arguments, defining the model architecture. Instantiating a configuration
with the defaults will yield a similar configuration to that of the ViT Hybrid
google/vit-hybrid-base-bit-384 architecture. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Example:   Copied >>> from transformers import ViTHybridConfig, ViTHybridModel
>>> # Initializing a ViT Hybrid vit-hybrid-base-bit-384 style configuration
>>> configuration = ViTHybridConfig()
>>> # Initializing a model (with random weights) from the vit-hybrid-base-bit-384 style configuration
>>> model = ViTHybridModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.confi


ViTHybridImageProcessor

class transformers.ViTHybridImageProcessor  < source > ( do_resize: bool = True size: typing.Dict[str, int] = None resample: Resampling = <Resampling.BICUBIC: 3> do_center_crop: bool = True crop_size: typing.Dict[str, int] = None do_rescale: bool = True rescale_factor: typing.Union[int, float] = 0.00392156862745098 do_normalize: bool = True image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float], NoneType] = None do_convert_rgb: bool = True **kwargs  )   Parameters   do_resize (bool, optional, defaults to True) —
Whether to resize the image’s (height, width) dimensions to the specified size. Can be overridden by
do_resize in the preprocess method.   size (Dict[str, int] optional, defaults to {"shortest_edge" -- 224}):
Size of the image after resizing. The shortest edge of the image is resized to size[“shortest_edge”], with
the longest edge resized to keep the input aspect ratio.


ViTHybridImageProcessor

 Can be overridden by size in the preprocess
method.   resample (PILImageResampling, optional, defaults to PILImageResampling.BICUBIC) —
Resampling filter to use if resizing the image. Can be overridden by resample in the preprocess method.   do_center_crop (bool, optional, defaults to True) —
Whether to center crop the image to the specified crop_size. Can be overridden by do_center_crop in the
preprocess method.   crop_size (Dict[str, int] optional, defaults to 224) —
Size of the output image after applying center_crop. Can be overridden by crop_size in the preprocess
method.   do_rescale (bool, optional, defaults to True) —
Whether to rescale the image by the specified scale rescale_factor. Can be overridden by do_rescale in
the preprocess method.   rescale_factor (int or float, optional, defaults to 1/255) —
Scale factor to use if rescaling the image. Can be overridden by rescale_factor in the preprocess
method.
do_normalize —
Whether t


ViTHybridImageProcessor

o normalize the image. Can be overridden by do_normalize in the preprocess method.   image_mean (float or List[float], optional, defaults to IMAGENET_STANDARD_MEAN) —
Mean to use if normalizing the image. This is a float or list of floats the length of the number of
channels in the image. Can be overridden by the image_mean parameter in the preprocess method.   image_std (float or List[float], optional, defaults to IMAGENET_STANDARD_STD) —
Standard deviation to use if normalizing the image. This is a float or list of floats the length of the
number of channels in the image. Can be overridden by the image_std parameter in the preprocess method.
Can be overridden by the image_std parameter in the preprocess method.   do_convert_rgb (bool, optional, defaults to True) —
Whether to convert the image to RGB.    Constructs a ViT Hybrid image processor.  preprocess  < source > ( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, For


ViTHybridImageProcessor

wardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]] do_resize: bool = None size: typing.Dict[str, int] = None resample: Resampling = None do_center_crop: bool = None crop_size: int = None do_rescale: bool = None rescale_factor: float = None do_normalize: bool = None image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float], NoneType] = None do_convert_rgb: bool = None return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None data_format: typing.Optional[transformers.image_utils.ChannelDimension] = <ChannelDimension.FIRST: 'channels_first'> input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None **kwargs  )   Parameters   images (ImageInput) —
Image to preprocess. Expects a single or batch of images with pixel values ranging


ViTHybridImageProcessor

from 0 to 255. If
passing in images with pixel values between 0 and 1, set do_rescale=False.   do_resize (bool, optional, defaults to self.do_resize) —
Whether to resize the image.   size (Dict[str, int], optional, defaults to self.size) —
Size of the image after resizing. Shortest edge of the image is resized to size[“shortest_edge”], with
the longest edge resized to keep the input aspect ratio.   resample (int, optional, defaults to self.resample) —
Resampling filter to use if resizing the image. This can be one of the enum PILImageResampling. Only
has an effect if do_resize is set to True.   do_center_crop (bool, optional, defaults to self.do_center_crop) —
Whether to center crop the image.   crop_size (Dict[str, int], optional, defaults to self.crop_size) —
Size of the center crop. Only has an effect if do_center_crop is set to True.   do_rescale (bool, optional, defaults to self.do_rescale) —
Whether to rescale the image.   rescale_fac


ViTHybridImageProcessor

tor (float, optional, defaults to self.rescale_factor) —
Rescale factor to rescale the image by if do_rescale is set to True.   do_normalize (bool, optional, defaults to self.do_normalize) —
Whether to normalize the image.   image_mean (float or List[float], optional, defaults to self.image_mean) —
Image mean to use for normalization. Only has an effect if do_normalize is set to True.   image_std (float or List[float], optional, defaults to self.image_std) —
Image standard deviation to use for normalization. Only has an effect if do_normalize is set to
True.   do_convert_rgb (bool, optional, defaults to self.do_convert_rgb) —
Whether to convert the image to RGB.   return_tensors (str or TensorType, optional) —
The type of tensors to return. Can be one of:
Unset: Return a list of np.ndarray.
TensorType.TENSORFLOW or 'tf': Return a batch of type tf.Tensor.
TensorType.PYTORCH or 'pt': Return a batch of type torch.Tensor.
TensorType.NUMPY or 'n


ViTHybridImageProcessor

p': Return a batch of type np.ndarray.
TensorType.JAX or 'jax': Return a batch of type jax.numpy.ndarray.
   data_format (ChannelDimension or str, optional, defaults to ChannelDimension.FIRST) —
The channel dimension format for the output image. Can be one of:
ChannelDimension.FIRST: image in (num_channels, height, width) format.
ChannelDimension.LAST: image in (height, width, num_channels) format.
Unset: defaults to the channel dimension format of the input image.
   input_data_format (ChannelDimension or str, optional) —
The channel dimension format for the input image. If unset, the channel dimension format is inferred
from the input image. Can be one of:
"channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
"channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
"none" or ChannelDimension.NONE: image in (height, width) format.
    Preprocess an image or batch of image


ViTHybridModel

class transformers.ViTHybridModel  < source > ( config: ViTHybridConfig add_pooling_layer: bool = True use_mask_token: bool = False  )   Parameters   config (ViTHybridConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    The bare ViT Hybrid Model transformer outputting raw hidden-states without any specific head on top.
This model is a PyTorch torch.nn.Module subclass. Use it
as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: typing.Optional[torch.Tensor] = None bool_masked_pos: typing.Optional[torch.BoolTensor] = None head_mask: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None interpolate_pos_enco


ViTHybridModel

ding: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.BaseModelOutputWithPooling or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
ViTHybridImageProcessor.call() for details.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —


ViTHybridModel

Whether or not to return a ModelOutput instead of a plain tuple.   bool_masked_pos (torch.BoolTensor of shape (batch_size, num_patches), optional) —
Boolean masked positions. Indicates which patches are masked (1) and which aren’t (0).   Returns
transformers.modeling_outputs.BaseModelOutputWithPooling or tuple(torch.FloatTensor)

A transformers.modeling_outputs.BaseModelOutputWithPooling or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (ViTHybridConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing
through the layers used for the auxiliary pretraining task. E.g. for BERT


ViTHybridModel

-family of models, this returns
the classification token after processing through a linear layer and a tanh activation function. The linear
layer weights are trained from the next sentence prediction (classification) objective during pretraining.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compu


ViTHybridModel

te the weighted average in the self-attention
heads.
  The ViTHybridModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoImageProcessor, ViTHybridModel
>>> import torch
>>> from datasets import load_dataset
>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]
>>> image_processor = AutoImageProcessor.from_pretrained("google/vit-hybrid-base-bit-384")
>>> model = ViTHybridModel.from_pretrained("google/vit-hybrid-base-bit-384")
>>> inputs = image_processor(image, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1, 197, 76


ViTHybridForImageClassification

class transformers.ViTHybridForImageClassification  < source > ( config: ViTHybridConfig  )   Parameters   config (ViTHybridConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.    ViT Hybrid Model transformer with an image classification head on top (a linear layer on top of the final hidden
state of the [CLS] token) e.g. for ImageNet. This model is a PyTorch torch.nn.Module subclass. Use it
as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and
behavior.  forward  < source > ( pixel_values: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = N


ViTHybridForImageClassification

one interpolate_pos_encoding: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.ImageClassifierOutput or tuple(torch.FloatTensor)  Parameters   pixel_values (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Pixel values. Pixel values can be obtained using AutoImageProcessor. See
ViTHybridImageProcessor.call() for details.   head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) —
Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
1 indicates the head is not masked,
0 indicates the head is masked.
   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
mor


ViTHybridForImageClassification

e detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the image classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-Entropy).   Returns
transformers.modeling_outputs.ImageClassifierOutput or tuple(torch.FloatTensor)

A transformers.modeling_outputs.ImageClassifierOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (ViTHybridConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, confi


ViTHybridForImageClassification

g.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each stage) of shape (batch_size, sequence_length, hidden_size). Hidden-states
(also called feature maps) of the model at the output of each stage.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, patch_size, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The ViTHybridForImageClassification forward method, overrides the __call__ special method. Alt


ViTHybridForImageClassification

hough the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoImageProcessor, ViTHybridForImageClassification
>>> import torch
>>> from datasets import load_dataset
>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]
>>> image_processor = AutoImageProcessor.from_pretrained("google/vit-hybrid-base-bit-384")
>>> model = ViTHybridForImageClassification.from_pretrained("google/vit-hybrid-base-bit-384")
>>> inputs = image_processor(image, return_tensors="pt")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
>>> # model predicts one of the 1000 ImageNet classes
>>> predicted_label = logits.argmax(-1).item()
>>> print(model.config.id2label[predicted_label])
tabby, tabby ca


Overview

The BERT models trained on Japanese text.
There are models with two different tokenization methods:
Tokenize with MeCab and WordPiece. This requires some extra dependencies, fugashi which is a wrapper around MeCab. Tokenize into characters.
To use MecabTokenizer, you should pip install transformers["ja"] (or pip install -e .["ja"] if you install
from source) to install dependencies.
See details on cl-tohoku repository.
Example of using a model with MeCab and WordPiece tokenization:
  Copied >>> import torch
>>> from transformers import AutoModel, AutoTokenizer
>>> bertjapanese = AutoModel.from_pretrained("cl-tohoku/bert-base-japanese")
>>> tokenizer = AutoTokenizer.from_pretrained("cl-tohoku/bert-base-japanese")
>>> ## Input Japanese Text
>>> line = "吾輩は猫である。"
>>> inpu


Overview

ts = tokenizer(line, return_tensors="pt")
>>> print(tokenizer.decode(inputs["input_ids"][0]))
[CLS] 吾輩 は 猫 で ある 。 [SEP]
>>> outputs = bertjapanese(**inputs)
Example of using a model with Character tokenization:
  Copied >>> bertjapanese = AutoModel.from_pretrained("cl-tohoku/bert-base-japanese-char")
>>> tokenizer = AutoTokenizer.from_pretrained("cl-tohoku/bert-base-japanese-char")
>>> ## Input Japanese Text
>>> line = "吾輩は猫である。"
>>> inputs = tokenizer(line, return_tensors="pt")
>>> print(tokenizer.decode(inputs["input_ids"][0]))
[CLS] 吾 輩 は 猫 で あ る 。 [SEP]
>>> outputs = bertjapanese(**inputs)
Tips:
This implementation is the same as BERT, except for tokenization method. Refer to the documentation of BERT for more usage examples.
This model was contributed by cl-tohoku


BertJapaneseTokenizer

class transformers.BertJapaneseTokenizer  < source > ( vocab_file spm_file = None do_lower_case = False do_word_tokenize = True do_subword_tokenize = True word_tokenizer_type = 'basic' subword_tokenizer_type = 'wordpiece' never_split = None unk_token = '[UNK]' sep_token = '[SEP]' pad_token = '[PAD]' cls_token = '[CLS]' mask_token = '[MASK]' mecab_kwargs = None sudachi_kwargs = None jumanpp_kwargs = None **kwargs  )   Parameters   vocab_file (str) —
Path to a one-wordpiece-per-line vocabulary file.   spm_file (str, optional) —
Path to SentencePiece file (generally has a .spm or .model
extension) that contains the vocabulary.   do_lower_case (bool, optional, defaults to True) —
Whether to lower case the input. Only has an effect when do_basic_tokenize=True.   do_word_tokenize (bool, optional, defaults to True) —
Whether to do word tokenization.   do_subword_tokenize (bool, optional, defaults to True) —
Whether to do subword tokenization.   word_tokenizer_type (str, optional,


BertJapaneseTokenizer

defaults to "basic") —
Type of word tokenizer. Choose from [“basic”, “mecab”, “sudachi”, “jumanpp”].   subword_tokenizer_type (str, optional, defaults to "wordpiece") —
Type of subword tokenizer. Choose from [“wordpiece”, “character”, “sentencepiece”,].   mecab_kwargs (dict, optional) —
Dictionary passed to the MecabTokenizer constructor.   sudachi_kwargs (dict, optional) —
Dictionary passed to the SudachiTokenizer constructor.   jumanpp_kwargs (dict, optional) —
Dictionary passed to the JumanppTokenizer constructor.    Construct a BERT tokenizer for Japanese text. This tokenizer inherits from PreTrainedTokenizer which contains most of the main methods. Users should refer
to: this superclass for more information regarding those methods.  build_inputs_with_special_tokens  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None  ) → List[int]  Parameters   token_ids_0 (List[int]) —
List of IDs to which the special tokens will be added.


BertJapaneseTokenizer

   token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.   Returns
List[int]

List of input IDs with the appropriate special tokens.
  Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A BERT sequence has the following format: single sequence: [CLS] X [SEP] pair of sequences: [CLS] A [SEP] B [SEP]  convert_tokens_to_string  < source > ( tokens  )    Converts a sequence of tokens (string) in a single string.  create_token_type_ids_from_sequences  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None  ) → List[int]  Parameters   token_ids_0 (List[int]) —
List of IDs.   token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.   Returns
List[int]

List of token type IDs according to the given sequence(s).
  Create a mask from the two sequences passed to be used in a sequence-pair classifica


BertJapaneseTokenizer

tion task. A BERT sequence  pair mask has the following format:   Copied 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
| first sequence    | second sequence | If token_ids_1 is None, this method only returns the first portion of the mask (0s).  get_special_tokens_mask  < source > ( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None already_has_special_tokens: bool = False  ) → List[int]  Parameters   token_ids_0 (List[int]) —
List of IDs.   token_ids_1 (List[int], optional) —
Optional second list of IDs for sequence pairs.   already_has_special_tokens (bool, optional, defaults to False) —
Whether or not the token list is already formatted with special tokens for the model.   Returns
List[int]

A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
  Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer prepare_for_model metho


Overview

The Persimmon model was created by ADEPT, and authored by Erich Elsen, Augustus Odena, Maxwell Nye, Sağnak Taşırlar, Tri Dao, Curtis Hawthorne, Deepak Moparthi, Arushi Somani.
The authors introduced Persimmon-8B, a decoder model based on the classic transformers architecture, with query and key normalization. Persimmon-8B is a fully permissively-licensed model with approximately 8 billion parameters, released under the Apache license.  Some of the key attributes of Persimmon-8B are long context size (16K), performance, and capabilities for multimodal extensions.
The authors showcase their approach to model evaluation, focusing on practical text generation, mirroring how users interact with language models. The work also includes a comparative analysis, pitting Persimmon-8B against other prominent


Overview

models (MPT 7B Instruct and Llama 2 Base 7B 1-Shot), across various evaluation tasks. The results demonstrate Persimmon-8B’s competitive performance, even with limited training data.
In terms of model details, the work outlines the architecture and training methodology of Persimmon-8B, providing insights into its design choices, sequence length, and dataset composition. The authors present a fast inference code that outperforms traditional implementations through operator fusion and CUDA graph utilization while maintaining code coherence. They express their anticipation of how the community will leverage this contribution to drive innovation, hinting at further upcoming releases as part of an ongoing series of developments.
The Persimmon models were trained using bfloat16, but the original inferen


Overview

ce uses float16 The checkpoints uploaded on the hub use torch_dtype = 'float16' which will be
used by the AutoModel API to cast the checkpoints from torch.float32 to torch.float16. The dtype of the online weights is mostly irrelevant, unless you are using torch_dtype="auto" when initializing a model using model = AutoModelForCausalLM.from_pretrained("path", torch_dtype = "auto"). The reason is that the model will first be downloaded ( using the dtype of the checkpoints online) then it will be cast to the default dtype of torch (becomes torch.float32). Users should specify the torch_dtype they want, and if they don’t it will be torch.float32. Finetuning the model in float16 is not recommended and known to produce nan, as such the model should be fine-tuned in bfloat16.
Tips:
To convert the model, y


Overview

ou need to clone the original repository using git clone https://github.com/persimmon-ai-labs/adept-inference, then get the checkpoints:
  Copied git clone https://github.com/persimmon-ai-labs/adept-inference
wget https://axtkn4xl5cip.objectstorage.us-phoenix-1.oci.customer-oci.com/n/axtkn4xl5cip/b/adept-public-data/o/8b_base_model_release.tar
tar -xvf 8b_base_model_release.tar
python src/transformers/models/persimmon/convert_persimmon_weights_to_hf.py  --input_dir /path/to/downloaded/persimmon/weights/ --output_dir /output/path \
    --pt_model_path /path/to/8b_chat_model_release/iter_0001251/mp_rank_00/model_optim_rng.pt
    --ada_lib_path /path/to/adept-inference
For the chat model:
  Copied wget https://axtkn4xl5cip.objectstorage.us-phoenix-1.oci.customer-oci.com/n/axtkn4xl5cip/b/adept-public-


Overview

data/o/8b_chat_model_release.tar
tar -xvf 8b_base_model_release.tar
Thereafter, models can be loaded via:
  Copied from transformers import PersimmonForCausalLM, PersimmonTokenizer
model = PersimmonForCausalLM.from_pretrained("/output/path")
tokenizer = PersimmonTokenizer.from_pretrained("/output/path")
This model was contributed by ArthurZ.
The original code can be found here.
Perismmon uses a sentencepiece based tokenizer, with a Unigram model. It supports bytefallback, which is only available in tokenizers==0.14.0 for the fast tokenizer.
The LlamaTokenizer is used as it is a standard wrapper around sentencepiece. The chat template will be updated with the templating functions in a follow up PR! The authors suggest to use the following prompt format for the chat mode: f"human: {prompt}\n\nadept:


PersimmonConfig

class transformers.PersimmonConfig  < source > ( vocab_size = 262144 hidden_size = 4096 intermediate_size = 16384 num_hidden_layers = 36 num_attention_heads = 64 hidden_act = 'relu2' max_position_embeddings = 16384 initializer_range = 0.02 layer_norm_eps = 1e-05 use_cache = True tie_word_embeddings = False rope_theta = 25000.0 rope_scaling = None qk_layernorm = True hidden_dropout = 0.0 attention_dropout = 0.0 partial_rotary_factor = 0.5 pad_token_id = None bos_token_id = 1 eos_token_id = 2 **kwargs  )   Parameters   vocab_size (int, optional, defaults to 262144) —
Vocabulary size of the Persimmon model. Defines the number of different tokens that can be represented by
the inputs_ids passed when calling PersimmonModel   hidden_size (int, optional, defaults to 4096) —
Dimension of the hidden representations.   intermediate_size (int, optional, defaults to 16384) —
Dimension of the MLP representations.   num_hidden_layers (int, optional, defaults to 36) —


PersimmonConfig

Number of hidden layers in the Transformer encoder.   num_attention_heads (int, optional, defaults to 64) —
Number of attention heads for each attention layer in the Transformer encoder.   hidden_act (str or function, optional, defaults to "relu2") —
The non-linear activation function (function or string) in the decoder.   max_position_embeddings (int, optional, defaults to 16384) —
The maximum sequence length that this model might ever be used with.   initializer_range (float, optional, defaults to 0.02) —
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.   layer_norm_eps (float, optional, defaults to 1e-5) —
The epsilon used by the rms normalization layers.   use_cache (bool, optional, defaults to True) —
Whether or not the model should return the last key/values attentions (not used by all models). Only
relevant if config.is_decoder=True.   tie_word_embeddings(bool, optional, defaults to False) —
Whether


PersimmonConfig

to tie weight embeddings   rope_theta (float, optional, defaults to 25000.0) —
The base period of the RoPE embeddings.   rope_scaling (Dict, optional) —
Dictionary containing the scaling configuration for the RoPE embeddings. Currently supports two scaling
strategies: linear and dynamic. Their scaling factor must be an float greater than 1. The expected format
is {"type": strategy name, "factor": scaling factor}. When using this flag, don’t update
max_position_embeddings to the expected new maximum. See the following thread for more information on how
these scaling strategies behave:
https://www.reddit.com/r/LocalPersimmon/comments/14mrgpr/dynamically_scaled_rope_further_increases/. This
is an experimental feature, subject to breaking API changes in future versions.   qk_layernorm (bool, optional, default to True) —
Whether or not to normalize the Queries and Keys after projecting the hidden states   hidden_dropout (float, optional, default to 0.0) —
The


PersimmonConfig

 dropout ratio after applying the MLP to the hidden states.   attention_dropout (float, optional, default to 0.0) —
The dropout ratio after computing the attention scores.   partial_rotary_factor (float, optional, default to 0.5) —
Percentage of the query and keys which will have rotary embedding.
Example —    This is the configuration class to store the configuration of a PersimmonModel. It is used to instantiate an
Persimmon model according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the
adept/persimmon-8b-base. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.    Copied >>> from transformers import PersimmonModel, PersimmonConfig
>>> # Initializing a Persimmon persimmon-7b style configuration
>>> configuration = PersimmonConfig()


PersimmonModel

class transformers.PersimmonModel  < source > ( config: PersimmonConfig  )   Parameters   config (PersimmonConfig) —
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
from_pretrained() method to load the model weights.
config — PersimmonConfig    The bare Persimmon Model outputting raw hidden-states without any specific head on top.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior. Transformer decoder consisting of config.num_hidden_layers layers. Each layer is a PersimmonDecoderLayer


PersimmonModel

  forward  < source > ( input_ids: LongTensor = None attention_mask: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.LongTensor] = None past_key_values: typing.Optional[typing.List[torch.FloatTensor]] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  )   Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1


PersimmonModel

]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
If past_key_values is used, optionally only the last decoder_input_ids have to be input (see
past_key_values).
If you want to change padding behavior, you should read modeling_opt._prepare_decoder_attention_mask
and modify to your needs. See diagram 1 in the paper for more
information on the default strategy.
1 indicates the head is not masked,
0 indicates the head is masked.
   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.n_positions - 1].
What are position IDs?   past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) —
Tuple of tuple(to


PersimmonModel

rch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indi


PersimmonModel

ces into associated vectors than the
model’s internal embedding lookup matrix.   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.    The PersimmonModel forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores the


PersimmonForCausalLM

class transformers.PersimmonForCausalLM  < source > ( config  )     forward  < source > ( input_ids: LongTensor = None attention_mask: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.LongTensor] = None past_key_values: typing.Optional[typing.List[torch.FloatTensor]] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.LongTensor] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None  ) → transformers.modeling_outputs.CausalLMOutputWithPast or tuple(torch.FloatTensor)  Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can


PersimmonForCausalLM

be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
If past_key_values is used, optionally only the last decoder_input_ids have to be input (see
past_key_values).
If you want to change padding behavior, you should read modeling_opt._prepare_decoder_attention_mask
and modify to your needs. See diagram 1 in the paper for more
information on the default strategy.
1 indicates the head is not masked,
0 indicates t


PersimmonForCausalLM

he head is masked.
   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.n_positions - 1].
What are position IDs?   past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) —
Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see past_key_values input) to speed up sequential decoding.
If past_key_values ar


PersimmonForCausalLM

e used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions unde


PersimmonForCausalLM

r returned
tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.
Args —
labels (torch.LongTensor of shape (batch_size, sequence_length), optional):
Labels for computing the masked language modeling loss. Indices should either be in [0, ..., config.vocab_size] or -100 (see input_ids docstring). Tokens with indices set to -100 are ignored
(masked), the loss is only computed for the tokens with labels in [0, ..., config.vocab_size].   Returns
transformers.modeling_outputs.CausalLMOutputWithPast or tuple(torch.FloatTensor)

A transformers.modeling_outputs.CausalLMOutputWithPast or a tuple of
torch.FloatTensor (if return_dict=False is passed or


PersimmonForCausalLM

when config.return_dict=False) comprising various
elements depending on the configuration (PersimmonConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Language modeling loss (for next-token prediction).
logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) — Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head))
Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see
past_key_values input) to speed u


PersimmonForCausalLM

p sequential decoding.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
  The PersimmonForCausalLM forward method, overrides


PersimmonForCausalLM

 the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.  Example:   Copied >>> from transformers import AutoTokenizer, PersimmonForCausalLM
>>> model = PersimmonForCausalLM.from_pretrained("adept/persimmon-8b-base")
>>> tokenizer = AutoTokenizer.from_pretrained("adept/persimmon-8b-base")
>>> prompt = "human: Hey, what should I eat for dinner?"
>>> inputs = tokenizer(prompt, return_tensors="pt")
>>> # Generate
>>> generate_ids = model.generate(inputs.input_ids, max_length=30)
>>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
'human: Hey, what should I eat for dinner?\n\ncat: 🐱\n\nhuman:


PersimmonForSequenceClassification

class transformers.PersimmonForSequenceClassification  < source > ( config  )   Parameters   config (PersimmonConfig) —
Model configuration class with all the parameters of the model. Initializing with a config file does not
load the weights associated with the model, only the configuration. Check out the
from_pretrained() method to load the model weights.    The Persimmon transformer with a sequence classification head on top (linear layer). PersimmonForSequenceClassification uses the last token in order to do the classification, as other causal
models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a
pad_token_id is defined in the configuration, it finds the last token that is not a padding token in each row. If
no pad_token_id is defined, it simply takes the last value in each row of the batch. Since it cannot guess the
padding tokens when inputs_embeds are passed instead of i


PersimmonForSequenceClassification

nput_ids, it does the same (take the last value in
each row of the batch). This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.) This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.  forward  < source > ( input_ids: LongTensor = None attention_mask: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.LongTensor] = None past_key_values: typing.Optional[typing.List[torch.FloatTensor]] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.LongTensor] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None retu


PersimmonForSequenceClassification

rn_dict: typing.Optional[bool] = None  )   Parameters   input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?   attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
If past_key_values is used, optionally only the last decoder_input_ids have to be input (see
past_key_values).
If you want to change padding behavior, you should read modeling_opt._prepare_decoder_a


PersimmonForSequenceClassification

ttention_mask
and modify to your needs. See diagram 1 in the paper for more
information on the default strategy.
1 indicates the head is not masked,
0 indicates the head is masked.
   position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.n_positions - 1].
What are position IDs?   past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) —
Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape
(batch_size, num_heads, sequence_length, embed_size_per_head)) and 2 additional tensors of shape
(batch_size, num_heads, encoder_sequence_length, embed_size_per_head).
Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
blocks) that can be used (see p


PersimmonForSequenceClassification

ast_key_values input) to speed up sequential decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).   inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) —
Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert input_ids indices into associated vectors than the
model’s internal embedding lookup matrix.   use_cache (bool, optional) —
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).   output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned


PersimmonForSequenceClassification

tensors for more detail.   output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.   return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.   labels (torch.LongTensor of shape (batch_size,), optional) —
Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If
config.num_labels > 1 a classification loss is computed (Cross-Entropy).    The PersimmonForSequenceClassification forward method, overrides the __call__ special method. Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores t


Overview

The OneFormer model was proposed in OneFormer: One Transformer to Rule Universal Image Segmentation by Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, Humphrey Shi. OneFormer is a universal image segmentation framework that can be trained on a single panoptic dataset to perform semantic, instance, and panoptic segmentation tasks. OneFormer uses a task token to condition the model on the task in focus, making the architecture task-guided for training, and task-dynamic for inference.
The abstract from the paper is the following:
Universal Image Segmentation is not a new concept. Past attempts to unify image segmentation in the last decades include scene parsing, panoptic segmentation, and, more recently, new panoptic architectures. However, such panoptic architectures do not truly unify image segmentation because they need to be trained individua


Overview

lly on the semantic, instance, or panoptic segmentation to achieve the best performance. Ideally, a truly universal framework should be trained only once and achieve SOTA performance across all three image segmentation tasks. To that end, we propose OneFormer, a universal image segmentation framework that unifies segmentation with a multi-task train-once design. We first propose a task-conditioned joint training strategy that enables training on ground truths of each domain (semantic, instance, and panoptic segmentation) within a single multi-task training process. Secondly, we introduce a task token to condition our model on the task at hand, making our model task-dynamic to support multi-task training and inference. Thirdly, we propose using a query-text contrastive loss during training to establish better inter-task and inter-class distinctions. Notably, our


Overview

single OneFormer model outperforms specialized Mask2Former models across all three segmentation tasks on ADE20k, CityScapes, and COCO, despite the latter being trained on each of the three tasks individually with three times the resources. With new ConvNeXt and DiNAT backbones, we observe even more performance improvement. We believe OneFormer is a significant step towards making image segmentation more universal and accessible.
Tips:
OneFormer requires two inputs during inference: image and task token. During training, OneFormer only uses panoptic annotations. If you want to train the model in a distributed environment across multiple nodes, then one should update the
get_num_masks function inside in the OneFormerLoss class of modeling_oneformer.py. When training on multiple nodes, this should be
set to the average number of target masks across all nodes, as ca


Overview

n be seen in the original implementation here. One can use OneFormerProcessor to prepare input images and task inputs for the model and optional targets for the model. OneformerProcessor wraps OneFormerImageProcessor and CLIPTokenizer into a single instance to both prepare the images and encode the task inputs. To get the final segmentation, depending on the task, you can call post_process_semantic_segmentation() or post_process_instance_segmentation() or post_process_panoptic_segmentation(). All three tasks can be solved using OneFormerForUniversalSegmentation output, panoptic segmentation accepts an optional label_ids_to_fuse argument to fuse instances of the target object/s (e.g. sky) together.
The figure below illustrates the architecture of OneFormer. Taken from the original paper.
This model was contributed by Jitesh Jain. The original code can be found he


Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with OneFormer.
Demo notebooks regarding inference + fine-tuning on custom data can be found here.
If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.
The resource should ideally demonstrate something new instead of duplicating an existing resource.


OneFormer specific outputs

class transformers.models.oneformer.modeling_oneformer.OneFormerModelOutput  < source > ( encoder_hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None pixel_decoder_hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None transformer_decoder_hidden_states: typing.Optional[torch.FloatTensor] = None transformer_decoder_object_queries: FloatTensor = None transformer_decoder_contrastive_queries: typing.Optional[torch.FloatTensor] = None transformer_decoder_mask_predictions: FloatTensor = None transformer_decoder_class_predictions: FloatTensor = None transformer_decoder_auxiliary_predictions: typing.Union[typing.Tuple[typing.Dict[str, torch.FloatTensor]], NoneType] = None text_queries: typing.Optional[torch.FloatTensor] = None task_token: FloatTensor = None attentions: typing.Optional[typing.Tuple[torch.FloatTensor]] = None  )   Parameters   encoder_hidden_states (tuple(torch.FloatTensor), optional, ret


OneFormer specific outputs

urned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the encoder
model at the output of each stage.   pixel_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the pixel
decoder model at the output of each stage.   transformer_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor (on


OneFormer specific outputs

e for the output of the embeddings + one for the output of each stage) of
shape (batch_size, sequence_length, hidden_size). Hidden-states (also called feature maps) of the
transformer decoder at the output of each stage.   transformer_decoder_object_queries (torch.FloatTensor of shape (batch_size, num_queries, hidden_dim)) —
Output object queries from the last layer in the transformer decoder.   transformer_decoder_contrastive_queries (torch.FloatTensor of shape (batch_size, num_queries, hidden_dim)) —
Contrastive queries from the transformer decoder.   transformer_decoder_mask_predictions (torch.FloatTensor of shape (batch_size, num_queries, height, width)) —
Mask Predictions from the last layer in the transformer decoder.   transformer_decoder_class_predictions (torch.FloatTensor of shape (batch_size, num_queries, num_classes+1)) —
Class Predictions from the last layer in the transformer decoder.   transformer_decoder_auxiliary


OneFormer specific outputs

_predictions (Tuple of Dict of str, torch.FloatTensor, optional) —
Tuple of class and mask predictions from each layer of the transformer decoder.   text_queries (torch.FloatTensor, optional of shape (batch_size, num_queries, hidden_dim)) —
Text queries derived from the input text list used for calculating contrastive loss during training.   task_token (torch.FloatTensor of shape (batch_size, hidden_dim)) —
1D task token to condition the queries.   attentions (tuple(tuple(torch.FloatTensor)), optional, returned when output_attentions=True is passed or when config.output_attentions=True) —
Tuple of tuple(torch.FloatTensor) (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Self and Cross Attentions weights from transformer decoder.    Class for outputs of OneFormerModel. This class returns all the needed hidden states to compute the logits.
 class transformers.models.oneformer.modeling_oneform


OneFormer specific outputs

er.OneFormerForUniversalSegmentationOutput  < source > ( loss: typing.Optional[torch.FloatTensor] = None class_queries_logits: FloatTensor = None masks_queries_logits: FloatTensor = None auxiliary_predictions: typing.List[typing.Dict[str, torch.FloatTensor]] = None encoder_hidden_states: typing.Optional[typing.Tuple[torch.FloatTensor]] = None pixel_decoder_hidden_states: typing.Optional[typing.List[torch.FloatTensor]] = None transformer_decoder_hidden_states: typing.Optional[torch.FloatTensor] = None transformer_decoder_object_queries: FloatTensor = None transformer_decoder_contrastive_queries: typing.Optional[torch.FloatTensor] = None transformer_decoder_mask_predictions: FloatTensor = None transformer_decoder_class_predictions: FloatTensor = None transformer_decoder_auxiliary_predictions: typing.Union[typing.List[typing.Dict[str, torch.FloatTensor]], NoneType] = None text_queries: typing.Optional[torch.FloatTensor] = None task_


OneFormer specific outputs

token: FloatTensor = None attentions: typing.Optional[typing.Tuple[typing.Tuple[torch.FloatTensor]]] = None  )   Parameters   loss (torch.Tensor, optional) —
The computed loss, returned when labels are present.   class_queries_logits (torch.FloatTensor) —
A tensor of shape (batch_size, num_queries, num_labels + 1) representing the proposed classes for each
query. Note the + 1 is needed because we incorporate the null class.   masks_queries_logits (torch.FloatTensor) —
A tensor of shape (batch_size, num_queries, height, width) representing the proposed masks for each
query.   auxiliary_predictions (List of Dict of str, torch.FloatTensor, optional) —
List of class and mask predictions from each layer of the transformer decoder.   encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor (one for the output of th


OneFormer specific outputs

e embeddings + one for the output of each stage) of
shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the encoder
model at the output of each stage.   pixel_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the pixel
decoder model at the output of each stage.   transformer_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) —
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of
shape (batch_size, sequence_length, hidden_size). Hidden-states (also


OneFormer specific outputs

 called feature maps) of the
transformer decoder at the output of each stage.   transformer_decoder_object_queries (torch.FloatTensor of shape (batch_size, num_queries, hidden_dim)) —
Output object queries from the last layer in the transformer decoder.   transformer_decoder_contrastive_queries (torch.FloatTensor of shape (batch_size, num_queries, hidden_dim)) —
Contrastive queries from the transformer decoder.   transformer_decoder_mask_predictions (torch.FloatTensor of shape (batch_size, num_queries, height, width)) —
Mask Predictions from the last layer in the transformer decoder.   transformer_decoder_class_predictions (torch.FloatTensor of shape (batch_size, num_queries, num_classes+1)) —
Class Predictions from the last layer in the transformer decoder.   transformer_decoder_auxiliary_predictions (List of Dict of str, torch.FloatTensor, optional) —
List of class and mask predictions from each layer of the transformer decoder


OneFormer specific outputs

.   text_queries (torch.FloatTensor, optional of shape (batch_size, num_queries, hidden_dim)) —
Text queries derived from the input text list used for calculating contrastive loss during training.   task_token (torch.FloatTensor of shape (batch_size, hidden_dim)) —
1D task token to condition the queries.   attentions (tuple(tuple(torch.FloatTensor)), optional, returned when output_attentions=True is passed or when config.output_attentions=True) —
Tuple of tuple(torch.FloatTensor) (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Self and Cross Attentions weights from transformer decoder.    Class for outputs of OneFormerForUniversalSegmentationOutput. This output can be directly passed to post_process_semantic_segmentation() or
post_process_instance_segmentation() or
post_process_panoptic_segmentation() depending on the task. Please, see
[`~OneFormerImageProcessor] for details regarding usag


OneFormerConfig

class transformers.OneFormerConfig  < source > ( backbone_config: typing.Optional[typing.Dict] = None ignore_value: int = 255 num_queries: int = 150 no_object_weight: int = 0.1 class_weight: float = 2.0 mask_weight: float = 5.0 dice_weight: float = 5.0 contrastive_weight: float = 0.5 contrastive_temperature: float = 0.07 train_num_points: int = 12544 oversample_ratio: float = 3.0 importance_sample_ratio: float = 0.75 init_std: float = 0.02 init_xavier_std: float = 1.0 layer_norm_eps: float = 1e-05 is_training: bool = False use_auxiliary_loss: bool = True output_auxiliary_logits: bool = True strides: typing.Optional[list] = [4, 8, 16, 32] task_seq_len: int = 77 text_encoder_width: int = 256 text_encoder_context_length: int = 77 text_encoder_num_layers: int = 6 text_encoder_vocab_size: int = 49408 text_encoder_proj_layers: int = 2 text_encoder_n_ctx: int = 16 conv_dim: int = 256 mask_dim: int = 256 hidden


OneFormerConfig

_dim: int = 256 encoder_feedforward_dim: int = 1024 norm: str = 'GN' encoder_layers: int = 6 decoder_layers: int = 10 use_task_norm: bool = True num_attention_heads: int = 8 dropout: float = 0.1 dim_feedforward: int = 2048 pre_norm: bool = False enforce_input_proj: bool = False query_dec_layers: int = 2 common_stride: int = 4 **kwargs  )   Parameters   backbone_config (PretrainedConfig, optional, defaults to SwinConfig) —
The configuration of the backbone model.   ignore_value (int, optional, defaults to 255) —
Values to be ignored in GT label while calculating loss.   num_queries (int, optional, defaults to 150) —
Number of object queries.   no_object_weight (float, optional, defaults to 0.1) —
Weight for no-object class predictions.   class_weight (float, optional, defaults to 2.0) —
Weight for Classification CE loss.   mask_weight (float, optional, defaults to 5.0) —
Weight for binary CE loss.   dice


OneFormerConfig

_weight (float, optional, defaults to 5.0) —
Weight for dice loss.   contrastive_weight (float, optional, defaults to 0.5) —
Weight for contrastive loss.   contrastive_temperature (float, optional, defaults to 0.07) —
Initial value for scaling the contrastive logits.   train_num_points (int, optional, defaults to 12544) —
Number of points to sample while calculating losses on mask predictions.   oversample_ratio (float, optional, defaults to 3.0) —
Ratio to decide how many points to oversample.   importance_sample_ratio (float, optional, defaults to 0.75) —
Ratio of points that are sampled via importance sampling.   init_std (float, optional, defaults to 0.02) —
Standard deviation for normal intialization.   init_xavier_std (float, optional, defaults to 1.0) —
Standard deviation for xavier uniform initialization.   layer_norm_eps (float, optional, defaults to 1e-05) —
Epsilon for layer normalization.


OneFormerConfig

is_training (bool, optional, defaults to False) —
Whether to run in training or inference mode.   use_auxiliary_loss (bool, optional, defaults to True) —
Whether to calculate loss using intermediate predictions from transformer decoder.   output_auxiliary_logits (bool, optional, defaults to True) —
Whether to return intermediate predictions from transformer decoder.   strides (list, optional, defaults to [4, 8, 16, 32]) —
List containing the strides for feature maps in the encoder.   task_seq_len (int, optional, defaults to 77) —
Sequence length for tokenizing text list input.   text_encoder_width (int, optional, defaults to 256) —
Hidden size for text encoder.   text_encoder_context_length (int, optional, defaults to 77) —
Input sequence length for text encoder.   text_encoder_num_layers (int, optional, defaults to 6) —
Number of layers for transformer in text encoder.   text_encoder_vocab_size (int, o


OneFormerConfig

ptional, defaults to 49408) —
Vocabulary size for tokenizer.   text_encoder_proj_layers (int, optional, defaults to 2) —
Number of layers in MLP for project text queries.   text_encoder_n_ctx (int, optional, defaults to 16) —
Number of learnable text context queries.   conv_dim (int, optional, defaults to 256) —
Feature map dimension to map outputs from the backbone.   mask_dim (int, optional, defaults to 256) —
Dimension for feature maps in pixel decoder.   hidden_dim (int, optional, defaults to 256) —
Dimension for hidden states in transformer decoder.   encoder_feedforward_dim (int, optional, defaults to 1024) —
Dimension for FFN layer in pixel decoder.   norm (str, optional, defaults to "GN") —
Type of normalization.   encoder_layers (int, optional, defaults to 6) —
Number of layers in pixel decoder.   decoder_layers (int, optional, defaults to 10) —
Number of layers in transformer decoder.   use_ta


OneFormerConfig

sk_norm (bool, optional, defaults to True) —
Whether to normalize the task token.   num_attention_heads (int, optional, defaults to 8) —
Number of attention heads in transformer layers in the pixel and transformer decoders.   dropout (float, optional, defaults to 0.1) —
Dropout probability for pixel and transformer decoders.   dim_feedforward (int, optional, defaults to 2048) —
Dimension for FFN layer in transformer decoder.   pre_norm (bool, optional, defaults to False) —
Whether to normalize hidden states before attention layers in transformer decoder.   enforce_input_proj (bool, optional, defaults to False) —
Whether to project hidden states in transformer decoder.   query_dec_layers (int, optional, defaults to 2) —
Number of layers in query transformer.   common_stride (int, optional, defaults to 4) —
Common stride used for features in pixel decoder.    This is the configuration class to store the c


OneFormerConfig

onfiguration of a OneFormerModel. It is used to instantiate a
OneFormer model according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the OneFormer
shi-labs/oneformer_ade20k_swin_tiny architecture
trained on ADE20k-150. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.  Examples:   Copied >>> from transformers import OneFormerConfig, OneFormerModel
>>> # Initializing a OneFormer shi-labs/oneformer_ade20k_swin_tiny configuration
>>> configuration = OneFormerConfig()
>>> # Initializing a model (with random weights) from the shi-labs/oneformer_ade20k_swin_tiny style configuration
>>> model = OneFormerModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.


OneFormerImageProcessor

class transformers.OneFormerImageProcessor  < source > ( do_resize: bool = True size: typing.Dict[str, int] = None resample: Resampling = <Resampling.BILINEAR: 2> do_rescale: bool = True rescale_factor: float = 0.00392156862745098 do_normalize: bool = True image_mean: typing.Union[float, typing.List[float]] = None image_std: typing.Union[float, typing.List[float]] = None ignore_index: typing.Optional[int] = None do_reduce_labels: bool = False repo_path: str = 'shi-labs/oneformer_demo' class_info_file: str = None num_text: typing.Optional[int] = None **kwargs  )   Parameters   do_resize (bool, optional, defaults to True) —
Whether to resize the input to a certain size.   size (int, optional, defaults to 800) —
Resize the input to the given size. Only has an effect if do_resize is set to True. If size is a
sequence like (width, height), output size will be matched to this. If size is an int, smaller edge of
the image will be matched to this number. i.e, if height > width, then imag


OneFormerImageProcessor

e will be rescaled to (size * height / width, size).   resample (int, optional, defaults to Resampling.BILINEAR) —
An optional resampling filter. This can be one of PIL.Image.Resampling.NEAREST,
PIL.Image.Resampling.BOX, PIL.Image.Resampling.BILINEAR, PIL.Image.Resampling.HAMMING,
PIL.Image.Resampling.BICUBIC or PIL.Image.Resampling.LANCZOS. Only has an effect if do_resize is set
to True.   do_rescale (bool, optional, defaults to True) —
Whether to rescale the input to a certain scale.   rescale_factor (float, optional, defaults to 1/ 255) —
Rescale the input by the given factor. Only has an effect if do_rescale is set to True.   do_normalize (bool, optional, defaults to True) —
Whether or not to normalize the input with mean and standard deviation.   image_mean (int, optional, defaults to [0.485, 0.456, 0.406]) —
The sequence of means for each channel, to be used when normalizing images. Defaults to the ImageNet mean.   image_std (int, optional, defaults to [0.229, 0.224, 0.225]


OneFormerImageProcessor

) —
The sequence of standard deviations for each channel, to be used when normalizing images. Defaults to the
ImageNet std.   ignore_index (int, optional) —
Label to be assigned to background pixels in segmentation maps. If provided, segmentation map pixels
denoted with 0 (background) will be replaced with ignore_index.   do_reduce_labels (bool, optional, defaults to False) —
Whether or not to decrement all label values of segmentation maps by 1. Usually used for datasets where 0
is used for background, and background itself is not included in all classes of a dataset (e.g. ADE20k).
The background label will be replaced by ignore_index.   repo_path (str, defaults to shi-labs/oneformer_demo, optional, defaults to "shi-labs/oneformer_demo") —
Dataset repository on huggingface hub containing the JSON file with class information for the dataset.   class_info_file (str, optional) —
JSON file containing class information for the dataset. It is stored inside on the repo_path dataset
rep


OneFormerImageProcessor

ository.   num_text (int, optional) —
Number of text entries in the text input list.    Constructs a OneFormer image processor. The image processor can be used to prepare image(s), task input(s) and
optional text inputs and targets for the model. This image processor inherits from BaseImageProcessor which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.  preprocess  < source > ( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')]] task_inputs: typing.Optional[typing.List[str]] = None segmentation_maps: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), typing.List[ForwardRef('PIL.Image.Image')], typing.List[numpy.ndarray], typing.List[ForwardRef('torch.Tensor')], NoneType] = None instance_id_to_semantic_id: typing.Union[typ