Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,14 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+model.rknn filter=lfs diff=lfs merge=lfs -text
+model_b1_s1024.rknn filter=lfs diff=lfs merge=lfs -text
+model_b1_s256.rknn filter=lfs diff=lfs merge=lfs -text
+rknn/model_b1_s1024_o1.rknn filter=lfs diff=lfs merge=lfs -text
+rknn/model_b1_s1024_o2.rknn filter=lfs diff=lfs merge=lfs -text
+rknn/model_b1_s1024_o3.rknn filter=lfs diff=lfs merge=lfs -text
+rknn/model_b1_s1024_w8a8.rknn filter=lfs diff=lfs merge=lfs -text
+rknn/model_o1.rknn filter=lfs diff=lfs merge=lfs -text
+rknn/model_o2.rknn filter=lfs diff=lfs merge=lfs -text
+rknn/model_o3.rknn filter=lfs diff=lfs merge=lfs -text
+rknn/model_w8a8.rknn filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,254 @@
+---
+library_name: rk-transformers
+license: apache-2.0
+language:
+- en
+tags:
+- fill-mask
+- masked-lm
+- long-context
+- modernbert
+- rknn
+- rockchip
+- npu
+- rk-transformers
+- rk3588
+pipeline_tag: fill-mask
+inference: false
+datasets:
+- sentence-transformers/natural-questions
+base_model: answerdotai/ModernBERT-base
+model_name: ModernBERT-base
+---
+# ModernBERT-base (RKNN2)
+
+> This is an RKNN-compatible version of the [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) model. It has been optimized for Rockchip NPUs using the [rk-transformers](https://github.com/emapco/rk-transformers) library.
+
+<details><summary>Click to see the RKNN model details and usage examples</summary>
+
+## Model Details
+
+- **Original Model:** [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base)
+- **Target Platform:** rk3588
+- **rknn-toolkit2 Version:** 2.3.2
+- **rk-transformers Version:** 0.3.1
+
+### Available Model Files
+
+| Model File | Optimization Level | Quantization | File Size |
+| :--------- | :----------------- | :----------- | :-------- |
+| [model.rknn](./model.rknn) | 0 | float16 | 316.2 MB |
+| [model_b1_s1024.rknn](./model_b1_s1024.rknn) | 0 | float16 | 370.2 MB |
+| [model_b1_s256.rknn](./model_b1_s256.rknn) | 0 | float16 | 301.4 MB |
+| [rknn/model_b1_s1024_o1.rknn](./rknn/model_b1_s1024_o1.rknn) | 1 | float16 | 370.2 MB |
+| [rknn/model_b1_s1024_o2.rknn](./rknn/model_b1_s1024_o2.rknn) | 2 | float16 | 370.2 MB |
+| [rknn/model_b1_s1024_o3.rknn](./rknn/model_b1_s1024_o3.rknn) | 3 | float16 | 370.2 MB |
+| [rknn/model_b1_s1024_w8a8.rknn](./rknn/model_b1_s1024_w8a8.rknn) | 0 | w8a8 | 193.0 MB |
+| [rknn/model_o1.rknn](./rknn/model_o1.rknn) | 1 | float16 | 316.2 MB |
+| [rknn/model_o2.rknn](./rknn/model_o2.rknn) | 2 | float16 | 316.2 MB |
+| [rknn/model_o3.rknn](./rknn/model_o3.rknn) | 3 | float16 | 316.2 MB |
+| [rknn/model_w8a8.rknn](./rknn/model_w8a8.rknn) | 0 | w8a8 | 164.9 MB |
+
+## Usage
+
+### Installation
+
+Install `rk-transformers` with inference dependencies to use this model:
+
+```bash
+pip install rk-transformers[inference]
+```
+
+#### RK-Transformers API
+
+```python
+from rktransformers import RKModelForFeatureExtraction
+from transformers import AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("rk-transformers/ModernBERT-base")
+model = RKModelForFeatureExtraction.from_pretrained(
+    "rk-transformers/ModernBERT-base",
+    platform="rk3588",
+    core_mask="auto",
+)
+
+inputs = tokenizer("My name is Philipp and I live in Germany.", return_tensors="np")
+outputs = model(**inputs)
+last_hidden_state = outputs.last_hidden_state
+print(last_hidden_state.shape)
+
+# Load specific optimized/quantized model file
+model = RKModelForFeatureExtraction.from_pretrained(
+    "rk-transformers/ModernBERT-base",
+    platform="rk3588",
+    file_name="rknn/model_b1_s1024_w8a8.rknn"
+)
+```
+
+## Configuration
+
+The full configuration for all exported RKNN models is available in the [config.json](./config.json) file.
+
+</details>
+
+---
+
+# ModernBERT
+
+## Table of Contents
+1. [Model Summary](#model-summary)
+2. [Usage](#Usage)
+3. [Evaluation](#Evaluation)
+4. [Limitations](#limitations)
+5. [Training](#training)
+6. [License](#license)
+7. [Citation](#citation)
+
+## Model Summary
+
+ModernBERT is a modernized bidirectional encoder-only Transformer model (BERT-style) pre-trained on 2 trillion tokens of English and code data with a native context length of up to 8,192 tokens. ModernBERT leverages recent architectural improvements such as:
+
+- **Rotary Positional Embeddings (RoPE)** for long-context support.  
+- **Local-Global Alternating Attention** for efficiency on long inputs.  
+- **Unpadding and Flash Attention** for efficient inference.  
+
+ModernBERT’s native long context length makes it ideal for tasks that require processing long documents, such as retrieval, classification, and semantic search within large corpora. The model was trained on a large corpus of text and code, making it suitable for a wide range of downstream tasks, including code retrieval and hybrid (text + code) semantic search.
+
+It is available in the following sizes:
+
+- [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) - 22 layers, 149 million parameters
+- [ModernBERT-large](https://huggingface.co/answerdotai/ModernBERT-large) - 28 layers, 395 million parameters
+
+For more information about ModernBERT, we recommend our [release blog post](https://huggingface.co/blog/modernbert) for a high-level overview, and our [arXiv pre-print](https://arxiv.org/abs/2412.13663) for in-depth information.
+
+*ModernBERT is a collaboration between [Answer.AI](https://answer.ai), [LightOn](https://lighton.ai), and friends.*
+
+## Usage
+
+You can use these models directly with the `transformers` library starting from v4.48.0:
+
+```sh
+pip install -U transformers>=4.48.0
+```
+
+Since ModernBERT is a Masked Language Model (MLM), you can use the `fill-mask` pipeline or load it via `AutoModelForMaskedLM`. To use ModernBERT for downstream tasks like classification, retrieval, or QA, fine-tune it following standard BERT fine-tuning recipes.
+
+**⚠️ If your GPU supports it, we recommend using ModernBERT with Flash Attention 2 to reach the highest efficiency. To do so, install Flash Attention as follows, then use the model as normal:**
+
+```bash
+pip install flash-attn
+```
+
+Using `AutoModelForMaskedLM`:
+
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+
+model_id = "answerdotai/ModernBERT-base"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForMaskedLM.from_pretrained(model_id)
+
+text = "The capital of France is [MASK]."
+inputs = tokenizer(text, return_tensors="pt")
+outputs = model(**inputs)
+
+# To get predictions for the mask:
+masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
+predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1)
+predicted_token = tokenizer.decode(predicted_token_id)
+print("Predicted token:", predicted_token)
+# Predicted token:  Paris
+```
+
+Using a pipeline:
+
+```python
+import torch
+from transformers import pipeline
+from pprint import pprint
+
+pipe = pipeline(
+    "fill-mask",
+    model="answerdotai/ModernBERT-base",
+    torch_dtype=torch.bfloat16,
+)
+
+input_text = "He walked to the [MASK]."
+results = pipe(input_text)
+pprint(results)
+```
+
+**Note:** ModernBERT does not use token type IDs, unlike some earlier BERT models. Most downstream usage is identical to standard BERT models on the Hugging Face Hub, except you can omit the `token_type_ids` parameter.
+
+## Evaluation
+
+We evaluate ModernBERT across a range of tasks, including natural language understanding (GLUE), general retrieval (BEIR), long-context retrieval (MLDR), and code retrieval (CodeSearchNet and StackQA).
+
+**Key highlights:**
+- On GLUE, ModernBERT-base surpasses other similarly-sized encoder models, and ModernBERT-large is second only to Deberta-v3-large.
+- For general retrieval tasks, ModernBERT performs well on BEIR in both single-vector (DPR-style) and multi-vector (ColBERT-style) settings.
+- Thanks to the inclusion of code data in its training mixture, ModernBERT as a backbone also achieves new state-of-the-art code retrieval results on CodeSearchNet and StackQA.
+
+### Base Models
+
+| Model       | IR (DPR)     | IR (DPR)     | IR (DPR)     | IR (ColBERT)  | IR (ColBERT)  | NLU  | Code | Code |
+|-------------|--------------|--------------|--------------|---------------|---------------|------|------|------|
+|             | BEIR         | MLDR_OOD     | MLDR_ID      | BEIR          | MLDR_OOD      | GLUE | CSN  | SQA  |
+| BERT        | 38.9         | 23.9         | 32.2         | 49.0          | 28.1          | 84.7 | 41.2 | 59.5 |
+| RoBERTa     | 37.7         | 22.9         | 32.8         | 48.7          | 28.2          | 86.4 | 44.3 | 59.6 |
+| DeBERTaV3   | 20.2         | 5.4          | 13.4         | 47.1          | 21.9          | 88.1 | 17.5 | 18.6 |
+| NomicBERT   | 41.0         | 26.7         | 30.3         | 49.9          | 61.3          | 84.0 | 41.6 | 61.4 |
+| GTE-en-MLM  | 41.4         | **34.3**    |**44.4**   | 48.2          | 69.3          | 85.6 | 44.9 | 71.4 |
+| ModernBERT  | **41.6**    | 27.4         | 44.0         | **51.3**    | **80.2**      | **88.4** | **56.4** |**73.6**|
+
+---
+
+### Large Models
+
+| Model       | IR (DPR)     | IR (DPR)     | IR (DPR)     | IR (ColBERT)  | IR (ColBERT)  | NLU  | Code | Code |
+|-------------|--------------|--------------|--------------|---------------|---------------|------|------|------|
+|             | BEIR         | MLDR_OOD     | MLDR_ID      | BEIR          | MLDR_OOD      | GLUE | CSN  | SQA  |
+| BERT        | 38.9         | 23.3         | 31.7         | 49.5          | 28.5          | 85.2 | 41.6 | 60.8 |
+| RoBERTa     | 41.4         | 22.6         | 36.1         | 49.8          | 28.8          | 88.9 | 47.3 | 68.1 |
+| DeBERTaV3   | 25.6         | 7.1          | 19.2         | 46.7          | 23.0          | **91.4**| 21.2 | 19.7 |
+| GTE-en-MLM  | 42.5         | **36.4**    | **48.9**  | 50.7          | 71.3          | 87.6 | 40.5 | 66.9 |
+| ModernBERT  | **44.0**    | 34.3         | 48.6         | **52.4**     | **80.4**     | 90.4 |**59.5** |**83.9**|
+
+*Table 1: Results for all models across an overview of all tasks. CSN refers to CodeSearchNet and SQA to StackQA. MLDRID refers to in-domain (fine-tuned on the training set) evaluation, and MLDR_OOD to out-of-domain.*
+
+ModernBERT’s strong results, coupled with its efficient runtime on long-context inputs, demonstrate that encoder-only models can be significantly improved through modern architectural choices and extensive pretraining on diversified data sources.
+
+
+## Limitations
+
+ModernBERT’s training data is primarily English and code, so performance may be lower for other languages. While it can handle long sequences efficiently, using the full 8,192 tokens window may be slower than short-context inference. Like any large language model, ModernBERT may produce representations that reflect biases present in its training data. Verify critical or sensitive outputs before relying on them.
+
+## Training
+
+- Architecture: Encoder-only, Pre-Norm Transformer with GeGLU activations.
+- Sequence Length: Pre-trained up to 1,024 tokens, then extended to 8,192 tokens.
+- Data: 2 trillion tokens of English text and code.
+- Optimizer: StableAdamW with trapezoidal LR scheduling and 1-sqrt decay.
+- Hardware: Trained on 8x H100 GPUs.
+
+See the paper for more details.
+
+## License
+
+We release the ModernBERT model architectures, model weights, training codebase under the Apache 2.0 license.
+
+## Citation
+
+If you use ModernBERT in your work, please cite:
+
+```
+@misc{modernbert,
+      title={Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference}, 
+      author={Benjamin Warner and Antoine Chaffin and Benjamin Clavié and Orion Weller and Oskar Hallström and Said Taghadouini and Alexis Gallagher and Raja Biswas and Faisal Ladhak and Tom Aarsen and Nathan Cooper and Griffin Adams and Jeremy Howard and Iacopo Poli},
+      year={2024},
+      eprint={2412.13663},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2412.13663}, 
+}
+```

config.json

@@ -0,0 +1,539 @@
+{
+  "architectures": [
+    "ModernBertForMaskedLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 50281,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 50281,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "embedding_dropout": 0.0,
+  "eos_token_id": 50282,
+  "global_attn_every_n_layers": 3,
+  "global_rope_theta": 160000.0,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 768,
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 1152,
+  "layer_norm_eps": 1e-05,
+  "local_attention": 128,
+  "local_rope_theta": 10000.0,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 12,
+  "num_hidden_layers": 22,
+  "pad_token_id": 50283,
+  "position_embedding_type": "absolute",
+  "repad_logits_with_grad": false,
+  "rknn": {
+    "model.rknn": {
+      "batch_size": 1,
+      "custom_string": null,
+      "dynamic_input": null,
+      "float_dtype": "float16",
+      "inputs_yuv_fmt": null,
+      "max_seq_length": 512,
+      "mean_values": null,
+      "model_input_names": [
+        "input_ids",
+        "attention_mask"
+      ],
+      "opset": 19,
+      "optimization": {
+        "compress_weight": false,
+        "enable_flash_attention": true,
+        "model_pruning": false,
+        "optimization_level": 0,
+        "remove_reshape": false,
+        "remove_weight": false,
+        "sparse_infer": false
+      },
+      "quantization": {
+        "auto_hybrid_cos_thresh": 0.98,
+        "auto_hybrid_euc_thresh": null,
+        "dataset_columns": null,
+        "dataset_name": null,
+        "dataset_size": 128,
+        "dataset_split": null,
+        "dataset_subset": null,
+        "do_quantization": false,
+        "quant_img_RGB2BGR": false,
+        "quantized_algorithm": "normal",
+        "quantized_dtype": "w8a8",
+        "quantized_hybrid_level": 0,
+        "quantized_method": "channel"
+      },
+      "rktransformers_version": "0.3.1",
+      "single_core_mode": false,
+      "std_values": null,
+      "target_platform": "rk3588",
+      "task": "feature-extraction",
+      "task_kwargs": null
+    },
+    "model_b1_s1024.rknn": {
+      "batch_size": 1,
+      "custom_string": null,
+      "dynamic_input": null,
+      "float_dtype": "float16",
+      "inputs_yuv_fmt": null,
+      "max_seq_length": 1024,
+      "mean_values": null,
+      "model_input_names": [
+        "input_ids",
+        "attention_mask"
+      ],
+      "opset": 19,
+      "optimization": {
+        "compress_weight": false,
+        "enable_flash_attention": true,
+        "model_pruning": false,
+        "optimization_level": 0,
+        "remove_reshape": false,
+        "remove_weight": false,
+        "sparse_infer": false
+      },
+      "quantization": {
+        "auto_hybrid_cos_thresh": 0.98,
+        "auto_hybrid_euc_thresh": null,
+        "dataset_columns": null,
+        "dataset_name": null,
+        "dataset_size": 128,
+        "dataset_split": null,
+        "dataset_subset": null,
+        "do_quantization": false,
+        "quant_img_RGB2BGR": false,
+        "quantized_algorithm": "normal",
+        "quantized_dtype": "w8a8",
+        "quantized_hybrid_level": 0,
+        "quantized_method": "channel"
+      },
+      "rktransformers_version": "0.3.1",
+      "single_core_mode": false,
+      "std_values": null,
+      "target_platform": "rk3588",
+      "task": "feature-extraction",
+      "task_kwargs": null
+    },
+    "model_b1_s256.rknn": {
+      "batch_size": 1,
+      "custom_string": null,
+      "dynamic_input": null,
+      "float_dtype": "float16",
+      "inputs_yuv_fmt": null,
+      "max_seq_length": 256,
+      "mean_values": null,
+      "model_input_names": [
+        "input_ids",
+        "attention_mask"
+      ],
+      "opset": 19,
+      "optimization": {
+        "compress_weight": false,
+        "enable_flash_attention": true,
+        "model_pruning": false,
+        "optimization_level": 0,
+        "remove_reshape": false,
+        "remove_weight": false,
+        "sparse_infer": false
+      },
+      "quantization": {
+        "auto_hybrid_cos_thresh": 0.98,
+        "auto_hybrid_euc_thresh": null,
+        "dataset_columns": null,
+        "dataset_name": null,
+        "dataset_size": 128,
+        "dataset_split": null,
+        "dataset_subset": null,
+        "do_quantization": false,
+        "quant_img_RGB2BGR": false,
+        "quantized_algorithm": "normal",
+        "quantized_dtype": "w8a8",
+        "quantized_hybrid_level": 0,
+        "quantized_method": "channel"
+      },
+      "rktransformers_version": "0.3.1",
+      "single_core_mode": false,
+      "std_values": null,
+      "target_platform": "rk3588",
+      "task": "feature-extraction",
+      "task_kwargs": null
+    },
+    "rknn/model_b1_s1024_o1.rknn": {
+      "batch_size": 1,
+      "custom_string": null,
+      "dynamic_input": null,
+      "float_dtype": "float16",
+      "inputs_yuv_fmt": null,
+      "max_seq_length": 1024,
+      "mean_values": null,
+      "model_input_names": [
+        "input_ids",
+        "attention_mask"
+      ],
+      "opset": 19,
+      "optimization": {
+        "compress_weight": false,
+        "enable_flash_attention": true,
+        "model_pruning": false,
+        "optimization_level": 1,
+        "remove_reshape": false,
+        "remove_weight": false,
+        "sparse_infer": false
+      },
+      "quantization": {
+        "auto_hybrid_cos_thresh": 0.98,
+        "auto_hybrid_euc_thresh": null,
+        "dataset_columns": null,
+        "dataset_name": null,
+        "dataset_size": 128,
+        "dataset_split": null,
+        "dataset_subset": null,
+        "do_quantization": false,
+        "quant_img_RGB2BGR": false,
+        "quantized_algorithm": "normal",
+        "quantized_dtype": "w8a8",
+        "quantized_hybrid_level": 0,
+        "quantized_method": "channel"
+      },
+      "rktransformers_version": "0.3.1",
+      "single_core_mode": false,
+      "std_values": null,
+      "target_platform": "rk3588",
+      "task": "feature-extraction",
+      "task_kwargs": null
+    },
+    "rknn/model_b1_s1024_o2.rknn": {
+      "batch_size": 1,
+      "custom_string": null,
+      "dynamic_input": null,
+      "float_dtype": "float16",
+      "inputs_yuv_fmt": null,
+      "max_seq_length": 1024,
+      "mean_values": null,
+      "model_input_names": [
+        "input_ids",
+        "attention_mask"
+      ],
+      "opset": 19,
+      "optimization": {
+        "compress_weight": false,
+        "enable_flash_attention": true,
+        "model_pruning": false,
+        "optimization_level": 2,
+        "remove_reshape": false,
+        "remove_weight": false,
+        "sparse_infer": false
+      },
+      "quantization": {
+        "auto_hybrid_cos_thresh": 0.98,
+        "auto_hybrid_euc_thresh": null,
+        "dataset_columns": null,
+        "dataset_name": null,
+        "dataset_size": 128,
+        "dataset_split": null,
+        "dataset_subset": null,
+        "do_quantization": false,
+        "quant_img_RGB2BGR": false,
+        "quantized_algorithm": "normal",
+        "quantized_dtype": "w8a8",
+        "quantized_hybrid_level": 0,
+        "quantized_method": "channel"
+      },
+      "rktransformers_version": "0.3.1",
+      "single_core_mode": false,
+      "std_values": null,
+      "target_platform": "rk3588",
+      "task": "feature-extraction",
+      "task_kwargs": null
+    },
+    "rknn/model_b1_s1024_o3.rknn": {
+      "batch_size": 1,
+      "custom_string": null,
+      "dynamic_input": null,
+      "float_dtype": "float16",
+      "inputs_yuv_fmt": null,
+      "max_seq_length": 1024,
+      "mean_values": null,
+      "model_input_names": [
+        "input_ids",
+        "attention_mask"
+      ],
+      "opset": 19,
+      "optimization": {
+        "compress_weight": false,
+        "enable_flash_attention": true,
+        "model_pruning": false,
+        "optimization_level": 3,
+        "remove_reshape": false,
+        "remove_weight": false,
+        "sparse_infer": false
+      },
+      "quantization": {
+        "auto_hybrid_cos_thresh": 0.98,
+        "auto_hybrid_euc_thresh": null,
+        "dataset_columns": null,
+        "dataset_name": null,
+        "dataset_size": 128,
+        "dataset_split": null,
+        "dataset_subset": null,
+        "do_quantization": false,
+        "quant_img_RGB2BGR": false,
+        "quantized_algorithm": "normal",
+        "quantized_dtype": "w8a8",
+        "quantized_hybrid_level": 0,
+        "quantized_method": "channel"
+      },
+      "rktransformers_version": "0.3.1",
+      "single_core_mode": false,
+      "std_values": null,
+      "target_platform": "rk3588",
+      "task": "feature-extraction",
+      "task_kwargs": null
+    },
+    "rknn/model_b1_s1024_w8a8.rknn": {
+      "batch_size": 1,
+      "custom_string": null,
+      "dynamic_input": null,
+      "float_dtype": "float16",
+      "inputs_yuv_fmt": null,
+      "max_seq_length": 1024,
+      "mean_values": null,
+      "model_input_names": [
+        "input_ids",
+        "attention_mask"
+      ],
+      "opset": 19,
+      "optimization": {
+        "compress_weight": false,
+        "enable_flash_attention": true,
+        "model_pruning": false,
+        "optimization_level": 0,
+        "remove_reshape": false,
+        "remove_weight": false,
+        "sparse_infer": false
+      },
+      "quantization": {
+        "auto_hybrid_cos_thresh": 0.98,
+        "auto_hybrid_euc_thresh": null,
+        "dataset_columns": [
+          "answer"
+        ],
+        "dataset_name": "sentence-transformers/natural-questions",
+        "dataset_size": 256,
+        "dataset_split": [
+          "train"
+        ],
+        "dataset_subset": null,
+        "do_quantization": true,
+        "quant_img_RGB2BGR": false,
+        "quantized_algorithm": "normal",
+        "quantized_dtype": "w8a8",
+        "quantized_hybrid_level": 0,
+        "quantized_method": "channel"
+      },
+      "rktransformers_version": "0.3.1",
+      "single_core_mode": false,
+      "std_values": null,
+      "target_platform": "rk3588",
+      "task": "feature-extraction",
+      "task_kwargs": null
+    },
+    "rknn/model_o1.rknn": {
+      "batch_size": 1,
+      "custom_string": null,
+      "dynamic_input": null,
+      "float_dtype": "float16",
+      "inputs_yuv_fmt": null,
+      "max_seq_length": 512,
+      "mean_values": null,
+      "model_input_names": [
+        "input_ids",
+        "attention_mask"
+      ],
+      "opset": 19,
+      "optimization": {
+        "compress_weight": false,
+        "enable_flash_attention": true,
+        "model_pruning": false,
+        "optimization_level": 1,
+        "remove_reshape": false,
+        "remove_weight": false,
+        "sparse_infer": false
+      },
+      "quantization": {
+        "auto_hybrid_cos_thresh": 0.98,
+        "auto_hybrid_euc_thresh": null,
+        "dataset_columns": null,
+        "dataset_name": null,
+        "dataset_size": 128,
+        "dataset_split": null,
+        "dataset_subset": null,
+        "do_quantization": false,
+        "quant_img_RGB2BGR": false,
+        "quantized_algorithm": "normal",
+        "quantized_dtype": "w8a8",
+        "quantized_hybrid_level": 0,
+        "quantized_method": "channel"
+      },
+      "rktransformers_version": "0.3.1",
+      "single_core_mode": false,
+      "std_values": null,
+      "target_platform": "rk3588",
+      "task": "feature-extraction",
+      "task_kwargs": null
+    },
+    "rknn/model_o2.rknn": {
+      "batch_size": 1,
+      "custom_string": null,
+      "dynamic_input": null,
+      "float_dtype": "float16",
+      "inputs_yuv_fmt": null,
+      "max_seq_length": 512,
+      "mean_values": null,
+      "model_input_names": [
+        "input_ids",
+        "attention_mask"
+      ],
+      "opset": 19,
+      "optimization": {
+        "compress_weight": false,
+        "enable_flash_attention": true,
+        "model_pruning": false,
+        "optimization_level": 2,
+        "remove_reshape": false,
+        "remove_weight": false,
+        "sparse_infer": false
+      },
+      "quantization": {
+        "auto_hybrid_cos_thresh": 0.98,
+        "auto_hybrid_euc_thresh": null,
+        "dataset_columns": null,
+        "dataset_name": null,
+        "dataset_size": 128,
+        "dataset_split": null,
+        "dataset_subset": null,
+        "do_quantization": false,
+        "quant_img_RGB2BGR": false,
+        "quantized_algorithm": "normal",
+        "quantized_dtype": "w8a8",
+        "quantized_hybrid_level": 0,
+        "quantized_method": "channel"
+      },
+      "rktransformers_version": "0.3.1",
+      "single_core_mode": false,
+      "std_values": null,
+      "target_platform": "rk3588",
+      "task": "feature-extraction",
+      "task_kwargs": null
+    },
+    "rknn/model_o3.rknn": {
+      "batch_size": 1,
+      "custom_string": null,
+      "dynamic_input": null,
+      "float_dtype": "float16",
+      "inputs_yuv_fmt": null,
+      "max_seq_length": 512,
+      "mean_values": null,
+      "model_input_names": [
+        "input_ids",
+        "attention_mask"
+      ],
+      "opset": 19,
+      "optimization": {
+        "compress_weight": false,
+        "enable_flash_attention": true,
+        "model_pruning": false,
+        "optimization_level": 3,
+        "remove_reshape": false,
+        "remove_weight": false,
+        "sparse_infer": false
+      },
+      "quantization": {
+        "auto_hybrid_cos_thresh": 0.98,
+        "auto_hybrid_euc_thresh": null,
+        "dataset_columns": null,
+        "dataset_name": null,
+        "dataset_size": 128,
+        "dataset_split": null,
+        "dataset_subset": null,
+        "do_quantization": false,
+        "quant_img_RGB2BGR": false,
+        "quantized_algorithm": "normal",
+        "quantized_dtype": "w8a8",
+        "quantized_hybrid_level": 0,
+        "quantized_method": "channel"
+      },
+      "rktransformers_version": "0.3.1",
+      "single_core_mode": false,
+      "std_values": null,
+      "target_platform": "rk3588",
+      "task": "feature-extraction",
+      "task_kwargs": null
+    },
+    "rknn/model_w8a8.rknn": {
+      "batch_size": 1,
+      "custom_string": null,
+      "dynamic_input": null,
+      "float_dtype": "float16",
+      "inputs_yuv_fmt": null,
+      "max_seq_length": 512,
+      "mean_values": null,
+      "model_input_names": [
+        "input_ids",
+        "attention_mask"
+      ],
+      "opset": 19,
+      "optimization": {
+        "compress_weight": false,
+        "enable_flash_attention": true,
+        "model_pruning": false,
+        "optimization_level": 0,
+        "remove_reshape": false,
+        "remove_weight": false,
+        "sparse_infer": false
+      },
+      "quantization": {
+        "auto_hybrid_cos_thresh": 0.98,
+        "auto_hybrid_euc_thresh": null,
+        "dataset_columns": [
+          "answer"
+        ],
+        "dataset_name": "sentence-transformers/natural-questions",
+        "dataset_size": 256,
+        "dataset_split": [
+          "train"
+        ],
+        "dataset_subset": null,
+        "do_quantization": true,
+        "quant_img_RGB2BGR": false,
+        "quantized_algorithm": "normal",
+        "quantized_dtype": "w8a8",
+        "quantized_hybrid_level": 0,
+        "quantized_method": "channel"
+      },
+      "rktransformers_version": "0.3.1",
+      "single_core_mode": false,
+      "std_values": null,
+      "target_platform": "rk3588",
+      "task": "feature-extraction",
+      "task_kwargs": null
+    }
+  },
+  "sep_token_id": 50282,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.55.4",
+  "vocab_size": 50368
+}

model.rknn

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:59480bb74c8f27fe5f0971a8513e0b9fa211a8fe3f3349fd7f5b22bde5f02115
+size 331520806

model_b1_s1024.rknn

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:47734deb70c567d2f7c1535714bac263a5a52fb7b97c77363e01bfbcc598a2f4
+size 388226153

model_b1_s256.rknn

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b794689c544ed093e2a13b1ad5bfeb909b677fa4bd18c9f8b26d4da51fed8cc5
+size 316003110

rknn/model_b1_s1024_o1.rknn

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f597f7a3a1e24201c410dcd7b5919a0c5a024172564bd271fe892b3808768566
+size 388226153

rknn/model_b1_s1024_o2.rknn

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b82757fa7a3c28a117b940a467cf1692ce63bcc56c5858c87cabbfb5eb7df637
+size 388226153

rknn/model_b1_s1024_o3.rknn

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd3a6817885ad3723bcb2b93f26fd191dce3a4bf78355d806c51864fcf4e5863
+size 388226153

rknn/model_o1.rknn

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3968978004632e0c32cafc7345fec5b8996826920985a658b70090acba9186cc
+size 331520806

rknn/model_o2.rknn

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c1af2713e47768b9ce96c94010da6d057279ebe80b84b6c826722bc6d812982b
+size 331520806

rknn/model_o3.rknn

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c4cb89f9b310ee6c3741a43caa45427944501f91fb451813191e542230ee0f1a
+size 331520806

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,945 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "|||IP_ADDRESS|||",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "1": {
+      "content": "<|padding|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50254": {
+      "content": "                        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50255": {
+      "content": "                       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50256": {
+      "content": "                      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50257": {
+      "content": "                     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50258": {
+      "content": "                    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50259": {
+      "content": "                   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50260": {
+      "content": "                  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50261": {
+      "content": "                 ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50262": {
+      "content": "                ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50263": {
+      "content": "               ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50264": {
+      "content": "              ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50265": {
+      "content": "             ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50266": {
+      "content": "            ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50267": {
+      "content": "           ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50268": {
+      "content": "          ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50269": {
+      "content": "         ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50270": {
+      "content": "        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50271": {
+      "content": "       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50272": {
+      "content": "      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50273": {
+      "content": "     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50274": {
+      "content": "    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50275": {
+      "content": "   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50276": {
+      "content": "  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50277": {
+      "content": "|||EMAIL_ADDRESS|||",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50278": {
+      "content": "|||PHONE_NUMBER|||",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50279": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50280": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50281": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50282": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50283": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50284": {
+      "content": "[MASK]",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50285": {
+      "content": "[unused0]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50286": {
+      "content": "[unused1]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50287": {
+      "content": "[unused2]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50288": {
+      "content": "[unused3]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50289": {
+      "content": "[unused4]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50290": {
+      "content": "[unused5]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50291": {
+      "content": "[unused6]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50292": {
+      "content": "[unused7]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50293": {
+      "content": "[unused8]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50294": {
+      "content": "[unused9]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50295": {
+      "content": "[unused10]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50296": {
+      "content": "[unused11]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50297": {
+      "content": "[unused12]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50298": {
+      "content": "[unused13]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50299": {
+      "content": "[unused14]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50300": {
+      "content": "[unused15]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50301": {
+      "content": "[unused16]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50302": {
+      "content": "[unused17]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50303": {
+      "content": "[unused18]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50304": {
+      "content": "[unused19]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50305": {
+      "content": "[unused20]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50306": {
+      "content": "[unused21]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50307": {
+      "content": "[unused22]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50308": {
+      "content": "[unused23]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50309": {
+      "content": "[unused24]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50310": {
+      "content": "[unused25]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50311": {
+      "content": "[unused26]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50312": {
+      "content": "[unused27]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50313": {
+      "content": "[unused28]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50314": {
+      "content": "[unused29]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50315": {
+      "content": "[unused30]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50316": {
+      "content": "[unused31]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50317": {
+      "content": "[unused32]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50318": {
+      "content": "[unused33]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50319": {
+      "content": "[unused34]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50320": {
+      "content": "[unused35]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50321": {
+      "content": "[unused36]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50322": {
+      "content": "[unused37]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50323": {
+      "content": "[unused38]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50324": {
+      "content": "[unused39]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50325": {
+      "content": "[unused40]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50326": {
+      "content": "[unused41]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50327": {
+      "content": "[unused42]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50328": {
+      "content": "[unused43]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50329": {
+      "content": "[unused44]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50330": {
+      "content": "[unused45]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50331": {
+      "content": "[unused46]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50332": {
+      "content": "[unused47]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50333": {
+      "content": "[unused48]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50334": {
+      "content": "[unused49]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50335": {
+      "content": "[unused50]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50336": {
+      "content": "[unused51]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50337": {
+      "content": "[unused52]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50338": {
+      "content": "[unused53]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50339": {
+      "content": "[unused54]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50340": {
+      "content": "[unused55]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50341": {
+      "content": "[unused56]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50342": {
+      "content": "[unused57]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50343": {
+      "content": "[unused58]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50344": {
+      "content": "[unused59]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50345": {
+      "content": "[unused60]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50346": {
+      "content": "[unused61]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50347": {
+      "content": "[unused62]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50348": {
+      "content": "[unused63]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50349": {
+      "content": "[unused64]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50350": {
+      "content": "[unused65]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50351": {
+      "content": "[unused66]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50352": {
+      "content": "[unused67]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50353": {
+      "content": "[unused68]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50354": {
+      "content": "[unused69]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50355": {
+      "content": "[unused70]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50356": {
+      "content": "[unused71]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50357": {
+      "content": "[unused72]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50358": {
+      "content": "[unused73]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50359": {
+      "content": "[unused74]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50360": {
+      "content": "[unused75]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50361": {
+      "content": "[unused76]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50362": {
+      "content": "[unused77]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50363": {
+      "content": "[unused78]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50364": {
+      "content": "[unused79]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50365": {
+      "content": "[unused80]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50366": {
+      "content": "[unused81]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50367": {
+      "content": "[unused82]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "[UNK]"
+}