Upload folder using huggingface_hub

README.md

@@ -0,0 +1,254 @@
+---
+language: en
+license: mit
+pipeline_tag: text-classification
+library_name: transformers
+tags:
+  - spam
+  - ham
+  - email
+  - tinybert
+  - enron
+  - text-classification
+model-index:
+  - name: prancyFox/tiny-bert-enron-spam
+    results:
+      - task:
+          type: text-classification
+          name: Spam / Ham Classification
+        dataset:
+          name: Enron (processed CSV)
+          type: enron_email
+          split: test
+        metrics:
+          - name: F1 (macro)
+            type: f1
+            value: 0.7666
+          - name: ROC-AUC
+            type: roc_auc
+            value: 0.9977
+          - name: Precision (spam)
+            type: precision
+            value: 0.9954
+          - name: Recall (spam)
+            type: recall
+            value: 0.5632
+          - name: Precision (ham)
+            type: precision
+            value: 0.6875
+          - name: Recall (ham)
+            type: recall
+            value: 0.9973
+base_model: huawei-noah/TinyBERT_General_4L_312D
+---
+
+# TinyBERT Spam Classifier (Enron)
+
+A compact **TinyBERT (4-layer, 312 hidden)** model fine-tuned to classify **email text** as **spam** or **ham**.  
+Trained on an Enron-derived CSV with light email-specific cleaning (e.g., removing quoted lines and base64-like blobs).  
+Optimized for **low false positives** by default; adjust the decision threshold if you want higher spam recall.
+
+> Labels: `ham` (0) and `spam` (1)
+
+---
+
+## ✨ Quick Start
+
+```python
+from transformers import pipeline
+
+clf = pipeline(
+    "text-classification",
+    model="prancyFox/tiny-bert-enron-spam",
+    truncation=True  # recommended for long emails
+)
+
+clf("Congratulations! You won a FREE iPhone. Click here now!")
+# [{'label': 'spam', 'score': 0.98}]
+````
+
+**Batch inference**
+
+```python
+texts = [
+    "Meeting moved to 3pm, see agenda attached.",
+    "FREE gift card!!! Act now!",
+]
+preds = clf(texts, truncation=True)
+```
+
+---
+
+## 🔎 Intended Use & Limitations
+
+**Intended use**
+
+* Classifying **email bodies (and optionally subject+body)** as spam vs ham.
+* Low-latency scenarios where a small model is preferred.
+
+**Out of scope / Limitations**
+
+* Non-English email content may reduce accuracy.
+* Long threads with heavy quoting/footers can dilute signal (use truncation + cleaning).
+* Trained on Enron-style corporate emails; consumer emails may differ (consider further fine-tuning).
+
+---
+
+## 🧰 How We Preprocessed the Data
+
+Light normalization aimed at keeping semantic content:
+
+* Remove long base64-like blobs.
+* Drop quoted lines starting with `>` or `|`.
+* Optional: concatenate `Subject + "\n" + Message` when available.
+* Collapse repeated whitespace.
+
+(You can replicate similar cleaning in your serving pipeline for alignment.)
+
+---
+
+## 🏋️ Training Details
+
+* **Base model:** `huawei-noah/TinyBERT_General_4L_312D`
+* **Task:** Binary text classification (`ham`=0, `spam`=1)
+* **Tokenizer:** fast BERT tokenizer (uncased)
+* **Max length:** 256 tokens
+* **Optimizer / LR:** AdamW, learning rate `2e-5 – 5e-5` (final run `3e-5`)
+* **Batch size:** 32
+* **Epochs:** 4 (early stopping enabled)
+* **Warmup:** 10%
+* **Weight decay:** 0.01
+* **Loss:** Cross-entropy with class weighting (ham/spam balanced from label distribution). Focal loss available in the trainer.
+* **Early stopping metric:** `eval_f1`
+* **Best checkpoint:** Saved using evaluation on validation set.
+
+> Trainer script: `train/train_tinybert.py` (TinyBERT-compatible, with legacy HF support shims).
+
+---
+
+## 📊 Evaluation (Chunked Benchmark Summary)
+
+Metrics below reflect a **chunked evaluation** pass (used for long emails), where the model sees up to 512 tokens per chunk with overlap. Threshold tuned to minimize false positives:
+
+### Classification Report
+
+|         Class |  Precision |     Recall |         F1 |
+| ------------: | ---------: | ---------: | ---------: |
+|           ham |     0.6875 |     0.9973 |     0.8139 |
+|          spam |     0.9954 |     0.5632 |     0.7194 |
+| **macro avg** | **0.8414** | **0.7802** | **0.7666** |
+
+* **ROC-AUC:** 0.9977
+
+**Confusion matrix**
+
+```
+[[16500    45]
+ [ 7500  9671]]
+```
+
+**Interpretation:** The model is conservative (very few false positives on ham). If you need to catch more spam, **lower the decision threshold** (e.g., from 0.5 → 0.35) or re-train with a spam-skewed class weight / focal loss.
+
+---
+
+## 🎛️ Threshold & Long-Email Guidance
+
+* **Threshold:** Default is 0.5. For higher spam recall, try **0.35–0.45** and evaluate impact on false positives.
+* **Long emails:** For multi-paragraph threads, consider **chunking** and aggregating chunk-level spam scores (e.g., max or average). Our reference app uses 512-token chunks with overlap.
+
+---
+
+## 🧪 Reproducibility
+
+**Environment**
+
+* Python 3.10/3.11
+* `transformers >= 4.40`
+* `datasets >= 2.20`
+* `evaluate >= 0.4.2`
+* `torch >= 2.1`
+
+**Training command (example)**
+
+```bash
+python train/train_tinybert.py \
+  --train data/enron.csv \
+  --text_col Message --label_col "Spam/Ham" \
+  --output_dir outputs/tiny-bert-enron-spam \
+  --epochs 4 --batch_size 32 --lr 3e-5 \
+  --max_length 256 --fp16
+```
+
+**Serving (FastAPI example)**
+
+```bash
+python spam_bert.py --serve \
+  --model prancyFox/tiny-bert-enron-spam \
+  --model-cache-dir ./models_cache
+```
+
+---
+
+## 📁 Files
+
+This repo should include:
+
+* `config.json`
+* `pytorch_model.bin` or `model.safetensors`
+* `tokenizer.json` and `tokenizer_config.json` (or `vocab.txt` etc.)
+* `README.md` (this file)
+* (Optional) `label_mapping.json` with `{"ham": 0, "spam": 1}`
+
+---
+
+## ⚖️ License
+
+* **Model weights & code**: MIT
+* **Dataset**: Check the original Enron dataset/license terms before redistribution.
+
+---
+
+## 🔬 Ethical Considerations & Risks
+
+* False positives can have operational cost (missed legitimate emails). This model is tuned to minimize them; if you change the threshold, validate carefully.
+* Spam evolves. Periodically re-train with fresh samples to maintain accuracy.
+* Non-English or code-mixed content may degrade performance.
+
+---
+
+## 🧩 Citation
+
+If you use this model, please cite:
+
+```
+@software{tinybert_enron_spam_2025,
+  title        = {TinyBERT Spam Classifier (Enron)},
+  author       = {Ing. Daniel Eder},
+  year         = {2025},
+  url          = {https://huggingface.co/prancyFox/tiny-bert-enron-spam}
+}
+```
+
+And the TinyBERT paper:
+
+```
+@article{jiao2020tinybert,
+  title={TinyBERT: Distilling BERT for Natural Language Understanding},
+  author={Jiao, Xiaoqi and Yin, Yichun and others},
+  journal={Findings of EMNLP},
+  year={2020}
+}
+```
+
+---
+
+## 🛠 Maintainers
+
+* **Daniel Eder** ([[email protected]](mailto:[email protected]?subject=tiny-bert-enron-spam))
+
+---
+
+### Notes
+
+* For a higher-recall variant, fine-tune with `--use_focal_loss` or increase the spam class weight, then re-evaluate thresholds.
+* If you want a **PyTorch Lightning** or **Accelerate** training variant, \~it’s easy to adapt the provided trainer.

checkpoint-1000/config.json

@@ -0,0 +1,36 @@
+{
+  "architectures": [
+    "BertForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "cell": {},
+  "classifier_dropout": null,
+  "emb_size": 312,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 312,
+  "id2label": {
+    "0": "ham",
+    "1": "spam"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 1200,
+  "label2id": {
+    "ham": 0,
+    "spam": 1
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 4,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "pre_trained": "",
+  "structure": [],
+  "torch_dtype": "float32",
+  "transformers_version": "4.55.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

checkpoint-1000/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a31c635559db3cefbb53de0825299d4849ec2c98ffa325475aa5c1b93b52a599
+size 57411808

checkpoint-1000/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f0a244f0ab5d36cbb5bcb5ee4678113bf196e7b7a105c5679d641bf0cd455d4
+size 114864267

checkpoint-1000/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ba6621c267fba871ab6674a5abb65fac14a998ca389540d43eca025f378fef4a
+size 14455

checkpoint-1000/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:691bbcc71ba9a97565707d1532995eda4e57b6dbdb0849ad042a36a2c432ffa0
+size 1465

checkpoint-1000/special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

checkpoint-1000/tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

checkpoint-1000/trainer_state.json

@@ -0,0 +1,243 @@
+{
+  "best_global_step": 1000,
+  "best_metric": 0.9968069666182874,
+  "best_model_checkpoint": "models/tinybert-enron\\checkpoint-1000",
+  "epoch": 1.0559662090813093,
+  "eval_steps": 200,
+  "global_step": 1000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.05279831045406547,
+      "grad_norm": 0.7054851055145264,
+      "learning_rate": 3.878627968337731e-06,
+      "loss": 0.6916,
+      "step": 50
+    },
+    {
+      "epoch": 0.10559662090813093,
+      "grad_norm": 4.758987903594971,
+      "learning_rate": 7.836411609498681e-06,
+      "loss": 0.6419,
+      "step": 100
+    },
+    {
+      "epoch": 0.1583949313621964,
+      "grad_norm": 2.878147602081299,
+      "learning_rate": 1.179419525065963e-05,
+      "loss": 0.4382,
+      "step": 150
+    },
+    {
+      "epoch": 0.21119324181626187,
+      "grad_norm": 0.9188410639762878,
+      "learning_rate": 1.575197889182058e-05,
+      "loss": 0.2162,
+      "step": 200
+    },
+    {
+      "epoch": 0.21119324181626187,
+      "eval_accuracy": 0.997920997920998,
+      "eval_f1": 0.997965707643127,
+      "eval_loss": 0.11409526318311691,
+      "eval_precision": 0.9959396751740139,
+      "eval_recall": 1.0,
+      "eval_runtime": 77.9734,
+      "eval_samples_per_second": 43.181,
+      "eval_steps_per_second": 2.706,
+      "step": 200
+    },
+    {
+      "epoch": 0.26399155227032733,
+      "grad_norm": 0.42353588342666626,
+      "learning_rate": 1.970976253298153e-05,
+      "loss": 0.0759,
+      "step": 250
+    },
+    {
+      "epoch": 0.3167898627243928,
+      "grad_norm": 0.22129850089550018,
+      "learning_rate": 2.3667546174142482e-05,
+      "loss": 0.0466,
+      "step": 300
+    },
+    {
+      "epoch": 0.36958817317845827,
+      "grad_norm": 0.14293035864830017,
+      "learning_rate": 2.762532981530343e-05,
+      "loss": 0.0333,
+      "step": 350
+    },
+    {
+      "epoch": 0.42238648363252373,
+      "grad_norm": 0.09417829662561417,
+      "learning_rate": 2.982399530654151e-05,
+      "loss": 0.015,
+      "step": 400
+    },
+    {
+      "epoch": 0.42238648363252373,
+      "eval_accuracy": 0.9988119988119988,
+      "eval_f1": 0.9988365328679465,
+      "eval_loss": 0.012017497792840004,
+      "eval_precision": 0.9976757699012202,
+      "eval_recall": 1.0,
+      "eval_runtime": 75.9348,
+      "eval_samples_per_second": 44.341,
+      "eval_steps_per_second": 2.779,
+      "step": 400
+    },
+    {
+      "epoch": 0.4751847940865892,
+      "grad_norm": 0.06869468092918396,
+      "learning_rate": 2.938398357289528e-05,
+      "loss": 0.0143,
+      "step": 450
+    },
+    {
+      "epoch": 0.5279831045406547,
+      "grad_norm": 0.05820750445127487,
+      "learning_rate": 2.8943971839249047e-05,
+      "loss": 0.0161,
+      "step": 500
+    },
+    {
+      "epoch": 0.5807814149947201,
+      "grad_norm": 0.053672004491090775,
+      "learning_rate": 2.8503960105602817e-05,
+      "loss": 0.0107,
+      "step": 550
+    },
+    {
+      "epoch": 0.6335797254487856,
+      "grad_norm": 121.33438873291016,
+      "learning_rate": 2.8063948371956588e-05,
+      "loss": 0.0068,
+      "step": 600
+    },
+    {
+      "epoch": 0.6335797254487856,
+      "eval_accuracy": 0.9991089991089991,
+      "eval_f1": 0.999127145766657,
+      "eval_loss": 0.006898785941302776,
+      "eval_precision": 0.9982558139534884,
+      "eval_recall": 1.0,
+      "eval_runtime": 87.7806,
+      "eval_samples_per_second": 38.357,
+      "eval_steps_per_second": 2.404,
+      "step": 600
+    },
+    {
+      "epoch": 0.6863780359028511,
+      "grad_norm": 0.2175891101360321,
+      "learning_rate": 2.7623936638310355e-05,
+      "loss": 0.0123,
+      "step": 650
+    },
+    {
+      "epoch": 0.7391763463569165,
+      "grad_norm": 0.031244030222296715,
+      "learning_rate": 2.7183924904664125e-05,
+      "loss": 0.0087,
+      "step": 700
+    },
+    {
+      "epoch": 0.791974656810982,
+      "grad_norm": 0.028063887730240822,
+      "learning_rate": 2.6743913171017896e-05,
+      "loss": 0.0122,
+      "step": 750
+    },
+    {
+      "epoch": 0.8447729672650475,
+      "grad_norm": 0.025719981640577316,
+      "learning_rate": 2.6303901437371663e-05,
+      "loss": 0.0077,
+      "step": 800
+    },
+    {
+      "epoch": 0.8447729672650475,
+      "eval_accuracy": 0.9994059994059994,
+      "eval_f1": 0.9994179278230501,
+      "eval_loss": 0.004702265374362469,
+      "eval_precision": 0.9988365328679465,
+      "eval_recall": 1.0,
+      "eval_runtime": 99.4766,
+      "eval_samples_per_second": 33.847,
+      "eval_steps_per_second": 2.121,
+      "step": 800
+    },
+    {
+      "epoch": 0.8975712777191129,
+      "grad_norm": 0.02354113757610321,
+      "learning_rate": 2.5863889703725433e-05,
+      "loss": 0.0058,
+      "step": 850
+    },
+    {
+      "epoch": 0.9503695881731784,
+      "grad_norm": 0.02310008369386196,
+      "learning_rate": 2.5423877970079204e-05,
+      "loss": 0.0056,
+      "step": 900
+    },
+    {
+      "epoch": 1.0031678986272439,
+      "grad_norm": 0.019235238432884216,
+      "learning_rate": 2.498386623643297e-05,
+      "loss": 0.0106,
+      "step": 950
+    },
+    {
+      "epoch": 1.0559662090813093,
+      "grad_norm": 208.1472930908203,
+      "learning_rate": 2.454385450278674e-05,
+      "loss": 0.0124,
+      "step": 1000
+    },
+    {
+      "epoch": 1.0559662090813093,
+      "eval_accuracy": 0.9967329967329968,
+      "eval_f1": 0.9968069666182874,
+      "eval_loss": 0.016226448118686676,
+      "eval_precision": 0.9936342592592593,
+      "eval_recall": 1.0,
+      "eval_runtime": 88.1246,
+      "eval_samples_per_second": 38.207,
+      "eval_steps_per_second": 2.394,
+      "step": 1000
+    }
+  ],
+  "logging_steps": 50,
+  "max_steps": 3788,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 4,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "EarlyStoppingCallback": {
+      "args": {
+        "early_stopping_patience": 5,
+        "early_stopping_threshold": 0.0005
+      },
+      "attributes": {
+        "early_stopping_patience_counter": 0
+      }
+    },
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 229373753097216.0,
+  "train_batch_size": 32,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1000/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0f6ee5be6db1ab816abaa77671d6a299c7d2015f383c82c395377bcfdce9d1cd
+size 5713

checkpoint-1200/config.json

@@ -0,0 +1,36 @@
+{
+  "architectures": [
+    "BertForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "cell": {},
+  "classifier_dropout": null,
+  "emb_size": 312,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 312,
+  "id2label": {
+    "0": "ham",
+    "1": "spam"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 1200,
+  "label2id": {
+    "ham": 0,
+    "spam": 1
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 4,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "pre_trained": "",
+  "structure": [],
+  "torch_dtype": "float32",
+  "transformers_version": "4.55.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

checkpoint-1200/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ed481638e470979590a6a0c03f9237e2ef00619bfde08b505adc569fc3f3e70
+size 57411808

checkpoint-1200/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:54b3ca65cdc181c36eb175979422431446d89583bba5da2fcdccca2ae639bb02
+size 114864267

checkpoint-1200/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:86b4a86c121d70b2aa0ea130d6238e100bb2801dea0fa773b97fc0541331b11a
+size 14455

checkpoint-1200/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:60b4839f57741fdc7ac5a9d26657752bea98ceb92dd0989c5b64592f830ab0aa
+size 1465

checkpoint-1200/special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

checkpoint-1200/tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

checkpoint-1200/trainer_state.json

@@ -0,0 +1,283 @@
+{
+  "best_global_step": 1000,
+  "best_metric": 0.9968069666182874,
+  "best_model_checkpoint": "models/tinybert-enron\\checkpoint-1000",
+  "epoch": 1.2671594508975712,
+  "eval_steps": 200,
+  "global_step": 1200,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.05279831045406547,
+      "grad_norm": 0.7054851055145264,
+      "learning_rate": 3.878627968337731e-06,
+      "loss": 0.6916,
+      "step": 50
+    },
+    {
+      "epoch": 0.10559662090813093,
+      "grad_norm": 4.758987903594971,
+      "learning_rate": 7.836411609498681e-06,
+      "loss": 0.6419,
+      "step": 100
+    },
+    {
+      "epoch": 0.1583949313621964,
+      "grad_norm": 2.878147602081299,
+      "learning_rate": 1.179419525065963e-05,
+      "loss": 0.4382,
+      "step": 150
+    },
+    {
+      "epoch": 0.21119324181626187,
+      "grad_norm": 0.9188410639762878,
+      "learning_rate": 1.575197889182058e-05,
+      "loss": 0.2162,
+      "step": 200
+    },
+    {
+      "epoch": 0.21119324181626187,
+      "eval_accuracy": 0.997920997920998,
+      "eval_f1": 0.997965707643127,
+      "eval_loss": 0.11409526318311691,
+      "eval_precision": 0.9959396751740139,
+      "eval_recall": 1.0,
+      "eval_runtime": 77.9734,
+      "eval_samples_per_second": 43.181,
+      "eval_steps_per_second": 2.706,
+      "step": 200
+    },
+    {
+      "epoch": 0.26399155227032733,
+      "grad_norm": 0.42353588342666626,
+      "learning_rate": 1.970976253298153e-05,
+      "loss": 0.0759,
+      "step": 250
+    },
+    {
+      "epoch": 0.3167898627243928,
+      "grad_norm": 0.22129850089550018,
+      "learning_rate": 2.3667546174142482e-05,
+      "loss": 0.0466,
+      "step": 300
+    },
+    {
+      "epoch": 0.36958817317845827,
+      "grad_norm": 0.14293035864830017,
+      "learning_rate": 2.762532981530343e-05,
+      "loss": 0.0333,
+      "step": 350
+    },
+    {
+      "epoch": 0.42238648363252373,
+      "grad_norm": 0.09417829662561417,
+      "learning_rate": 2.982399530654151e-05,
+      "loss": 0.015,
+      "step": 400
+    },
+    {
+      "epoch": 0.42238648363252373,
+      "eval_accuracy": 0.9988119988119988,
+      "eval_f1": 0.9988365328679465,
+      "eval_loss": 0.012017497792840004,
+      "eval_precision": 0.9976757699012202,
+      "eval_recall": 1.0,
+      "eval_runtime": 75.9348,
+      "eval_samples_per_second": 44.341,
+      "eval_steps_per_second": 2.779,
+      "step": 400
+    },
+    {
+      "epoch": 0.4751847940865892,
+      "grad_norm": 0.06869468092918396,
+      "learning_rate": 2.938398357289528e-05,
+      "loss": 0.0143,
+      "step": 450
+    },
+    {
+      "epoch": 0.5279831045406547,
+      "grad_norm": 0.05820750445127487,
+      "learning_rate": 2.8943971839249047e-05,
+      "loss": 0.0161,
+      "step": 500
+    },
+    {
+      "epoch": 0.5807814149947201,
+      "grad_norm": 0.053672004491090775,
+      "learning_rate": 2.8503960105602817e-05,
+      "loss": 0.0107,
+      "step": 550
+    },
+    {
+      "epoch": 0.6335797254487856,
+      "grad_norm": 121.33438873291016,
+      "learning_rate": 2.8063948371956588e-05,
+      "loss": 0.0068,
+      "step": 600
+    },
+    {
+      "epoch": 0.6335797254487856,
+      "eval_accuracy": 0.9991089991089991,
+      "eval_f1": 0.999127145766657,
+      "eval_loss": 0.006898785941302776,
+      "eval_precision": 0.9982558139534884,
+      "eval_recall": 1.0,
+      "eval_runtime": 87.7806,
+      "eval_samples_per_second": 38.357,
+      "eval_steps_per_second": 2.404,
+      "step": 600
+    },
+    {
+      "epoch": 0.6863780359028511,
+      "grad_norm": 0.2175891101360321,
+      "learning_rate": 2.7623936638310355e-05,
+      "loss": 0.0123,
+      "step": 650
+    },
+    {
+      "epoch": 0.7391763463569165,
+      "grad_norm": 0.031244030222296715,
+      "learning_rate": 2.7183924904664125e-05,
+      "loss": 0.0087,
+      "step": 700
+    },
+    {
+      "epoch": 0.791974656810982,
+      "grad_norm": 0.028063887730240822,
+      "learning_rate": 2.6743913171017896e-05,
+      "loss": 0.0122,
+      "step": 750
+    },
+    {
+      "epoch": 0.8447729672650475,
+      "grad_norm": 0.025719981640577316,
+      "learning_rate": 2.6303901437371663e-05,
+      "loss": 0.0077,
+      "step": 800
+    },
+    {
+      "epoch": 0.8447729672650475,
+      "eval_accuracy": 0.9994059994059994,
+      "eval_f1": 0.9994179278230501,
+      "eval_loss": 0.004702265374362469,
+      "eval_precision": 0.9988365328679465,
+      "eval_recall": 1.0,
+      "eval_runtime": 99.4766,
+      "eval_samples_per_second": 33.847,
+      "eval_steps_per_second": 2.121,
+      "step": 800
+    },
+    {
+      "epoch": 0.8975712777191129,
+      "grad_norm": 0.02354113757610321,
+      "learning_rate": 2.5863889703725433e-05,
+      "loss": 0.0058,
+      "step": 850
+    },
+    {
+      "epoch": 0.9503695881731784,
+      "grad_norm": 0.02310008369386196,
+      "learning_rate": 2.5423877970079204e-05,
+      "loss": 0.0056,
+      "step": 900
+    },
+    {
+      "epoch": 1.0031678986272439,
+      "grad_norm": 0.019235238432884216,
+      "learning_rate": 2.498386623643297e-05,
+      "loss": 0.0106,
+      "step": 950
+    },
+    {
+      "epoch": 1.0559662090813093,
+      "grad_norm": 208.1472930908203,
+      "learning_rate": 2.454385450278674e-05,
+      "loss": 0.0124,
+      "step": 1000
+    },
+    {
+      "epoch": 1.0559662090813093,
+      "eval_accuracy": 0.9967329967329968,
+      "eval_f1": 0.9968069666182874,
+      "eval_loss": 0.016226448118686676,
+      "eval_precision": 0.9936342592592593,
+      "eval_recall": 1.0,
+      "eval_runtime": 88.1246,
+      "eval_samples_per_second": 38.207,
+      "eval_steps_per_second": 2.394,
+      "step": 1000
+    },
+    {
+      "epoch": 1.1087645195353748,
+      "grad_norm": 0.01713796705007553,
+      "learning_rate": 2.4103842769140512e-05,
+      "loss": 0.0012,
+      "step": 1050
+    },
+    {
+      "epoch": 1.1615628299894403,
+      "grad_norm": 0.015752457082271576,
+      "learning_rate": 2.366383103549428e-05,
+      "loss": 0.001,
+      "step": 1100
+    },
+    {
+      "epoch": 1.2143611404435057,
+      "grad_norm": 0.014375035651028156,
+      "learning_rate": 2.322381930184805e-05,
+      "loss": 0.0054,
+      "step": 1150
+    },
+    {
+      "epoch": 1.2671594508975712,
+      "grad_norm": 0.014459795318543911,
+      "learning_rate": 2.278380756820182e-05,
+      "loss": 0.0009,
+      "step": 1200
+    },
+    {
+      "epoch": 1.2671594508975712,
+      "eval_accuracy": 0.9997029997029997,
+      "eval_f1": 0.9997088791848617,
+      "eval_loss": 0.0029172594659030437,
+      "eval_precision": 0.9994179278230501,
+      "eval_recall": 1.0,
+      "eval_runtime": 70.0694,
+      "eval_samples_per_second": 48.052,
+      "eval_steps_per_second": 3.011,
+      "step": 1200
+    }
+  ],
+  "logging_steps": 50,
+  "max_steps": 3788,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 4,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "EarlyStoppingCallback": {
+      "args": {
+        "early_stopping_patience": 5,
+        "early_stopping_threshold": 0.0005
+      },
+      "attributes": {
+        "early_stopping_patience_counter": 1
+      }
+    },
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 275258541014016.0,
+  "train_batch_size": 32,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1200/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0f6ee5be6db1ab816abaa77671d6a299c7d2015f383c82c395377bcfdce9d1cd
+size 5713

checkpoint-1400/config.json

@@ -0,0 +1,36 @@
+{
+  "architectures": [
+    "BertForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "cell": {},
+  "classifier_dropout": null,
+  "emb_size": 312,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 312,
+  "id2label": {
+    "0": "ham",
+    "1": "spam"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 1200,
+  "label2id": {
+    "ham": 0,
+    "spam": 1
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 4,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "pre_trained": "",
+  "structure": [],
+  "torch_dtype": "float32",
+  "transformers_version": "4.55.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

checkpoint-1400/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:afa8912f674d23e64c0e45a67afffeed25fcbce273fdc8906b4d241b5954833d
+size 57411808

checkpoint-1400/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f741ecb400556d723d2d0a88f1211dbdb7b894c40a68fe2ebed38b0c194fae7d
+size 114864267

checkpoint-1400/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:abef661c396d70ac459e7ba6608cac0365c9ffeb4d0e6dada67d33354417afdd
+size 14455

checkpoint-1400/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:484087520d4e9e312982d2bfa6b0a475c2703200e2eb36812500de0a364fed26
+size 1465

checkpoint-1400/special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

checkpoint-1400/tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

checkpoint-1400/trainer_state.json

@@ -0,0 +1,323 @@
+{
+  "best_global_step": 1000,
+  "best_metric": 0.9968069666182874,
+  "best_model_checkpoint": "models/tinybert-enron\\checkpoint-1000",
+  "epoch": 1.478352692713833,
+  "eval_steps": 200,
+  "global_step": 1400,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.05279831045406547,
+      "grad_norm": 0.7054851055145264,
+      "learning_rate": 3.878627968337731e-06,
+      "loss": 0.6916,
+      "step": 50
+    },
+    {
+      "epoch": 0.10559662090813093,
+      "grad_norm": 4.758987903594971,
+      "learning_rate": 7.836411609498681e-06,
+      "loss": 0.6419,
+      "step": 100
+    },
+    {
+      "epoch": 0.1583949313621964,
+      "grad_norm": 2.878147602081299,
+      "learning_rate": 1.179419525065963e-05,
+      "loss": 0.4382,
+      "step": 150
+    },
+    {
+      "epoch": 0.21119324181626187,
+      "grad_norm": 0.9188410639762878,
+      "learning_rate": 1.575197889182058e-05,
+      "loss": 0.2162,
+      "step": 200
+    },
+    {
+      "epoch": 0.21119324181626187,
+      "eval_accuracy": 0.997920997920998,
+      "eval_f1": 0.997965707643127,
+      "eval_loss": 0.11409526318311691,
+      "eval_precision": 0.9959396751740139,
+      "eval_recall": 1.0,
+      "eval_runtime": 77.9734,
+      "eval_samples_per_second": 43.181,
+      "eval_steps_per_second": 2.706,
+      "step": 200
+    },
+    {
+      "epoch": 0.26399155227032733,
+      "grad_norm": 0.42353588342666626,
+      "learning_rate": 1.970976253298153e-05,
+      "loss": 0.0759,
+      "step": 250
+    },
+    {
+      "epoch": 0.3167898627243928,
+      "grad_norm": 0.22129850089550018,
+      "learning_rate": 2.3667546174142482e-05,
+      "loss": 0.0466,
+      "step": 300
+    },
+    {
+      "epoch": 0.36958817317845827,
+      "grad_norm": 0.14293035864830017,
+      "learning_rate": 2.762532981530343e-05,
+      "loss": 0.0333,
+      "step": 350
+    },
+    {
+      "epoch": 0.42238648363252373,
+      "grad_norm": 0.09417829662561417,
+      "learning_rate": 2.982399530654151e-05,
+      "loss": 0.015,
+      "step": 400
+    },
+    {
+      "epoch": 0.42238648363252373,
+      "eval_accuracy": 0.9988119988119988,
+      "eval_f1": 0.9988365328679465,
+      "eval_loss": 0.012017497792840004,
+      "eval_precision": 0.9976757699012202,
+      "eval_recall": 1.0,
+      "eval_runtime": 75.9348,
+      "eval_samples_per_second": 44.341,
+      "eval_steps_per_second": 2.779,
+      "step": 400
+    },
+    {
+      "epoch": 0.4751847940865892,
+      "grad_norm": 0.06869468092918396,
+      "learning_rate": 2.938398357289528e-05,
+      "loss": 0.0143,
+      "step": 450
+    },
+    {
+      "epoch": 0.5279831045406547,
+      "grad_norm": 0.05820750445127487,
+      "learning_rate": 2.8943971839249047e-05,
+      "loss": 0.0161,
+      "step": 500
+    },
+    {
+      "epoch": 0.5807814149947201,
+      "grad_norm": 0.053672004491090775,
+      "learning_rate": 2.8503960105602817e-05,
+      "loss": 0.0107,
+      "step": 550
+    },
+    {
+      "epoch": 0.6335797254487856,
+      "grad_norm": 121.33438873291016,
+      "learning_rate": 2.8063948371956588e-05,
+      "loss": 0.0068,
+      "step": 600
+    },
+    {
+      "epoch": 0.6335797254487856,
+      "eval_accuracy": 0.9991089991089991,
+      "eval_f1": 0.999127145766657,
+      "eval_loss": 0.006898785941302776,
+      "eval_precision": 0.9982558139534884,
+      "eval_recall": 1.0,
+      "eval_runtime": 87.7806,
+      "eval_samples_per_second": 38.357,
+      "eval_steps_per_second": 2.404,
+      "step": 600
+    },
+    {
+      "epoch": 0.6863780359028511,
+      "grad_norm": 0.2175891101360321,
+      "learning_rate": 2.7623936638310355e-05,
+      "loss": 0.0123,
+      "step": 650
+    },
+    {
+      "epoch": 0.7391763463569165,
+      "grad_norm": 0.031244030222296715,
+      "learning_rate": 2.7183924904664125e-05,
+      "loss": 0.0087,
+      "step": 700
+    },
+    {
+      "epoch": 0.791974656810982,
+      "grad_norm": 0.028063887730240822,
+      "learning_rate": 2.6743913171017896e-05,
+      "loss": 0.0122,
+      "step": 750
+    },
+    {
+      "epoch": 0.8447729672650475,
+      "grad_norm": 0.025719981640577316,
+      "learning_rate": 2.6303901437371663e-05,
+      "loss": 0.0077,
+      "step": 800
+    },
+    {
+      "epoch": 0.8447729672650475,
+      "eval_accuracy": 0.9994059994059994,
+      "eval_f1": 0.9994179278230501,
+      "eval_loss": 0.004702265374362469,
+      "eval_precision": 0.9988365328679465,
+      "eval_recall": 1.0,
+      "eval_runtime": 99.4766,
+      "eval_samples_per_second": 33.847,
+      "eval_steps_per_second": 2.121,
+      "step": 800
+    },
+    {
+      "epoch": 0.8975712777191129,
+      "grad_norm": 0.02354113757610321,
+      "learning_rate": 2.5863889703725433e-05,
+      "loss": 0.0058,
+      "step": 850
+    },
+    {
+      "epoch": 0.9503695881731784,
+      "grad_norm": 0.02310008369386196,
+      "learning_rate": 2.5423877970079204e-05,
+      "loss": 0.0056,
+      "step": 900
+    },
+    {
+      "epoch": 1.0031678986272439,
+      "grad_norm": 0.019235238432884216,
+      "learning_rate": 2.498386623643297e-05,
+      "loss": 0.0106,
+      "step": 950
+    },
+    {
+      "epoch": 1.0559662090813093,
+      "grad_norm": 208.1472930908203,
+      "learning_rate": 2.454385450278674e-05,
+      "loss": 0.0124,
+      "step": 1000
+    },
+    {
+      "epoch": 1.0559662090813093,
+      "eval_accuracy": 0.9967329967329968,
+      "eval_f1": 0.9968069666182874,
+      "eval_loss": 0.016226448118686676,
+      "eval_precision": 0.9936342592592593,
+      "eval_recall": 1.0,
+      "eval_runtime": 88.1246,
+      "eval_samples_per_second": 38.207,
+      "eval_steps_per_second": 2.394,
+      "step": 1000
+    },
+    {
+      "epoch": 1.1087645195353748,
+      "grad_norm": 0.01713796705007553,
+      "learning_rate": 2.4103842769140512e-05,
+      "loss": 0.0012,
+      "step": 1050
+    },
+    {
+      "epoch": 1.1615628299894403,
+      "grad_norm": 0.015752457082271576,
+      "learning_rate": 2.366383103549428e-05,
+      "loss": 0.001,
+      "step": 1100
+    },
+    {
+      "epoch": 1.2143611404435057,
+      "grad_norm": 0.014375035651028156,
+      "learning_rate": 2.322381930184805e-05,
+      "loss": 0.0054,
+      "step": 1150
+    },
+    {
+      "epoch": 1.2671594508975712,
+      "grad_norm": 0.014459795318543911,
+      "learning_rate": 2.278380756820182e-05,
+      "loss": 0.0009,
+      "step": 1200
+    },
+    {
+      "epoch": 1.2671594508975712,
+      "eval_accuracy": 0.9997029997029997,
+      "eval_f1": 0.9997088791848617,
+      "eval_loss": 0.0029172594659030437,
+      "eval_precision": 0.9994179278230501,
+      "eval_recall": 1.0,
+      "eval_runtime": 70.0694,
+      "eval_samples_per_second": 48.052,
+      "eval_steps_per_second": 3.011,
+      "step": 1200
+    },
+    {
+      "epoch": 1.3199577613516367,
+      "grad_norm": 0.012884082272648811,
+      "learning_rate": 2.2343795834555587e-05,
+      "loss": 0.0008,
+      "step": 1250
+    },
+    {
+      "epoch": 1.3727560718057021,
+      "grad_norm": 0.013439378701150417,
+      "learning_rate": 2.1903784100909357e-05,
+      "loss": 0.0127,
+      "step": 1300
+    },
+    {
+      "epoch": 1.4255543822597676,
+      "grad_norm": 0.014692062512040138,
+      "learning_rate": 2.1463772367263128e-05,
+      "loss": 0.0101,
+      "step": 1350
+    },
+    {
+      "epoch": 1.478352692713833,
+      "grad_norm": 0.012950174510478973,
+      "learning_rate": 2.10237606336169e-05,
+      "loss": 0.0054,
+      "step": 1400
+    },
+    {
+      "epoch": 1.478352692713833,
+      "eval_accuracy": 0.9997029997029997,
+      "eval_f1": 0.9997088791848617,
+      "eval_loss": 0.002878110622987151,
+      "eval_precision": 0.9994179278230501,
+      "eval_recall": 1.0,
+      "eval_runtime": 82.6217,
+      "eval_samples_per_second": 40.752,
+      "eval_steps_per_second": 2.554,
+      "step": 1400
+    }
+  ],
+  "logging_steps": 50,
+  "max_steps": 3788,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 4,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "EarlyStoppingCallback": {
+      "args": {
+        "early_stopping_patience": 5,
+        "early_stopping_threshold": 0.0005
+      },
+      "attributes": {
+        "early_stopping_patience_counter": 2
+      }
+    },
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 321143328930816.0,
+  "train_batch_size": 32,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1400/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0f6ee5be6db1ab816abaa77671d6a299c7d2015f383c82c395377bcfdce9d1cd
+size 5713

checkpoint-1600/config.json

@@ -0,0 +1,36 @@
+{
+  "architectures": [
+    "BertForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "cell": {},
+  "classifier_dropout": null,
+  "emb_size": 312,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 312,
+  "id2label": {
+    "0": "ham",
+    "1": "spam"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 1200,
+  "label2id": {
+    "ham": 0,
+    "spam": 1
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 4,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "pre_trained": "",
+  "structure": [],
+  "torch_dtype": "float32",
+  "transformers_version": "4.55.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

checkpoint-1600/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2cf7c7dc0c5367beaf50e6e894a369c265343db0b965e64596e70e2c3dbc8811
+size 57411808

checkpoint-1600/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7de1b059ed6bbe31af0fe288bce87f379dfe5eb52d6a06f069975e78a8e22e63
+size 114864267

checkpoint-1600/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:281de45a32b26fc7fabb959de798e1b3fd70e15a16525a06643ef973b8c12bfa
+size 14455

checkpoint-1600/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:825169d8720681852b22e97bf6d4c178b22717216530e5ce830bdd78c7efdd37
+size 1465

checkpoint-1600/special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

checkpoint-1600/tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

checkpoint-1600/trainer_state.json

@@ -0,0 +1,363 @@
+{
+  "best_global_step": 1000,
+  "best_metric": 0.9968069666182874,
+  "best_model_checkpoint": "models/tinybert-enron\\checkpoint-1000",
+  "epoch": 1.689545934530095,
+  "eval_steps": 200,
+  "global_step": 1600,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.05279831045406547,
+      "grad_norm": 0.7054851055145264,
+      "learning_rate": 3.878627968337731e-06,
+      "loss": 0.6916,
+      "step": 50
+    },
+    {
+      "epoch": 0.10559662090813093,
+      "grad_norm": 4.758987903594971,
+      "learning_rate": 7.836411609498681e-06,
+      "loss": 0.6419,
+      "step": 100
+    },
+    {
+      "epoch": 0.1583949313621964,
+      "grad_norm": 2.878147602081299,
+      "learning_rate": 1.179419525065963e-05,
+      "loss": 0.4382,
+      "step": 150
+    },
+    {
+      "epoch": 0.21119324181626187,
+      "grad_norm": 0.9188410639762878,
+      "learning_rate": 1.575197889182058e-05,
+      "loss": 0.2162,
+      "step": 200
+    },
+    {
+      "epoch": 0.21119324181626187,
+      "eval_accuracy": 0.997920997920998,
+      "eval_f1": 0.997965707643127,
+      "eval_loss": 0.11409526318311691,
+      "eval_precision": 0.9959396751740139,
+      "eval_recall": 1.0,
+      "eval_runtime": 77.9734,
+      "eval_samples_per_second": 43.181,
+      "eval_steps_per_second": 2.706,
+      "step": 200
+    },
+    {
+      "epoch": 0.26399155227032733,
+      "grad_norm": 0.42353588342666626,
+      "learning_rate": 1.970976253298153e-05,
+      "loss": 0.0759,
+      "step": 250
+    },
+    {
+      "epoch": 0.3167898627243928,
+      "grad_norm": 0.22129850089550018,
+      "learning_rate": 2.3667546174142482e-05,
+      "loss": 0.0466,
+      "step": 300
+    },
+    {
+      "epoch": 0.36958817317845827,
+      "grad_norm": 0.14293035864830017,
+      "learning_rate": 2.762532981530343e-05,
+      "loss": 0.0333,
+      "step": 350
+    },
+    {
+      "epoch": 0.42238648363252373,
+      "grad_norm": 0.09417829662561417,
+      "learning_rate": 2.982399530654151e-05,
+      "loss": 0.015,
+      "step": 400
+    },
+    {
+      "epoch": 0.42238648363252373,
+      "eval_accuracy": 0.9988119988119988,
+      "eval_f1": 0.9988365328679465,
+      "eval_loss": 0.012017497792840004,
+      "eval_precision": 0.9976757699012202,
+      "eval_recall": 1.0,
+      "eval_runtime": 75.9348,
+      "eval_samples_per_second": 44.341,
+      "eval_steps_per_second": 2.779,
+      "step": 400
+    },
+    {
+      "epoch": 0.4751847940865892,
+      "grad_norm": 0.06869468092918396,
+      "learning_rate": 2.938398357289528e-05,
+      "loss": 0.0143,
+      "step": 450
+    },
+    {
+      "epoch": 0.5279831045406547,
+      "grad_norm": 0.05820750445127487,
+      "learning_rate": 2.8943971839249047e-05,
+      "loss": 0.0161,
+      "step": 500
+    },
+    {
+      "epoch": 0.5807814149947201,
+      "grad_norm": 0.053672004491090775,
+      "learning_rate": 2.8503960105602817e-05,
+      "loss": 0.0107,
+      "step": 550
+    },
+    {
+      "epoch": 0.6335797254487856,
+      "grad_norm": 121.33438873291016,
+      "learning_rate": 2.8063948371956588e-05,
+      "loss": 0.0068,
+      "step": 600
+    },
+    {
+      "epoch": 0.6335797254487856,
+      "eval_accuracy": 0.9991089991089991,
+      "eval_f1": 0.999127145766657,
+      "eval_loss": 0.006898785941302776,
+      "eval_precision": 0.9982558139534884,
+      "eval_recall": 1.0,
+      "eval_runtime": 87.7806,
+      "eval_samples_per_second": 38.357,
+      "eval_steps_per_second": 2.404,
+      "step": 600
+    },
+    {
+      "epoch": 0.6863780359028511,
+      "grad_norm": 0.2175891101360321,
+      "learning_rate": 2.7623936638310355e-05,
+      "loss": 0.0123,
+      "step": 650
+    },
+    {
+      "epoch": 0.7391763463569165,
+      "grad_norm": 0.031244030222296715,
+      "learning_rate": 2.7183924904664125e-05,
+      "loss": 0.0087,
+      "step": 700
+    },
+    {
+      "epoch": 0.791974656810982,
+      "grad_norm": 0.028063887730240822,
+      "learning_rate": 2.6743913171017896e-05,
+      "loss": 0.0122,
+      "step": 750
+    },
+    {
+      "epoch": 0.8447729672650475,
+      "grad_norm": 0.025719981640577316,
+      "learning_rate": 2.6303901437371663e-05,
+      "loss": 0.0077,
+      "step": 800
+    },
+    {
+      "epoch": 0.8447729672650475,
+      "eval_accuracy": 0.9994059994059994,
+      "eval_f1": 0.9994179278230501,
+      "eval_loss": 0.004702265374362469,
+      "eval_precision": 0.9988365328679465,
+      "eval_recall": 1.0,
+      "eval_runtime": 99.4766,
+      "eval_samples_per_second": 33.847,
+      "eval_steps_per_second": 2.121,
+      "step": 800
+    },
+    {
+      "epoch": 0.8975712777191129,
+      "grad_norm": 0.02354113757610321,
+      "learning_rate": 2.5863889703725433e-05,
+      "loss": 0.0058,
+      "step": 850
+    },
+    {
+      "epoch": 0.9503695881731784,
+      "grad_norm": 0.02310008369386196,
+      "learning_rate": 2.5423877970079204e-05,
+      "loss": 0.0056,
+      "step": 900
+    },
+    {
+      "epoch": 1.0031678986272439,
+      "grad_norm": 0.019235238432884216,
+      "learning_rate": 2.498386623643297e-05,
+      "loss": 0.0106,
+      "step": 950
+    },
+    {
+      "epoch": 1.0559662090813093,
+      "grad_norm": 208.1472930908203,
+      "learning_rate": 2.454385450278674e-05,
+      "loss": 0.0124,
+      "step": 1000
+    },
+    {
+      "epoch": 1.0559662090813093,
+      "eval_accuracy": 0.9967329967329968,
+      "eval_f1": 0.9968069666182874,
+      "eval_loss": 0.016226448118686676,
+      "eval_precision": 0.9936342592592593,
+      "eval_recall": 1.0,
+      "eval_runtime": 88.1246,
+      "eval_samples_per_second": 38.207,
+      "eval_steps_per_second": 2.394,
+      "step": 1000
+    },
+    {
+      "epoch": 1.1087645195353748,
+      "grad_norm": 0.01713796705007553,
+      "learning_rate": 2.4103842769140512e-05,
+      "loss": 0.0012,
+      "step": 1050
+    },
+    {
+      "epoch": 1.1615628299894403,
+      "grad_norm": 0.015752457082271576,
+      "learning_rate": 2.366383103549428e-05,
+      "loss": 0.001,
+      "step": 1100
+    },
+    {
+      "epoch": 1.2143611404435057,
+      "grad_norm": 0.014375035651028156,
+      "learning_rate": 2.322381930184805e-05,
+      "loss": 0.0054,
+      "step": 1150
+    },
+    {
+      "epoch": 1.2671594508975712,
+      "grad_norm": 0.014459795318543911,
+      "learning_rate": 2.278380756820182e-05,
+      "loss": 0.0009,
+      "step": 1200
+    },
+    {
+      "epoch": 1.2671594508975712,
+      "eval_accuracy": 0.9997029997029997,
+      "eval_f1": 0.9997088791848617,
+      "eval_loss": 0.0029172594659030437,
+      "eval_precision": 0.9994179278230501,
+      "eval_recall": 1.0,
+      "eval_runtime": 70.0694,
+      "eval_samples_per_second": 48.052,
+      "eval_steps_per_second": 3.011,
+      "step": 1200
+    },
+    {
+      "epoch": 1.3199577613516367,
+      "grad_norm": 0.012884082272648811,
+      "learning_rate": 2.2343795834555587e-05,
+      "loss": 0.0008,
+      "step": 1250
+    },
+    {
+      "epoch": 1.3727560718057021,
+      "grad_norm": 0.013439378701150417,
+      "learning_rate": 2.1903784100909357e-05,
+      "loss": 0.0127,
+      "step": 1300
+    },
+    {
+      "epoch": 1.4255543822597676,
+      "grad_norm": 0.014692062512040138,
+      "learning_rate": 2.1463772367263128e-05,
+      "loss": 0.0101,
+      "step": 1350
+    },
+    {
+      "epoch": 1.478352692713833,
+      "grad_norm": 0.012950174510478973,
+      "learning_rate": 2.10237606336169e-05,
+      "loss": 0.0054,
+      "step": 1400
+    },
+    {
+      "epoch": 1.478352692713833,
+      "eval_accuracy": 0.9997029997029997,
+      "eval_f1": 0.9997088791848617,
+      "eval_loss": 0.002878110622987151,
+      "eval_precision": 0.9994179278230501,
+      "eval_recall": 1.0,
+      "eval_runtime": 82.6217,
+      "eval_samples_per_second": 40.752,
+      "eval_steps_per_second": 2.554,
+      "step": 1400
+    },
+    {
+      "epoch": 1.5311510031678988,
+      "grad_norm": 0.01537719089537859,
+      "learning_rate": 2.0583748899970665e-05,
+      "loss": 0.0053,
+      "step": 1450
+    },
+    {
+      "epoch": 1.583949313621964,
+      "grad_norm": 0.01542325783520937,
+      "learning_rate": 2.0143737166324436e-05,
+      "loss": 0.0053,
+      "step": 1500
+    },
+    {
+      "epoch": 1.6367476240760297,
+      "grad_norm": 0.012001622468233109,
+      "learning_rate": 1.9703725432678206e-05,
+      "loss": 0.0061,
+      "step": 1550
+    },
+    {
+      "epoch": 1.689545934530095,
+      "grad_norm": 0.015623683109879494,
+      "learning_rate": 1.9263713699031974e-05,
+      "loss": 0.0007,
+      "step": 1600
+    },
+    {
+      "epoch": 1.689545934530095,
+      "eval_accuracy": 0.9994059994059994,
+      "eval_f1": 0.9994179278230501,
+      "eval_loss": 0.003968308679759502,
+      "eval_precision": 0.9988365328679465,
+      "eval_recall": 1.0,
+      "eval_runtime": 73.337,
+      "eval_samples_per_second": 45.911,
+      "eval_steps_per_second": 2.877,
+      "step": 1600
+    }
+  ],
+  "logging_steps": 50,
+  "max_steps": 3788,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 4,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "EarlyStoppingCallback": {
+      "args": {
+        "early_stopping_patience": 5,
+        "early_stopping_threshold": 0.0005
+      },
+      "attributes": {
+        "early_stopping_patience_counter": 3
+      }
+    },
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 367028116847616.0,
+  "train_batch_size": 32,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1600/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0f6ee5be6db1ab816abaa77671d6a299c7d2015f383c82c395377bcfdce9d1cd
+size 5713

checkpoint-1800/config.json

@@ -0,0 +1,36 @@
+{
+  "architectures": [
+    "BertForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "cell": {},
+  "classifier_dropout": null,
+  "emb_size": 312,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 312,
+  "id2label": {
+    "0": "ham",
+    "1": "spam"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 1200,
+  "label2id": {
+    "ham": 0,
+    "spam": 1
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 4,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "pre_trained": "",
+  "structure": [],
+  "torch_dtype": "float32",
+  "transformers_version": "4.55.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

checkpoint-1800/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c8f5d4180455ea736ab182def6b38b00dc107672bfcbbbec35ebd8f8f1eb0fd4
+size 57411808

checkpoint-1800/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d8b70853056ab19a8af77db7451a979f3b67ebf653055a81e61b9af9dbbfe47
+size 114864267

checkpoint-1800/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:21ee00a18c8ad610e6aa84c09f68969220cd60698fda59a843c30c511d648944
+size 14455

checkpoint-1800/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa5df7cb9434da5febf4c571f7669ce12f83e4a51705d45719fff294ceb28256
+size 1465