---
library_name: transformers
base_model: minpeter/tiny-ko-187m-base-250725
tags:
- axolotl
- generated_from_trainer
datasets:
- HuggingFaceTB/smol-smoltalk
- trillionlabs/multisystem-curated
- allenai/tulu-3-sft-personas-instruction-following
- lemon-mint/smol-koreantalk
- lemon-mint/Korean-FineTome-100k
- heegyu/open-korean-instructions-v20231020
- coastral/korean-writing-style-instruct
- devngho/korean-instruction-mix
model-index:
- name: tiny-ko-187m-sft-250725
results: []
---
[
](https://github.com/axolotl-ai-cloud/axolotl)
See axolotl config
axolotl version: `0.12.0.dev0`
```yaml
base_model: minpeter/tiny-ko-187m-base-250725
hub_model_id: minpeter/tiny-ko-187m-sft-250725
output_dir: ./outputs/tiny-ko-187m-sft-250725
wandb_project: "axolotl"
wandb_entity: "kasfiekfs-e"
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer
strict: false
chat_template: chatml
datasets:
- path: HuggingFaceTB/smol-smoltalk
type: chat_template
split: train
field_messages: messages
message_property_mappings:
role: role
content: content
- path: trillionlabs/multisystem-curated
type: chat_template
split: train
field_messages: messages
message_property_mappings:
role: role
content: content
- path: allenai/tulu-3-sft-personas-instruction-following
type: chat_template
split: train
field_messages: messages
message_property_mappings:
role: role
content: content
- path: lemon-mint/smol-koreantalk
type: chat_template
split: train
field_messages: messages
message_property_mappings:
role: role
content: content
- path: lemon-mint/Korean-FineTome-100k
type: chat_template
split: train
field_messages: messages
message_property_mappings:
role: role
content: content
- path: heegyu/open-korean-instructions-v20231020
type: chat_template
split: train
field_messages: conversations
message_property_mappings:
role: from
content: value
roles:
user: ["human", "user"]
assistant: ["gpt", "assistant", "bot"]
system: ["system", "input"]
- path: coastral/korean-writing-style-instruct
type: chat_template
split: train
field_messages: conversations
message_property_mappings:
role: from
content: value
- path: devngho/korean-instruction-mix
type: chat_template
split: train
field_messages: messages
message_property_mappings:
role: from
content: value
dataset_prepared_path: last_run_prepared
val_set_size: 0.001
save_safetensors: true
sequence_len: 8192
sample_packing: false
pad_to_sequence_len: false
use_pose: true
pose_max_context_len: 65536
overrides_of_model_config:
rope_theta: 1000000.0
max_position_embeddings: 65536
gradient_accumulation_steps: 8
micro_batch_size: 16
num_epochs: 1
optimizer: muon
lr_scheduler: cosine
learning_rate: 3e-4
train_on_inputs: false
group_by_length: false
bf16: true
fp16:
tf32: true
gradient_checkpointing: false
gradient_checkpointing_kwargs:
use_reentrant: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
sdp_attention:
s2_attention:
save_steps: 200
warmup_steps: 20
eval_steps: 200
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
eos_token: '<|im_end|>'
plugins:
- axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
- axolotl.integrations.liger.LigerPlugin
# - axolotl.integrations.lm_eval.LMEvalPlugin
# lm_eval_tasks:
# - gsm8k
# - hellaswag
# - arc_easy
# - arc_challenge
# - piqa
# - winogrande
# - openbookqa
# - wsc
# - boolq
liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_layer_norm: true
liger_fused_linear_cross_entropy: true
```
# tiny-ko-187m-sft-250725
This model is a fine-tuned version of [minpeter/tiny-ko-187m-base-250725](https://huggingface.co/minpeter/tiny-ko-187m-base-250725) on the HuggingFaceTB/smol-smoltalk, the trillionlabs/multisystem-curated, the allenai/tulu-3-sft-personas-instruction-following, the lemon-mint/smol-koreantalk, the lemon-mint/Korean-FineTome-100k, the heegyu/open-korean-instructions-v20231020, the coastral/korean-writing-style-instruct and the devngho/korean-instruction-mix datasets.
It achieves the following results on the evaluation set:
- Loss: 1.6810
## Model description
More information needed
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0003
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 128
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 20
- training_steps: 13880
### Training results
| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:------:|:-----:|:---------------:|
| No log | 0 | 0 | 2.1583 |
| 2.0492 | 0.0144 | 200 | 1.9095 |
| 1.9084 | 0.0288 | 400 | 1.8642 |
| 1.8437 | 0.0432 | 600 | 1.8363 |
| 1.8601 | 0.0576 | 800 | 1.8160 |
| 1.8143 | 0.0720 | 1000 | 1.8001 |
| 1.7259 | 0.0865 | 1200 | 1.7871 |
| 1.8008 | 0.1009 | 1400 | 1.7764 |
| 1.7179 | 0.1153 | 1600 | 1.7668 |
| 1.8134 | 0.1297 | 1800 | 1.7587 |
| 1.7683 | 0.1441 | 2000 | 1.7518 |
| 1.8099 | 0.1585 | 2200 | 1.7458 |
| 1.7953 | 0.1729 | 2400 | 1.7397 |
| 1.7673 | 0.1873 | 2600 | 1.7339 |
| 1.721 | 0.2017 | 2800 | 1.7297 |
| 1.8727 | 0.2161 | 3000 | 1.7252 |
| 1.7334 | 0.2306 | 3200 | 1.7209 |
| 1.7078 | 0.2450 | 3400 | 1.7172 |
| 1.7751 | 0.2594 | 3600 | 1.7139 |
| 1.8221 | 0.2738 | 3800 | 1.7106 |
| 1.7385 | 0.2882 | 4000 | 1.7077 |
| 1.8214 | 0.3026 | 4200 | 1.7049 |
| 1.7042 | 0.3170 | 4400 | 1.7026 |
| 1.735 | 0.3314 | 4600 | 1.7005 |
| 1.647 | 0.3458 | 4800 | 1.6985 |
| 1.7222 | 0.3602 | 5000 | 1.6972 |
| 1.6963 | 0.3746 | 5200 | 1.6955 |
| 1.8047 | 0.3891 | 5400 | 1.6937 |
| 1.6151 | 0.4035 | 5600 | 1.6923 |
| 1.8008 | 0.4179 | 5800 | 1.6918 |
| 1.7152 | 0.4323 | 6000 | 1.6904 |
| 1.7522 | 0.4467 | 6200 | 1.6894 |
| 1.7645 | 0.4611 | 6400 | 1.6887 |
| 1.6721 | 0.4755 | 6600 | 1.6876 |
| 1.7343 | 0.4899 | 6800 | 1.6867 |
| 1.6748 | 0.5043 | 7000 | 1.6861 |
| 1.7417 | 0.5187 | 7200 | 1.6853 |
| 1.6245 | 0.5332 | 7400 | 1.6849 |
| 1.6081 | 0.5476 | 7600 | 1.6844 |
| 1.6696 | 0.5620 | 7800 | 1.6840 |
| 1.714 | 0.5764 | 8000 | 1.6836 |
| 1.8028 | 0.5908 | 8200 | 1.6832 |
| 1.6714 | 0.6052 | 8400 | 1.6828 |
| 1.7276 | 0.6196 | 8600 | 1.6827 |
| 1.6248 | 0.6340 | 8800 | 1.6823 |
| 1.7279 | 0.6484 | 9000 | 1.6823 |
| 1.7176 | 0.6628 | 9200 | 1.6820 |
| 1.8175 | 0.6773 | 9400 | 1.6819 |
| 1.7082 | 0.6917 | 9600 | 1.6817 |
| 1.7886 | 0.7061 | 9800 | 1.6815 |
| 1.7777 | 0.7205 | 10000 | 1.6814 |
| 1.7943 | 0.7349 | 10200 | 1.6814 |
| 1.79 | 0.7493 | 10400 | 1.6813 |
| 1.6757 | 0.7637 | 10600 | 1.6813 |
| 1.6708 | 0.7781 | 10800 | 1.6813 |
| 1.7519 | 0.7925 | 11000 | 1.6811 |
| 1.7789 | 0.8069 | 11200 | 1.6812 |
| 1.7562 | 0.8213 | 11400 | 1.6811 |
| 1.7137 | 0.8358 | 11600 | 1.6811 |
| 1.7407 | 0.8502 | 11800 | 1.6812 |
| 1.6515 | 0.8646 | 12000 | 1.6811 |
| 1.6929 | 0.8790 | 12200 | 1.6812 |
| 1.7125 | 0.8934 | 12400 | 1.6812 |
| 1.6112 | 0.9078 | 12600 | 1.6812 |
| 1.7437 | 0.9222 | 12800 | 1.6811 |
| 1.7824 | 0.9366 | 13000 | 1.6811 |
| 1.6166 | 0.9510 | 13200 | 1.6811 |
| 1.743 | 0.9654 | 13400 | 1.6812 |
| 1.6377 | 0.9799 | 13600 | 1.6812 |
| 1.7345 | 0.9943 | 13800 | 1.6810 |
### Framework versions
- Transformers 4.53.2
- Pytorch 2.7.1+cu126
- Datasets 4.0.0
- Tokenizers 0.21.2