Hugging Face's logo

KitsuVp
/
NeoLLM

Text Generation
Transformers
Safetensors
neollm
Generated from Trainer
conversational
custom_code
Model card Files
xet
Community
KitsuVp commited on
Commit
85da451
·
verified ·
1 Parent(s): 8c3fe8f

Model save

Browse files
Files changed (5) hide show
  1. README.md +22 -19
  2. config.json +3 -3
  3. generation_config.json +1 -1
  4. model.safetensors +1 -1
  5. training_args.bin +2 -2
README.md CHANGED
@@ -14,7 +14,7 @@ should probably proofread and complete it, then remove this comment. -->
14
 
15
  This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
16
  It achieves the following results on the evaluation set:
17
- - Loss: 3.2544
18
 
19
  ## Model description
20
 
@@ -44,28 +44,31 @@ The following hyperparameters were used during training:
44
 
45
  ### Training results
46
 
47
- | Training Loss | Epoch | Step | Validation Loss |
48
- |:-------------:|:-----:|:-----:|:---------------:|
49
- | 3.8254 | 0.064 | 5000 | 3.8259 |
50
- | 3.5877 | 0.128 | 10000 | 3.6043 |
51
- | 3.4987 | 0.192 | 15000 | 3.5176 |
52
- | 3.4495 | 0.256 | 20000 | 3.4692 |
53
- | 3.4157 | 0.32 | 25000 | 3.4390 |
54
- | 3.4003 | 0.384 | 30000 | 3.4159 |
55
- | 3.382 | 0.448 | 35000 | 3.4012 |
56
- | 3.3649 | 0.512 | 40000 | 3.3881 |
57
- | 3.3592 | 0.576 | 45000 | 3.3781 |
58
- | 3.3518 | 0.64 | 50000 | 3.3702 |
59
- | 3.3447 | 0.704 | 55000 | 3.3609 |
60
- | 3.3348 | 0.768 | 60000 | 3.3474 |
61
- | 3.2925 | 0.832 | 65000 | 3.3085 |
62
- | 3.2576 | 0.896 | 70000 | 3.2757 |
63
- | 3.2402 | 0.96 | 75000 | 3.2544 |
 
 
 
64
 
65
 
66
  ### Framework versions
67
 
68
- - Transformers 4.57.0.dev0
69
  - Pytorch 2.8.0+cu129
70
  - Datasets 3.6.0
71
  - Tokenizers 0.22.1
 
14
 
15
  This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
16
  It achieves the following results on the evaluation set:
17
+ - Loss: 3.2500
18
 
19
  ## Model description
20
 
 
44
 
45
  ### Training results
46
 
47
+ | Training Loss | Epoch | Step | Validation Loss |
48
+ |:-------------:|:------:|:-----:|:---------------:|
49
+ | 3.8685 | 0.0533 | 5000 | 3.8281 |
50
+ | 3.6486 | 0.1067 | 10000 | 3.6086 |
51
+ | 3.5596 | 0.16 | 15000 | 3.5224 |
52
+ | 3.5262 | 0.2133 | 20000 | 3.4727 |
53
+ | 3.4856 | 0.2667 | 25000 | 3.4395 |
54
+ | 3.4585 | 0.32 | 30000 | 3.4181 |
55
+ | 3.4457 | 0.3733 | 35000 | 3.4013 |
56
+ | 3.4247 | 0.4267 | 40000 | 3.3865 |
57
+ | 3.4283 | 0.48 | 45000 | 3.3743 |
58
+ | 3.4162 | 0.5333 | 50000 | 3.3662 |
59
+ | 3.4076 | 0.5867 | 55000 | 3.3583 |
60
+ | 3.3995 | 0.64 | 60000 | 3.3524 |
61
+ | 3.3996 | 0.6933 | 65000 | 3.3458 |
62
+ | 3.3921 | 0.7467 | 70000 | 3.3410 |
63
+ | 3.3648 | 0.8 | 75000 | 3.3157 |
64
+ | 3.3406 | 0.8533 | 80000 | 3.2877 |
65
+ | 3.3174 | 0.9067 | 85000 | 3.2641 |
66
+ | 3.2925 | 0.96 | 90000 | 3.2500 |
67
 
68
 
69
  ### Framework versions
70
 
71
+ - Transformers 4.57.0
72
  - Pytorch 2.8.0+cu129
73
  - Datasets 3.6.0
74
  - Tokenizers 0.22.1
config.json CHANGED
@@ -3,13 +3,13 @@
3
  "NeoLLMForCausalLM"
4
  ],
5
  "attention_bias": false,
6
- "attention_dropout": 0.0,
7
  "auto_map": {
8
  "AutoConfig": "configuration_neollm.NeoLLMConfig",
9
  "AutoModel": "modeling_neollm.NeoLLMModel",
10
  "AutoModelForCausalLM": "modeling_neollm.NeoLLMForCausalLM"
11
  },
12
- "dropout_rate": 0.0,
13
  "dtype": "bfloat16",
14
  "eos_token_id": 151645,
15
  "fan_ratio": 0.125,
@@ -47,6 +47,6 @@
47
  "rms_norm_eps": 1e-06,
48
  "rope_scaling": null,
49
  "rope_theta": 10000.0,
50
- "transformers_version": "4.57.0.dev0",
51
  "vocab_size": 151665
52
  }
 
3
  "NeoLLMForCausalLM"
4
  ],
5
  "attention_bias": false,
6
+ "attention_dropout": 0.1,
7
  "auto_map": {
8
  "AutoConfig": "configuration_neollm.NeoLLMConfig",
9
  "AutoModel": "modeling_neollm.NeoLLMModel",
10
  "AutoModelForCausalLM": "modeling_neollm.NeoLLMForCausalLM"
11
  },
12
+ "dropout_rate": 0.1,
13
  "dtype": "bfloat16",
14
  "eos_token_id": 151645,
15
  "fan_ratio": 0.125,
 
47
  "rms_norm_eps": 1e-06,
48
  "rope_scaling": null,
49
  "rope_theta": 10000.0,
50
+ "transformers_version": "4.57.0",
51
  "vocab_size": 151665
52
  }
generation_config.json CHANGED
@@ -4,5 +4,5 @@
4
  151645
5
  ],
6
  "pad_token_id": 151643,
7
- "transformers_version": "4.57.0.dev0"
8
  }
 
4
  151645
5
  ],
6
  "pad_token_id": 151643,
7
+ "transformers_version": "4.57.0"
8
  }
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:f2b7df33ffc0fd82f9e293b00783d8193039e5d145609168535212fa6b4523ea
3
  size 245234560
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:83957e75447ef4c2b1dcd2dac38119d174aeab7d9075cdab8e1107f2aceebfe4
3
  size 245234560
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:376e1ae04e84cca9a92e134ee9edbbd668e7aa2e9d98dfb5c05a99260bd967e6
3
- size 5585
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:25c46a2850e36fe6a43353ef94521e9dbda2ddad2b2211a691e88e5726c689b7
3
+ size 6033