| | --- |
| | license: apache-2.0 |
| | library_name: peft |
| | tags: |
| | - trl |
| | - sft |
| | - generated_from_trainer |
| | base_model: BioMistral/BioMistral-7B |
| | model-index: |
| | - name: QA_Finetune_BioMistral |
| | results: [] |
| | --- |
| | |
| | <!-- This model card has been generated automatically according to the information the Trainer had access to. You |
| | should probably proofread and complete it, then remove this comment. --> |
| |
|
| | # QA_Finetune_BioMistral |
| |
|
| | This model is a fine-tuned version of [BioMistral/BioMistral-7B](https://huggingface.co/BioMistral/BioMistral-7B) on an unknown dataset. |
| | It achieves the following results on the evaluation set: |
| | - Loss: 0.8514 |
| |
|
| | ## Model description |
| |
|
| | More information needed |
| |
|
| | ## Intended uses & limitations |
| |
|
| | More information needed |
| |
|
| | ## Training and evaluation data |
| |
|
| | More information needed |
| |
|
| | ## Training procedure |
| |
|
| | ### Training hyperparameters |
| |
|
| | The following hyperparameters were used during training: |
| | - learning_rate: 0.00025 |
| | - train_batch_size: 2 |
| | - eval_batch_size: 8 |
| | - seed: 42 |
| | - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
| | - lr_scheduler_type: cosine |
| | - lr_scheduler_warmup_ratio: 0.03 |
| | - num_epochs: 2 |
| | |
| | ### Training results |
| | |
| | | Training Loss | Epoch | Step | Validation Loss | |
| | |:-------------:|:------:|:----:|:---------------:| |
| | | 0.9391 | 0.0619 | 200 | 0.9290 | |
| | | 0.9056 | 0.1238 | 400 | 0.9171 | |
| | | 0.8717 | 0.1858 | 600 | 0.9138 | |
| | | 0.822 | 0.2477 | 800 | 0.9174 | |
| | | 0.8539 | 0.3096 | 1000 | 0.9075 | |
| | | 0.8902 | 0.3715 | 1200 | 0.9061 | |
| | | 0.936 | 0.4334 | 1400 | 0.9088 | |
| | | 0.8572 | 0.4954 | 1600 | 0.8990 | |
| | | 0.8669 | 0.5573 | 1800 | 0.8933 | |
| | | 0.875 | 0.6192 | 2000 | 0.8868 | |
| | | 0.8369 | 0.6811 | 2200 | 0.8801 | |
| | | 0.8445 | 0.7430 | 2400 | 0.8772 | |
| | | 0.8316 | 0.8050 | 2600 | 0.8692 | |
| | | 0.8573 | 0.8669 | 2800 | 0.8614 | |
| | | 0.8104 | 0.9288 | 3000 | 0.8542 | |
| | | 0.8182 | 0.9907 | 3200 | 0.8488 | |
| | | 0.5912 | 1.0526 | 3400 | 0.8659 | |
| | | 0.5579 | 1.1146 | 3600 | 0.8557 | |
| | | 0.5834 | 1.1765 | 3800 | 0.8608 | |
| | | 0.547 | 1.2384 | 4000 | 0.8514 | |
| | |
| | |
| | ### Framework versions |
| | |
| | - PEFT 0.11.1 |
| | - Transformers 4.41.2 |
| | - Pytorch 2.1.2 |
| | - Datasets 2.19.2 |
| | - Tokenizers 0.19.1 |
