Hugging Face's logo

gvij
/
llama-3.2-3b-swe-bench-ft

Text Generation
PEFT
Safetensors
English
code
swe-bench
software-engineering
fine-tuned
lora
Model card Files
xet
Community

Llama 3.2 3B Fine-tuned on SWE-Bench Coding Tasks

This model is a fine-tuned version of meta-llama/Llama-3.2-3B specialized for software engineering problem-solving using the SWE-Bench coding tasks dataset.

Model Description

This is a PEFT/LoRA adapter fine-tuned to improve performance on coding tasks including:

  • Code generation from natural language descriptions
  • Bug fixing and code completion
  • Software engineering problem-solving
  • Implementation of coding requirements

Training Configuration

Base Model: meta-llama/Llama-3.2-3B (3B parameters)

Dataset: simongraves/swe-bench-coding-tasks from Kaggle

Training Method: Supervised Fine-tuning with LoRA/PEFT

LoRA Configuration

{ "alora_invocation_tokens": null, "alpha_pattern": {}, "arrow_config": null, "auto_mapping": null, "base_model_name_or_path": "meta-llama/Llama-3.2-3B", "bias": "none", "corda_config": null, "ensure_weight_tying": false, "eva_config": null, "exclude_modules": null, "fan_in_fan_out": false, "inference_mode": true, "init_lora_weights": true, "layer_replication": null, "layers_pattern": null, "layers_to_transform": null, "loftq_config": {}, "lora_alpha": 64, "lora_bias": false, "lora_dropout": 0.05, "megatron_config": null, "megatron_core": "megatron.core", "modules_to_save": null, "peft_type": "LORA", "peft_version": "0.18.0", "qalora_group_size": 16, "r": 32, "rank_pattern": {}, "revision": null, "target_modules": [ "k_proj", "q_proj", "up_proj", "v_proj", "gate_proj", "down_proj", "o_proj" ], "target_parameters": null, "task_type": "CAUSAL_LM", "trainable_token_indices": null, "use_dora": false, "use_qalora": false, "use_rslora": false }

Performance Metrics

Training Metrics

{ "final_train_loss": 0.0815, "final_eval_loss": 3.3696420192718506, "total_epochs": 15, "train_samples": 5000, "val_samples": 225, "log_history": [ { "loss": 1.9948, "grad_norm": 0.7486909627914429, "learning_rate": 6.3829787234042555e-06, "entropy": 1.9819545656442643, "num_tokens": 71686.0, "mean_token_accuracy": 0.6049848683178425, "epoch": 0.016, "step": 10 }, { "loss": 1.8763, "grad_norm": 0.5951855182647705, "learning_rate": 1.347517730496454e-05, "entropy": 1.8898604318499566, "num_tokens": 143193.0, "mean_token_accuracy": 0.6278719268739223, "epoch": 0.032, "step": 20 }, { "loss": 1.7767, "grad_norm": 0.6825416684150696, "learning_rate": 2.0567375886524822e-05, "entropy": 1.836610708385706, "num_tokens": 212967.0, "mean_token_accuracy": 0.6336529016494751, "epoch": 0.048, "step": 30 }, { "loss": 1.6681, "grad_norm": 0.5455057621002197, "learning_rate": 2.765957446808511e-05, "entropy": 1.7314270935952663, "num_tokens": 280360.0, "mean_token_accuracy": 0.6564944453537465, "epoch": 0.064, "step": 40 }, { "loss": 1.649, "grad_norm": 0.5020584464073181, "learning_rate": 3.4751773049645395e-05, "entropy": 1.6479419037699699, "num_tokens": 352305.0, "mean_token_accuracy": 0.6645340114831925, "epoch": 0.08, "step": 50 }, { "loss": 1.5025, "grad_norm": 0.535496711730957, "learning_rate": 4.1843971631205674e-05, "entropy": 1.536753164231777, "num_tokens": 424123.0, "mean_token_accuracy": 0.6884755827486515, "epoch": 0.096, "step": 60 }, { "loss": 1.5259, "grad_norm": 0.5495978593826294, "learning_rate": 4.893617021276596e-05, "entropy": 1.5516022074967624, "num_tokens": 495357.0, "mean_token_accuracy": 0.6787064120173454, "epoch": 0.112, "step": 70 }, { "loss": 1.5756, "grad_norm": 0.5106986165046692, "learning_rate": 5.602836879432625e-05, "entropy": 1.596459686011076, "num_tokens": 565056.0, "mean_token_accuracy": 0.6740202151238919, "epoch": 0.128, "step": 80 }, { "loss": 1.4982, "grad_norm": 0.6457958817481995, "learning_rate": 6.312056737588653e-05, "entropy": 1.5267462871968747, "num_tokens": 636648.0, "mean_token_accuracy": 0.6839821688830853, "epoch": 0.144, "step": 90 }, { "loss": 1.5455, "grad_norm": 0.537060022354126, "learning_rate": 7.021276595744681e-05, "entropy": 1.5841539904475213, "num_tokens": 706938.0, "mean_token_accuracy": 0.672185130417347, "epoch": 0.16, "step": 100 }, { "loss": 1.4628, "grad_norm": 0.49043846130371094, "learning_rate": 7.73049645390071e-05, "entropy": 1.4655668564140796, "num_tokens": 780976.0, "mean_token_accuracy": 0.6976235516369342, "epoch": 0.176, "step": 110 }, { "loss": 1.5031, "grad_norm": 0.5280927419662476, "learning_rate": 8.439716312056739e-05, "entropy": 1.5558012172579765, "num_tokens": 851844.0, "mean_token_accuracy": 0.6802852541208267, "epoch": 0.192, "step": 120 }, { "loss": 1.4677, "grad_norm": 0.5463521480560303, "learning_rate": 9.148936170212766e-05, "entropy": 1.5025246798992158, "num_tokens": 926704.0, "mean_token_accuracy": 0.6847469419240951, "epoch": 0.208, "step": 130 }, { "loss": 1.4653, "grad_norm": 0.6182817220687866, "learning_rate": 9.858156028368794e-05, "entropy": 1.5020036295056343, "num_tokens": 994535.0, "mean_token_accuracy": 0.6898098006844521, "epoch": 0.224, "step": 140 }, { "loss": 1.44, "grad_norm": 0.4842669665813446, "learning_rate": 0.00010567375886524824, "entropy": 1.4702169127762317, "num_tokens": 1067156.0, "mean_token_accuracy": 0.6916787229478359, "epoch": 0.24, "step": 150 }, { "loss": 1.3588, "grad_norm": 0.4752415418624878, "learning_rate": 0.00011276595744680852, "entropy": 1.4036796674132348, "num_tokens": 1137368.0, "mean_token_accuracy": 0.7042367912828922, "epoch": 0.256, "step": 160 }, { "loss": 1.4378, "grad_norm": 0.5278449654579163, "learning_rate": 0.0001198581560283688, "entropy": 1.4550598032772541, "num_tokens": 1209961.0, "mean_token_accuracy": 0.6916994586586952, "epoch": 0.272, "step": 170 }, { "loss": 1.4366, "grad_norm": 0.48351314663887024, "learning_rate": 0.0001269503546099291, "entropy": 1.461528491973877, "num_tokens": 1282137.0, "mean_token_accuracy": 0.6974675498902798, "epoch": 0.288, "step": 180 }, { "loss": 1.4501, "grad_norm": 0.47627273201942444, "learning_rate": 0.00013404255319148938, "entropy": 1.4737755820155143, "num_tokens": 1352588.0, "mean_token_accuracy": 0.6923532616347075, "epoch": 0.304, "step": 190 }, { "loss": 1.3316, "grad_norm": 0.4203171730041504, "learning_rate": 0.00014113475177304964, "entropy": 1.3838087745010852, "num_tokens": 1424610.0, "mean_token_accuracy": 0.7135859195142984, "epoch": 0.32, "step": 200 }, { "loss": 1.3683, "grad_norm": 0.42347168922424316, "learning_rate": 0.00014822695035460995, "entropy": 1.3961401782929896, "num_tokens": 1494188.0, "mean_token_accuracy": 0.701457566767931, "epoch": 0.336, "step": 210 }, { "loss": 1.3848, "grad_norm": 0.395522803068161, "learning_rate": 0.0001553191489361702, "entropy": 1.4141620382666589, "num_tokens": 1565347.0, "mean_token_accuracy": 0.7018114134669304, "epoch": 0.352, "step": 220 }, { "loss": 1.4559, "grad_norm": 0.43445274233818054, "learning_rate": 0.0001624113475177305, "entropy": 1.4895919755101203, "num_tokens": 1638477.0, "mean_token_accuracy": 0.6866721548140049, "epoch": 0.368, "step": 230 }, { "loss": 1.4136, "grad_norm": 0.411599338054657, "learning_rate": 0.00016950354609929078, "entropy": 1.4408979743719101, "num_tokens": 1709050.0, "mean_token_accuracy": 0.6949560333043336, "epoch": 0.384, "step": 240 }, { "loss": 1.3856, "grad_norm": 0.47367680072784424, "learning_rate": 0.00017659574468085107, "entropy": 1.3981009744107724, "num_tokens": 1781646.0, "mean_token_accuracy": 0.7042551681399345, "epoch": 0.4, "step": 250 }, { "loss": 1.3996, "grad_norm": 0.4378238916397095, "learning_rate": 0.00018368794326241135, "entropy": 1.4358101204037665, "num_tokens": 1851100.0, "mean_token_accuracy": 0.7007784590125083, "epoch": 0.416, "step": 260 }, { "loss": 1.2763, "grad_norm": 0.43129193782806396, "learning_rate": 0.00019078014184397164, "entropy": 1.3177187278866769, "num_tokens": 1922805.0, "mean_token_accuracy": 0.7222738478332758, "epoch": 0.432, "step": 270 }, { "loss": 1.4247, "grad_norm": 0.386320561170578, "learning_rate": 0.00019787234042553193, "entropy": 1.4367552738636733, "num_tokens": 1994015.0, "mean_token_accuracy": 0.6939685396850109, "epoch": 0.448, "step": 280 }, { "loss": 1.4532, "grad_norm": 0.5261701345443726, "learning_rate": 0.00019999970755026251, "entropy": 1.4758896235376597, "num_tokens": 2063563.0, "mean_token_accuracy": 0.6888365402817727, "epoch": 0.464, "step": 290 }, { "loss": 1.2478, "grad_norm": 0.5084525942802429, "learning_rate": 0.00019999827514750274, "entropy": 1.275223758071661, "num_tokens": 2136349.0, "mean_token_accuracy": 0.7304733999073505, "epoch": 0.48, "step": 300 }, { "loss": 1.3124, "grad_norm": 0.37344890832901, "learning_rate": 0.00019999564909353962, "entropy": 1.3534214630723, "num_tokens": 2206118.0, "mean_token_accuracy": 0.7152372144162655, "epoch": 0.496, "step": 310 }, { "loss": 1.4326, "grad_norm": 0.4442685544490814, "learning_rate": 0.00019999182941971957, "entropy": 1.4260568536818028, "num_tokens": 2275404.0, "mean_token_accuracy": 0.694905637204647, "epoch": 0.512, "step": 320 }, { "loss": 1.2551, "grad_norm": 0.3869602680206299, "learning_rate": 0.000199986816171637, "entropy": 1.2809278029948472, "num_tokens": 2347095.0, "mean_token_accuracy": 0.7306458696722984, "epoch": 0.528, "step": 330 }, { "loss": 1.2628, "grad_norm": 0.40451544523239136, "learning_rate": 0.00019998060940913366, "entropy": 1.3015176679939031, "num_tokens": 2419295.0, "mean_token_accuracy": 0.7233606927096844, "epoch": 0.544, "step": 340 }, { "loss": 1.4416, "grad_norm": 0.41146278381347656, "learning_rate": 0.0001999732092062979, "entropy": 1.453403216600418, "num_tokens": 2488778.0, "mean_token_accuracy": 0.6912899497896433, "epoch": 0.56, "step": 350 }, { "loss": 1.3686, "grad_norm": 0.46797484159469604, "learning_rate": 0.00019996461565146382, "entropy": 1.3723408557474612, "num_tokens": 2559999.0, "mean_token_accuracy": 0.7082154989242554, "epoch": 0.576, "step": 360 }, { "loss": 1.2978, "grad_norm": 0.5259590148925781, "learning_rate": 0.0001999548288472103, "entropy": 1.348337971046567, "num_tokens": 2631141.0, "mean_token_accuracy": 0.7174708023667336, "epoch": 0.592, "step": 370 }, { "loss": 1.3396, "grad_norm": 0.3720746636390686, "learning_rate": 0.00019994384891035968, "entropy": 1.3625607885420323, "num_tokens": 2703632.0, "mean_token_accuracy": 0.7107365172356367, "epoch": 0.608, "step": 380 }, { "loss": 1.3262, "grad_norm": 0.3667563796043396, "learning_rate": 0.00019993167597197632, "entropy": 1.3428934179246426, "num_tokens": 2776366.0, "mean_token_accuracy": 0.7123019598424435, "epoch": 0.624, "step": 390 }, { "loss": 1.2946, "grad_norm": 0.4993073642253876, "learning_rate": 0.00019991831017736518, "entropy": 1.3254120852798223, "num_tokens": 2847442.0, "mean_token_accuracy": 0.718818012624979, "epoch": 0.64, "step": 400 }, { "loss": 1.3243, "grad_norm": 0.39404183626174927, "learning_rate": 0.00019990375168606997, "entropy": 1.360147947818041, "num_tokens": 2915842.0, "mean_token_accuracy": 0.7167202346026897, "epoch": 0.656, "step": 410 }, { "loss": 1.3479, "grad_norm": 0.4524689018726349, "learning_rate": 0.0001998880006718713, "entropy": 1.3626457568258048, "num_tokens": 2989026.0, "mean_token_accuracy": 0.7087345108389854, "epoch": 0.672, "step": 420 }, { "loss": 1.2623, "grad_norm": 0.47156020998954773, "learning_rate": 0.00019987105732278458, "entropy": 1.294278558343649, "num_tokens": 3054009.0, "mean_token_accuracy": 0.7230308122932911, "epoch": 0.688, "step": 430 }, { "loss": 1.3101, "grad_norm": 0.38615041971206665, "learning_rate": 0.00019985292184105776, "entropy": 1.3357798032462598, "num_tokens": 3126343.0, "mean_token_accuracy": 0.7129710622131824, "epoch": 0.704, "step": 440 }, { "loss": 1.2825, "grad_norm": 0.36571189761161804, "learning_rate": 0.00019983359444316901, "entropy": 1.290989424288273, "num_tokens": 3199352.0, "mean_token_accuracy": 0.7210433587431908, "epoch": 0.72, "step": 450 }, { "loss": 1.2414, "grad_norm": 0.45147979259490967, "learning_rate": 0.00019981307535982406, "entropy": 1.2785895802080631, "num_tokens": 3269842.0, "mean_token_accuracy": 0.7266046453267336, "epoch": 0.736, "step": 460 }, { "loss": 1.3332, "grad_norm": 0.4632173478603363, "learning_rate": 0.00019979136483595333, "entropy": 1.371579215861857, "num_tokens": 3339455.0, "mean_token_accuracy": 0.7108632929623127, "epoch": 0.752, "step": 470 }, { "loss": 1.3275, "grad_norm": 0.39540112018585205, "learning_rate": 0.00019976846313070928, "entropy": 1.3393389645963907, "num_tokens": 3411252.0, "mean_token_accuracy": 0.7145411107689142, "epoch": 0.768, "step": 480 }, { "loss": 1.2925, "grad_norm": 0.3513447940349579, "learning_rate": 0.0001997443705174631, "entropy": 1.3189247287809849, "num_tokens": 3485622.0, "mean_token_accuracy": 0.7181398883461952, "epoch": 0.784, "step": 490 }, { "loss": 1.3037, "grad_norm": 0.4739632308483124, "learning_rate": 0.0001997190872838015, "entropy": 1.3312291190028192, "num_tokens": 3556782.0, "mean_token_accuracy": 0.7156806670129299, "epoch": 0.8, "step": 500 }, { "loss": 1.2766, "grad_norm": 0.3911494016647339, "learning_rate": 0.00019969261373152332, "entropy": 1.30355613976717, "num_tokens": 3630312.0, "mean_token_accuracy": 0.7226576715707779, "epoch": 0.816, "step": 510 }, { "loss": 1.2058, "grad_norm": 0.40945300459861755, "learning_rate": 0.0001996649501766359, "entropy": 1.2415074806660413, "num_tokens": 3700730.0, "mean_token_accuracy": 0.7356499239802361, "epoch": 0.832, "step": 520 }, { "loss": 1.2376, "grad_norm": 0.37360429763793945, "learning_rate": 0.00019963609694935127, "entropy": 1.2670705320313573, "num_tokens": 3774998.0, "mean_token_accuracy": 0.7300258550792933, "epoch": 0.848, "step": 530 }, { "loss": 1.2181, "grad_norm": 0.40702569484710693, "learning_rate": 0.00019960605439408224, "entropy": 1.2638164822012186, "num_tokens": 3842921.0, "mean_token_accuracy": 0.7284378685057163, "epoch": 0.864, "step": 540 }, { "loss": 1.244, "grad_norm": 0.44862040877342224, "learning_rate": 0.00019957482286943838, "entropy": 1.274675579369068, "num_tokens": 3913933.0, "mean_token_accuracy": 0.7269890259951353, "epoch": 0.88, "step": 550 }, { "loss": 1.2889, "grad_norm": 0.44631901383399963, "learning_rate": 0.00019954240274822153, "entropy": 1.3203198406845331, "num_tokens": 3988398.0, "mean_token_accuracy": 0.7194302972406149, "epoch": 0.896, "step": 560 }, { "loss": 1.307, "grad_norm": 0.4080927073955536, "learning_rate": 0.0001995087944174216, "entropy": 1.3237323697656393, "num_tokens": 4058355.0, "mean_token_accuracy": 0.7134534392505885, "epoch": 0.912, "step": 570 }, { "loss": 1.2429, "grad_norm": 0.47242629528045654, "learning_rate": 0.00019947399827821167, "entropy": 1.2848535921424626, "num_tokens": 4128856.0, "mean_token_accuracy": 0.7282398778945207, "epoch": 0.928, "step": 580 }, { "loss": 1.3198, "grad_norm": 0.46692216396331787, "learning_rate": 0.0001994380147459435, "entropy": 1.3414464402943849, "num_tokens": 4197548.0, "mean_token_accuracy": 0.7105044238269329, "epoch": 0.944, "step": 590 }, { "loss": 1.2773, "grad_norm": 0.4496074616909027, "learning_rate": 0.00019940084425014237, "entropy": 1.3169708456844091, "num_tokens": 4267820.0, "mean_token_accuracy": 0.7220833536237479, "epoch": 0.96, "step": 600 }, { "loss": 1.3616, "grad_norm": 0.39233991503715515, "learning_rate": 0.00019936248723450195, "entropy": 1.3580100305378437, "num_tokens": 4340952.0, "mean_token_accuracy": 0.7052121929824352, "epoch": 0.976, "step": 610 }, { "loss": 1.3485, "grad_norm": 0.4215790033340454, "learning_rate": 0.00019932294415687918, "entropy": 1.3914434991776943, "num_tokens": 4412912.0, "mean_token_accuracy": 0.7035498872399331, "epoch": 0.992, "step": 620 }, { "eval_loss": 1.6514616012573242, "eval_runtime": 53.8639, "eval_samples_per_second": 4.177, "eval_steps_per_second": 4.177, "eval_entropy": 1.607075858645969, "eval_num_tokens": 4446722.0, "eval_mean_token_accuracy": 0.6563644862174988, "epoch": 1.0, "step": 625 }, { "loss": 1.2199, "grad_norm": 0.37617871165275574, "learning_rate": 0.00019928221548928856, "entropy": 1.254193838685751, "num_tokens": 4484529.0, "mean_token_accuracy": 0.733865063637495, "epoch": 1.008, "step": 630 }, { "loss": 1.255, "grad_norm": 0.4245050847530365, "learning_rate": 0.00019924030171789676, "entropy": 1.2880515411496163, "num_tokens": 4558163.0, "mean_token_accuracy": 0.7216448508203029, "epoch": 1.024, "step": 640 }, { "loss": 1.2453, "grad_norm": 0.528607189655304, "learning_rate": 0.00019919720334301663, "entropy": 1.286012227460742, "num_tokens": 4624818.0, "mean_token_accuracy": 0.7230253227055072, "epoch": 1.04, "step": 650 }, { "loss": 1.1865, "grad_norm": 0.40067175030708313, "learning_rate": 0.0001991529208791013, "entropy": 1.2207407971844078, "num_tokens": 4692104.0, "mean_token_accuracy": 0.738229251652956, "epoch": 1.056, "step": 660 }, { "loss": 1.1083, "grad_norm": 0.43201762437820435, "learning_rate": 0.00019910745485473804, "entropy": 1.1699247900396585, "num_tokens": 4763989.0, "mean_token_accuracy": 0.7470208257436752, "epoch": 1.072, "step": 670 }, { "loss": 1.1023, "grad_norm": 0.4284451901912689, "learning_rate": 0.000199060805812642, "entropy": 1.1472565781325101, "num_tokens": 4835107.0, "mean_token_accuracy": 0.7508105374872684, "epoch": 1.088, "step": 680 }, { "loss": 1.2272, "grad_norm": 0.44135841727256775, "learning_rate": 0.0001990129743096496, "entropy": 1.2847526386380195, "num_tokens": 4902029.0, "mean_token_accuracy": 0.7237243212759494, "epoch": 1.104, "step": 690 }, { "loss": 1.1017, "grad_norm": 0.47729751467704773, "learning_rate": 0.000198963960916712, "entropy": 1.1450510159134866, "num_tokens": 4971516.0, "mean_token_accuracy": 0.7504730649292469, "epoch": 1.12, "step": 700 }, { "loss": 1.157, "grad_norm": 0.447290301322937, "learning_rate": 0.00019891376621888828, "entropy": 1.2040530007332564, "num_tokens": 5043158.0, "mean_token_accuracy": 0.7401271492242814, "epoch": 1.1360000000000001, "step": 710 }, { "loss": 1.1607, "grad_norm": 0.46857067942619324, "learning_rate": 0.00019886239081533838, "entropy": 1.2040101885795593, "num_tokens": 5112319.0, "mean_token_accuracy": 0.7364955507218838, "epoch": 1.152, "step": 720 }, { "loss": 1.2236, "grad_norm": 0.3849755823612213, "learning_rate": 0.00019880983531931596, "entropy": 1.2752032484859228, "num_tokens": 5185024.0, "mean_token_accuracy": 0.7275632154196501, "epoch": 1.168, "step": 730 }, { "loss": 1.0759, "grad_norm": 0.42884477972984314, "learning_rate": 0.0001987561003581612, "entropy": 1.1102558821439743, "num_tokens": 5262375.0, "mean_token_accuracy": 0.7617518465965987, "epoch": 1.184, "step": 740 }, { "loss": 1.0835, "grad_norm": 0.4471180737018585, "learning_rate": 0.00019870118657329316, "entropy": 1.1245277592912317, "num_tokens": 5334588.0, "mean_token_accuracy": 0.7560351178050041, "epoch": 1.2, "step": 750 }, { "loss": 1.1543, "grad_norm": 0.511607825756073, "learning_rate": 0.00019864509462020217, "entropy": 1.184562399238348, "num_tokens": 5402411.0, "mean_token_accuracy": 0.7417333047837019, "epoch": 1.216, "step": 760 }, { "loss": 1.2011, "grad_norm": 0.43140727281570435, "learning_rate": 0.0001985878251684421, "entropy": 1.2428947329521178, "num_tokens": 5475281.0, "mean_token_accuracy": 0.7328000146895647, "epoch": 1.232, "step": 770 }, { "loss": 1.1519, "grad_norm": 0.46558865904808044, "learning_rate": 0.00019852937890162218, "entropy": 1.1817235488444566, "num_tokens": 5547071.0, "mean_token_accuracy": 0.7421239599585533, "epoch": 1.248, "step": 780 }, { "loss": 1.1626, "grad_norm": 0.490234911441803, "learning_rate": 0.0001984697565173991, "entropy": 1.1980197869241238, "num_tokens": 5617800.0, "mean_token_accuracy": 0.7421411775052548, "epoch": 1.264, "step": 790 }, { "loss": 1.1429, "grad_norm": 0.4566928446292877, "learning_rate": 0.00019840895872746833, "entropy": 1.1743735186755657, "num_tokens": 5689519.0, "mean_token_accuracy": 0.7413917820900678, "epoch": 1.28, "step": 800 }, { "loss": 1.1979, "grad_norm": 0.49966302514076233, "learning_rate": 0.00019834698625755603, "entropy": 1.2271290764212608, "num_tokens": 5760881.0, "mean_token_accuracy": 0.7310593172907829, "epoch": 1.296, "step": 810 }, { "loss": 1.1207, "grad_norm": 0.45370253920555115, "learning_rate": 0.00019828383984741007, "entropy": 1.162224245816469, "num_tokens": 5835372.0, "mean_token_accuracy": 0.7436192393302917, "epoch": 1.312, "step": 820 }, { "loss": 1.1517, "grad_norm": 0.46117326617240906, "learning_rate": 0.00019821952025079133, "entropy": 1.1935209576040506, "num_tokens": 5909664.0, "mean_token_accuracy": 0.7423036534339189, "epoch": 1.328, "step": 830 }, { "loss": 1.1694, "grad_norm": 0.4315175414085388, "learning_rate": 0.00019815402823546467, "entropy": 1.2176305908709764, "num_tokens": 5977406.0, "mean_token_accuracy": 0.7369622588157654, "epoch": 1.3439999999999999, "step": 840 }, { "loss": 1.1593, "grad_norm": 0.4738141894340515, "learning_rate": 0.00019808736458318987, "entropy": 1.1922734286636114, "num_tokens": 6051795.0, "mean_token_accuracy": 0.7396270222961903, "epoch": 1.3599999999999999, "step": 850 }, { "loss": 1.2308, "grad_norm": 0.466564804315567, "learning_rate": 0.00019801953008971202, "entropy": 1.2686352148652076, "num_tokens": 6121424.0, "mean_token_accuracy": 0.7265080660581589, "epoch": 1.376, "step": 860 }, { "loss": 1.0973, "grad_norm": 0.5347933769226074, "learning_rate": 0.00019795052556475246, "entropy": 1.1349049855023623, "num_tokens": 6194124.0, "mean_token_accuracy": 0.749181279540062, "epoch": 1.392, "step": 870 }, { "loss": 1.2007, "grad_norm": 0.5176814794540405, "learning_rate": 0.00019788035183199867, "entropy": 1.2523904383182525, "num_tokens": 6265712.0, "mean_token_accuracy": 0.7308040276169777, "epoch": 1.408, "step": 880 }, { "loss": 1.1874, "grad_norm": 0.5059093236923218, "learning_rate": 0.00019780900972909472, "entropy": 1.214011199772358, "num_tokens": 6336439.0, "mean_token_accuracy": 0.7323950476944446, "epoch": 1.424, "step": 890 }, { "loss": 1.1586, "grad_norm": 0.5010010600090027, "learning_rate": 0.0001977365001076312, "entropy": 1.222124165482819, "num_tokens": 6404854.0, "mean_token_accuracy": 0.7379618931561709, "epoch": 1.44, "step": 900 }, { "loss": 1.1011, "grad_norm": 0.46565452218055725, "learning_rate": 0.00019766282383313496, "entropy": 1.1317486308515072, "num_tokens": 6477160.0, "mean_token_accuracy": 0.7503468841314316, "epoch": 1.456, "step": 910 }, { "loss": 1.1213, "grad_norm": 0.49189579486846924, "learning_rate": 0.00019758798178505895, "entropy": 1.161119259521365, "num_tokens": 6548489.0, "mean_token_accuracy": 0.7536120630800724, "epoch": 1.472, "step": 920 }, { "loss": 1.1973, "grad_norm": 0.4749371111392975, "learning_rate": 0.00019751197485677152, "entropy": 1.246105576120317, "num_tokens": 6619514.0, "mean_token_accuracy": 0.7342945698648691, "epoch": 1.488, "step": 930 }, { "loss": 1.1726, "grad_norm": 0.5040514469146729, "learning_rate": 0.000197434803955546, "entropy": 1.2014099791646005, "num_tokens": 6691743.0, "mean_token_accuracy": 0.7391631372272969, "epoch": 1.504, "step": 940 }, { "loss": 1.1813, "grad_norm": 0.4449273645877838, "learning_rate": 0.00019735647000254967, "entropy": 1.209279171936214, "num_tokens": 6764504.0, "mean_token_accuracy": 0.737961059063673, "epoch": 1.52, "step": 950 }, { "loss": 1.1318, "grad_norm": 0.4458376169204712, "learning_rate": 0.00019727697393283276, "entropy": 1.1810262396931648, "num_tokens": 6835187.0, "mean_token_accuracy": 0.7471827574074268, "epoch": 1.536, "step": 960 }, { "loss": 1.1481, "grad_norm": 0.48734229803085327, "learning_rate": 0.0001971963166953175, "entropy": 1.1968950856477023, "num_tokens": 6902385.0, "mean_token_accuracy": 0.7390185371041298, "epoch": 1.552, "step": 970 }, { "loss": 1.1126, "grad_norm": 0.49335455894470215, "learning_rate": 0.00019711449925278657, "entropy": 1.16581725589931, "num_tokens": 6974112.0, "mean_token_accuracy": 0.749110171943903, "epoch": 1.568, "step": 980 }, { "loss": 1.1273, "grad_norm": 0.5503702163696289, "learning_rate": 0.0001970315225818717, "entropy": 1.1577822044491768, "num_tokens": 7047193.0, "mean_token_accuracy": 0.7442155458033085, "epoch": 1.584, "step": 990 }, { "loss": 1.1613, "grad_norm": 0.48770779371261597, "learning_rate": 0.00019694738767304197, "entropy": 1.2273796085268258, "num_tokens": 7118445.0, "mean_token_accuracy": 0.7344189159572124, "epoch": 1.6, "step": 1000 }, { "loss": 1.1412, "grad_norm": 0.5353670120239258, "learning_rate": 0.0001968620955305921, "entropy": 1.163969399780035, "num_tokens": 7189764.0, "mean_token_accuracy": 0.7420754589140415, "epoch": 1.616, "step": 1010 }, { "loss": 1.1834, "grad_norm": 0.5608494877815247, "learning_rate": 0.00019677564717263032, "entropy": 1.2149179238826036, "num_tokens": 7264323.0, "mean_token_accuracy": 0.7355741441249848, "epoch": 1.6320000000000001, "step": 1020 }, { "loss": 1.2126, "grad_norm": 0.5233872532844543, "learning_rate": 0.00019668804363106627, "entropy": 1.2384574502706527, "num_tokens": 7332706.0, "mean_token_accuracy": 0.7354658141732215, "epoch": 1.6480000000000001, "step": 1030 }, { "loss": 1.1159, "grad_norm": 0.5007599592208862, "learning_rate": 0.0001965992859515988, "entropy": 1.1609713550657035, "num_tokens": 7401120.0, "mean_token_accuracy": 0.7472618170082569, "epoch": 1.6640000000000001, "step": 1040 }, { "loss": 1.078, "grad_norm": 0.4869406521320343, "learning_rate": 0.00019650937519370313, "entropy": 1.10781458504498, "num_tokens": 7472038.0, "mean_token_accuracy": 0.7583202414214611, "epoch": 1.6800000000000002, "step": 1050 }, { "loss": 1.1075, "grad_norm": 0.4964727759361267, "learning_rate": 0.0001964183124306188, "entropy": 1.1309296403080225, "num_tokens": 7544043.0, "mean_token_accuracy": 0.7532989703118801, "epoch": 1.696, "step": 1060 }, { "loss": 1.1254, "grad_norm": 0.4597083032131195, "learning_rate": 0.00019632609874933619, "entropy": 1.1619811242446303, "num_tokens": 7612933.0, "mean_token_accuracy": 0.7445178624242544, "epoch": 1.712, "step": 1070 }, { "loss": 1.1394, "grad_norm": 0.4475013017654419, "learning_rate": 0.00019623273525058406, "entropy": 1.190695314668119, "num_tokens": 7681866.0, "mean_token_accuracy": 0.7421732414513826, "epoch": 1.728, "step": 1080 }, { "loss": 1.148, "grad_norm": 0.5165102481842041, "learning_rate": 0.0001961382230488162, "entropy": 1.1672584012150764, "num_tokens": 7749474.0, "mean_token_accuracy": 0.7430988229811192, "epoch": 1.744, "step": 1090 }, { "loss": 1.123, "grad_norm": 0.4841471016407013, "learning_rate": 0.000196042563272198, "entropy": 1.1595928162336349, "num_tokens": 7821296.0, "mean_token_accuracy": 0.7526533514261246, "epoch": 1.76, "step": 1100 }, { "loss": 1.0997, "grad_norm": 0.4719853699207306, "learning_rate": 0.00019594575706259333, "entropy": 1.1306645534932613, "num_tokens": 7896829.0, "mean_token_accuracy": 0.7519307494163513, "epoch": 1.776, "step": 1110 }, { "loss": 1.1471, "grad_norm": 0.47517725825309753, "learning_rate": 0.00019584780557555055, "entropy": 1.175300621986389, "num_tokens": 7967061.0, "mean_token_accuracy": 0.7416853681206703, "epoch": 1.792, "step": 1120 }, { "loss": 1.1877, "grad_norm": 0.4702308773994446, "learning_rate": 0.00019574870998028893, "entropy": 1.2220322269946338, "num_tokens": 8039221.0, "mean_token_accuracy": 0.7379240494221448, "epoch": 1.808, "step": 1130 }, { "loss": 1.1149, "grad_norm": 0.46563196182250977, "learning_rate": 0.00019564847145968467, "entropy": 1.1420297373086215, "num_tokens": 8108927.0, "mean_token_accuracy": 0.7475541032850742, "epoch": 1.8239999999999998, "step": 1140 }, { "loss": 1.1489, "grad_norm": 0.4813663065433502, "learning_rate": 0.00019554709121025668, "entropy": 1.172530260309577, "num_tokens": 8183714.0, "mean_token_accuracy": 0.7462154671549797, "epoch": 1.8399999999999999, "step": 1150 }, { "loss": 1.2017, "grad_norm": 0.44143185019493103, "learning_rate": 0.0001954445704421524, "entropy": 1.2249045295640826, "num_tokens": 8254462.0, "mean_token_accuracy": 0.7346134804189205, "epoch": 1.8559999999999999, "step": 1160 }, { "loss": 1.1741, "grad_norm": 0.4847259223461151, "learning_rate": 0.0001953409103791334, "entropy": 1.1954007063060998, "num_tokens": 8323208.0, "mean_token_accuracy": 0.7357832938432693, "epoch": 1.8719999999999999, "step": 1170 }, { "loss": 1.1528, "grad_norm": 0.543472945690155, "learning_rate": 0.00019523611225856052, "entropy": 1.185812197253108, "num_tokens": 8396473.0, "mean_token_accuracy": 0.7436682526022196, "epoch": 1.888, "step": 1180 }, { "loss": 1.1445, "grad_norm": 0.4779781997203827, "learning_rate": 0.00019513017733137938, "entropy": 1.2023186199367046, "num_tokens": 8467116.0, "mean_token_accuracy": 0.7408940136432648, "epoch": 1.904, "step": 1190 }, { "loss": 1.1261, "grad_norm": 0.5252004861831665, "learning_rate": 0.00019502310686210535, "entropy": 1.1502687316387892, "num_tokens": 8537947.0, "mean_token_accuracy": 0.7499999478459358, "epoch": 1.92, "step": 1200 }, { "loss": 1.0326, "grad_norm": 0.5127160549163818, "learning_rate": 0.00019491490212880842, "entropy": 1.0876031598076223, "num_tokens": 8610487.0, "mean_token_accuracy": 0.7600250542163849, "epoch": 1.936, "step": 1210 }, { "loss": 1.0994, "grad_norm": 0.5108591914176941, "learning_rate": 0.00019480556442309796, "entropy": 1.1331037379801274, "num_tokens": 8681882.0, "mean_token_accuracy": 0.7524488516151905, "epoch": 1.952, "step": 1220 }, { "loss": 1.0444, "grad_norm": 0.4390319287776947, "learning_rate": 0.00019469509505010732, "entropy": 1.1067850556224585, "num_tokens": 8752529.0, "mean_token_accuracy": 0.7629696264863014, "epoch": 1.968, "step": 1230 }, { "loss": 1.1131, "grad_norm": 0.4385789632797241, "learning_rate": 0.00019458349532847823, "entropy": 1.1420386414974928, "num_tokens": 8822606.0, "mean_token_accuracy": 0.7509460024535656, "epoch": 1.984, "step": 1240 }, { "loss": 1.156, "grad_norm": 0.48881295323371887, "learning_rate": 0.00019447076659034513, "entropy": 1.1831431476399303, "num_tokens": 8893444.0, "mean_token_accuracy": 0.7443063445389271, "epoch": 2.0, "step": 1250 }, { "eval_loss": 1.694083333015442, "eval_runtime": 53.7259, "eval_samples_per_second": 4.188, "eval_steps_per_second": 4.188, "eval_entropy": 1.5593494982189602, "eval_num_tokens": 8893444.0, "eval_mean_token_accuracy": 0.6529413857724932, "epoch": 2.0, "step": 1250 }, { "loss": 0.9244, "grad_norm": 0.5743361115455627, "learning_rate": 0.00019435691018131914, "entropy": 0.9969420040026307, "num_tokens": 8964945.0, "mean_token_accuracy": 0.7868913397192955, "epoch": 2.016, "step": 1260 }, { "loss": 0.9269, "grad_norm": 0.500278115272522, "learning_rate": 0.00019424192746047208, "entropy": 0.9957487585023046, "num_tokens": 9032760.0, "mean_token_accuracy": 0.7852505221962929, "epoch": 2.032, "step": 1270 }, { "loss": 0.9706, "grad_norm": 0.5637155771255493, "learning_rate": 0.00019412581980032028, "entropy": 1.0206938754767179, "num_tokens": 9102740.0, "mean_token_accuracy": 0.7756361834704876, "epoch": 2.048, "step": 1280 }, { "loss": 0.8276, "grad_norm": 0.5537267923355103, "learning_rate": 0.00019400858858680813, "entropy": 0.9020865254104138, "num_tokens": 9170729.0, "mean_token_accuracy": 0.8033500172197818, "epoch": 2.064, "step": 1290 }, { "loss": 0.9247, "grad_norm": 0.5204894542694092, "learning_rate": 0.00019389023521929156, "entropy": 0.9940342519432306, "num_tokens": 9240162.0, "mean_token_accuracy": 0.790066198259592, "epoch": 2.08, "step": 1300 }, { "loss": 1.0241, "grad_norm": 0.5501781702041626, "learning_rate": 0.00019377076111052127, "entropy": 1.0685310505330563, "num_tokens": 9312645.0, "mean_token_accuracy": 0.764731926098466, "epoch": 2.096, "step": 1310 }, { "loss": 0.9996, "grad_norm": 0.5966712236404419, "learning_rate": 0.0001936501676866261, "entropy": 1.078461030125618, "num_tokens": 9382103.0, "mean_token_accuracy": 0.768770469725132, "epoch": 2.112, "step": 1320 }, { "loss": 1.0079, "grad_norm": 0.5208315253257751, "learning_rate": 0.0001935284563870957, "entropy": 1.0595249433070422, "num_tokens": 9453929.0, "mean_token_accuracy": 0.7636033929884434, "epoch": 2.128, "step": 1330 }, { "loss": 0.9196, "grad_norm": 0.4879686236381531, "learning_rate": 0.00019340562866476346, "entropy": 0.9946582894772291, "num_tokens": 9524472.0, "mean_token_accuracy": 0.7831553496420384, "epoch": 2.144, "step": 1340 }, { "loss": 0.8975, "grad_norm": 0.5127525925636292, "learning_rate": 0.00019328168598578934, "entropy": 0.9507135501131415, "num_tokens": 9596158.0, "mean_token_accuracy": 0.791859669983387, "epoch": 2.16, "step": 1350 }, { "loss": 0.9609, "grad_norm": 0.6453097462654114, "learning_rate": 0.00019315662982964207, "entropy": 1.0353197909891605, "num_tokens": 9665118.0, "mean_token_accuracy": 0.7720922604203224, "epoch": 2.176, "step": 1360 }, { "loss": 0.9383, "grad_norm": 0.5106361508369446, "learning_rate": 0.00019303046168908175, "entropy": 0.9984045408666133, "num_tokens": 9737358.0, "mean_token_accuracy": 0.7770775146782398, "epoch": 2.192, "step": 1370 }, { "loss": 0.9066, "grad_norm": 0.5386838912963867, "learning_rate": 0.00019290318307014188, "entropy": 0.9718356661498546, "num_tokens": 9808857.0, "mean_token_accuracy": 0.7863212361931801, "epoch": 2.208, "step": 1380 }, { "loss": 0.9439, "grad_norm": 0.5410692095756531, "learning_rate": 0.00019277479549211144, "entropy": 1.0157162923365832, "num_tokens": 9878604.0, "mean_token_accuracy": 0.7807501688599586, "epoch": 2.224, "step": 1390 }, { "loss": 0.8767, "grad_norm": 0.5981262922286987, "learning_rate": 0.00019264530048751667, "entropy": 0.9412238096818328, "num_tokens": 9951998.0, "mean_token_accuracy": 0.793684670329094, "epoch": 2.24, "step": 1400 }, { "loss": 0.9549, "grad_norm": 0.4889194667339325, "learning_rate": 0.000192514699602103, "entropy": 1.020562462694943, "num_tokens": 10022810.0, "mean_token_accuracy": 0.7754987187683582, "epoch": 2.2560000000000002, "step": 1410 }, { "loss": 0.9066, "grad_norm": 0.6018396615982056, "learning_rate": 0.00019238299439481633, "entropy": 0.952093257009983, "num_tokens": 10095471.0, "mean_token_accuracy": 0.7880559608340263, "epoch": 2.2720000000000002, "step": 1420 }, { "loss": 1.0215, "grad_norm": 0.5175665616989136, "learning_rate": 0.00019225018643778455, "entropy": 1.0892718441784381, "num_tokens": 10166977.0, "mean_token_accuracy": 0.761530926078558, "epoch": 2.288, "step": 1430 }, { "loss": 0.9859, "grad_norm": 0.5673478245735168, "learning_rate": 0.00019211627731629876, "entropy": 1.044171106815338, "num_tokens": 10241410.0, "mean_token_accuracy": 0.7680308371782303, "epoch": 2.304, "step": 1440 }, { "loss": 0.9876, "grad_norm": 0.5850645303726196, "learning_rate": 0.00019198126862879442, "entropy": 1.0460022101178765, "num_tokens": 10313989.0, "mean_token_accuracy": 0.7714769683778286, "epoch": 2.32, "step": 1450 }, { "loss": 0.9535, "grad_norm": 0.6382830739021301, "learning_rate": 0.00019184516198683213, "entropy": 1.03202133923769, "num_tokens": 10385717.0, "mean_token_accuracy": 0.7739889822900295, "epoch": 2.336, "step": 1460 }, { "loss": 0.8958, "grad_norm": 0.642211377620697, "learning_rate": 0.00019170795901507853, "entropy": 0.9398733902722597, "num_tokens": 10455862.0, "mean_token_accuracy": 0.7910606682300567, "epoch": 2.352, "step": 1470 }, { "loss": 0.9292, "grad_norm": 0.5539968609809875, "learning_rate": 0.0001915696613512867, "entropy": 0.9828695841133595, "num_tokens": 10527711.0, "mean_token_accuracy": 0.7850929960608483, "epoch": 2.368, "step": 1480 }, { "loss": 0.9621, "grad_norm": 0.6045369505882263, "learning_rate": 0.0001914302706462769, "entropy": 1.0211696345359087, "num_tokens": 10596998.0, "mean_token_accuracy": 0.7766989633440972, "epoch": 2.384, "step": 1490 }, { "loss": 1.0078, "grad_norm": 0.6861565113067627, "learning_rate": 0.00019128978856391667, "entropy": 1.059575468301773, "num_tokens": 10666524.0, "mean_token_accuracy": 0.7668830923736095, "epoch": 2.4, "step": 1500 }, { "loss": 0.8968, "grad_norm": 0.5144836902618408, "learning_rate": 0.00019114821678110094, "entropy": 0.9725998263806105, "num_tokens": 10738993.0, "mean_token_accuracy": 0.7878397397696972, "epoch": 2.416, "step": 1510 }, { "loss": 0.9668, "grad_norm": 0.6449041366577148, "learning_rate": 0.0001910055569877322, "entropy": 1.0108907911926508, "num_tokens": 10808666.0, "mean_token_accuracy": 0.7752245619893074, "epoch": 2.432, "step": 1520 }, { "loss": 0.9524, "grad_norm": 0.6391110420227051, "learning_rate": 0.00019086181088670014, "entropy": 1.0119984313845634, "num_tokens": 10877421.0, "mean_token_accuracy": 0.7792512528598309, "epoch": 2.448, "step": 1530 }, { "loss": 0.9773, "grad_norm": 0.553873598575592, "learning_rate": 0.00019071698019386144, "entropy": 1.0298538390547036, "num_tokens": 10947324.0, "mean_token_accuracy": 0.776578588783741, "epoch": 2.464, "step": 1540 }, { "loss": 0.966, "grad_norm": 0.6222025156021118, "learning_rate": 0.00019057106663801922, "entropy": 1.0286174569278956, "num_tokens": 11018076.0, "mean_token_accuracy": 0.7760774359107018, "epoch": 2.48, "step": 1550 }, { "loss": 0.9526, "grad_norm": 0.6243301630020142, "learning_rate": 0.00019042407196090242, "entropy": 1.0075746016576885, "num_tokens": 11092176.0, "mean_token_accuracy": 0.7758703336119652, "epoch": 2.496, "step": 1560 }, { "loss": 0.9369, "grad_norm": 0.6675435304641724, "learning_rate": 0.00019027599791714503, "entropy": 1.0192183941602706, "num_tokens": 11159537.0, "mean_token_accuracy": 0.7785408392548561, "epoch": 2.512, "step": 1570 }, { "loss": 0.9948, "grad_norm": 0.5861629247665405, "learning_rate": 0.0001901268462742652, "entropy": 1.01413600333035, "num_tokens": 11234396.0, "mean_token_accuracy": 0.77096186876297, "epoch": 2.528, "step": 1580 }, { "loss": 0.9724, "grad_norm": 0.5685335397720337, "learning_rate": 0.00018997661881264396, "entropy": 1.0443084770813584, "num_tokens": 11306679.0, "mean_token_accuracy": 0.7751825019717217, "epoch": 2.544, "step": 1590 }, { "loss": 0.9261, "grad_norm": 0.5818600654602051, "learning_rate": 0.0001898253173255042, "entropy": 0.9846690095961094, "num_tokens": 11377488.0, "mean_token_accuracy": 0.7792207457125186, "epoch": 2.56, "step": 1600 }, { "loss": 1.0246, "grad_norm": 0.6534175276756287, "learning_rate": 0.00018967294361888902, "entropy": 1.0887094482779502, "num_tokens": 11447655.0, "mean_token_accuracy": 0.7642393581569195, "epoch": 2.576, "step": 1610 }, { "loss": 0.9371, "grad_norm": 0.5200822949409485, "learning_rate": 0.0001895194995116404, "entropy": 0.9803299453109503, "num_tokens": 11519590.0, "mean_token_accuracy": 0.7849389344453812, "epoch": 2.592, "step": 1620 }, { "loss": 0.9642, "grad_norm": 0.6352348923683167, "learning_rate": 0.0001893649868353774, "entropy": 1.0169053738936782, "num_tokens": 11592021.0, "mean_token_accuracy": 0.7767709240317344, "epoch": 2.608, "step": 1630 }, { "loss": 0.9112, "grad_norm": 0.5717297792434692, "learning_rate": 0.00018920940743447426, "entropy": 0.9720978084951639, "num_tokens": 11662252.0, "mean_token_accuracy": 0.7896641224622727, "epoch": 2.624, "step": 1640 }, { "loss": 0.9235, "grad_norm": 0.6402314305305481, "learning_rate": 0.00018905276316603833, "entropy": 0.9671182595193386, "num_tokens": 11735712.0, "mean_token_accuracy": 0.7845655634999276, "epoch": 2.64, "step": 1650 }, { "loss": 1.0624, "grad_norm": 0.5721269249916077, "learning_rate": 0.00018889505589988814, "entropy": 1.1073316588997841, "num_tokens": 11806916.0, "mean_token_accuracy": 0.7572607420384884, "epoch": 2.656, "step": 1660 }, { "loss": 0.9518, "grad_norm": 0.5525614023208618, "learning_rate": 0.0001887362875185308, "entropy": 1.0050595359876753, "num_tokens": 11878726.0, "mean_token_accuracy": 0.775943149626255, "epoch": 2.672, "step": 1670 }, { "loss": 1.0203, "grad_norm": 0.5779228806495667, "learning_rate": 0.00018857645991713967, "entropy": 1.0632322260178626, "num_tokens": 11952555.0, "mean_token_accuracy": 0.7683139380067587, "epoch": 2.6879999999999997, "step": 1680 }, { "loss": 0.9063, "grad_norm": 0.6312578916549683, "learning_rate": 0.00018841557500353176, "entropy": 0.9740103092044592, "num_tokens": 12014859.0, "mean_token_accuracy": 0.788094861060381, "epoch": 2.7039999999999997, "step": 1690 }, { "loss": 0.8718, "grad_norm": 0.6351263523101807, "learning_rate": 0.00018825363469814491, "entropy": 0.9355510108172893, "num_tokens": 12086218.0, "mean_token_accuracy": 0.7947040140628815, "epoch": 2.7199999999999998, "step": 1700 }, { "loss": 0.9559, "grad_norm": 0.5946796536445618, "learning_rate": 0.0001880906409340149, "entropy": 1.017713338136673, "num_tokens": 12158813.0, "mean_token_accuracy": 0.7741880998015404, "epoch": 2.7359999999999998, "step": 1710 }, { "loss": 0.9165, "grad_norm": 0.5909619927406311, "learning_rate": 0.0001879265956567523, "entropy": 0.9765233092010022, "num_tokens": 12229336.0, "mean_token_accuracy": 0.7841075330972671, "epoch": 2.752, "step": 1720 }, { "loss": 0.9678, "grad_norm": 0.6419169902801514, "learning_rate": 0.00018776150082451922, "entropy": 1.0202443137764932, "num_tokens": 12299478.0, "mean_token_accuracy": 0.7725979194045067, "epoch": 2.768, "step": 1730 }, { "loss": 0.9347, "grad_norm": 0.5946188569068909, "learning_rate": 0.00018759535840800624, "entropy": 1.004103245586157, "num_tokens": 12371311.0, "mean_token_accuracy": 0.7796913161873817, "epoch": 2.784, "step": 1740 }, { "loss": 0.9802, "grad_norm": 0.6080518364906311, "learning_rate": 0.00018742817039040844, "entropy": 1.0344772312790156, "num_tokens": 12442140.0, "mean_token_accuracy": 0.7729787580668926, "epoch": 2.8, "step": 1750 }, { "loss": 1.0161, "grad_norm": 0.5784133672714233, "learning_rate": 0.00018725993876740206, "entropy": 1.068070276454091, "num_tokens": 12516286.0, "mean_token_accuracy": 0.7644765108823777, "epoch": 2.816, "step": 1760 }, { "loss": 0.9619, "grad_norm": 0.6517199277877808, "learning_rate": 0.0001870906655471205, "entropy": 1.0062848946079612, "num_tokens": 12585192.0, "mean_token_accuracy": 0.7779544457793236, "epoch": 2.832, "step": 1770 }, { "loss": 0.9296, "grad_norm": 0.6178792715072632, "learning_rate": 0.00018692035275013046, "entropy": 0.9763122972100973, "num_tokens": 12658170.0, "mean_token_accuracy": 0.7831808589398861, "epoch": 2.848, "step": 1780 }, { "loss": 0.9609, "grad_norm": 0.5982915759086609, "learning_rate": 0.00018674900240940777, "entropy": 1.0169632386416196, "num_tokens": 12734428.0, "mean_token_accuracy": 0.7759267628192902, "epoch": 2.864, "step": 1790 }, { "loss": 0.9752, "grad_norm": 0.5707559585571289, "learning_rate": 0.00018657661657031307, "entropy": 1.027672618255019, "num_tokens": 12806942.0, "mean_token_accuracy": 0.7737420730292797, "epoch": 2.88, "step": 1800 }, { "loss": 1.0165, "grad_norm": 0.5510401725769043, "learning_rate": 0.00018640319729056753, "entropy": 1.0580551374703646, "num_tokens": 12878581.0, "mean_token_accuracy": 0.7647794969379902, "epoch": 2.896, "step": 1810 }, { "loss": 0.9149, "grad_norm": 0.5887361764907837, "learning_rate": 0.0001862287466402282, "entropy": 0.98265972584486, "num_tokens": 12944852.0, "mean_token_accuracy": 0.7888715952634812, "epoch": 2.912, "step": 1820 }, { "loss": 0.9166, "grad_norm": 0.6644715070724487, "learning_rate": 0.00018605326670166324, "entropy": 0.9410983495414257, "num_tokens": 13017606.0, "mean_token_accuracy": 0.7900199055671692, "epoch": 2.928, "step": 1830 }, { "loss": 0.9689, "grad_norm": 0.6002123951911926, "learning_rate": 0.00018587675956952717, "entropy": 1.0307129632681609, "num_tokens": 13090730.0, "mean_token_accuracy": 0.7752901323139667, "epoch": 2.944, "step": 1840 }, { "loss": 0.9347, "grad_norm": 0.5782667398452759, "learning_rate": 0.0001856992273507359, "entropy": 0.9831677440553903, "num_tokens": 13160967.0, "mean_token_accuracy": 0.7831101983785629, "epoch": 2.96, "step": 1850 }, { "loss": 0.989, "grad_norm": 0.6209278702735901, "learning_rate": 0.00018552067216444135, "entropy": 1.0556957563385367, "num_tokens": 13231964.0, "mean_token_accuracy": 0.7674557425081729, "epoch": 2.976, "step": 1860 }, { "loss": 0.9407, "grad_norm": 0.6044370532035828, "learning_rate": 0.00018534109614200652, "entropy": 0.9933723926544189, "num_tokens": 13303444.0, "mean_token_accuracy": 0.7822301931679249, "epoch": 2.992, "step": 1870 }, { "eval_loss": 1.7862353324890137, "eval_runtime": 53.6758, "eval_samples_per_second": 4.192, "eval_steps_per_second": 4.192, "eval_entropy": 1.4615480579270257, "eval_num_tokens": 13340166.0, "eval_mean_token_accuracy": 0.6428498334354824, "epoch": 3.0, "step": 1875 }, { "loss": 0.7933, "grad_norm": 0.6109716296195984, "learning_rate": 0.00018516050142697966, "entropy": 0.8859017558395863, "num_tokens": 13376869.0, "mean_token_accuracy": 0.8128133170306683, "epoch": 3.008, "step": 1880 }, { "loss": 0.7719, "grad_norm": 0.7230874300003052, "learning_rate": 0.000184978890175069, "entropy": 0.8589608281850815, "num_tokens": 13448474.0, "mean_token_accuracy": 0.8141321405768395, "epoch": 3.024, "step": 1890 }, { "loss": 0.7632, "grad_norm": 0.7446550726890564, "learning_rate": 0.00018479626455411677, "entropy": 0.8419235173612833, "num_tokens": 13521848.0, "mean_token_accuracy": 0.8152008697390556, "epoch": 3.04, "step": 1900 }, { "loss": 0.7238, "grad_norm": 0.6568620800971985, "learning_rate": 0.00018461262674407348, "entropy": 0.8074177308008075, "num_tokens": 13596848.0, "mean_token_accuracy": 0.8276491455733777, "epoch": 3.056, "step": 1910 }, { "loss": 0.7418, "grad_norm": 0.7549406290054321, "learning_rate": 0.00018442797893697196, "entropy": 0.8381395028904081, "num_tokens": 13666231.0, "mean_token_accuracy": 0.817295140773058, "epoch": 3.072, "step": 1920 }, { "loss": 0.7353, "grad_norm": 0.6451891660690308, "learning_rate": 0.00018424232333690094, "entropy": 0.8252025406807662, "num_tokens": 13740140.0, "mean_token_accuracy": 0.8232167534530163, "epoch": 3.088, "step": 1930 }, { "loss": 0.7523, "grad_norm": 0.698747992515564, "learning_rate": 0.00018405566215997895, "entropy": 0.8380415461957454, "num_tokens": 13809347.0, "mean_token_accuracy": 0.8184896022081375, "epoch": 3.104, "step": 1940 }, { "loss": 0.7455, "grad_norm": 0.5851796865463257, "learning_rate": 0.00018386799763432782, "entropy": 0.8350590158253908, "num_tokens": 13880723.0, "mean_token_accuracy": 0.8213269144296647, "epoch": 3.12, "step": 1950 }, { "loss": 0.6849, "grad_norm": 0.7222751975059509, "learning_rate": 0.000183679332000046, "entropy": 0.7629145236685873, "num_tokens": 13949147.0, "mean_token_accuracy": 0.833473163843155, "epoch": 3.136, "step": 1960 }, { "loss": 0.7023, "grad_norm": 0.6085461378097534, "learning_rate": 0.00018348966750918205, "entropy": 0.8075309634208679, "num_tokens": 14023778.0, "mean_token_accuracy": 0.8277238517999649, "epoch": 3.152, "step": 1970 }, { "loss": 0.6965, "grad_norm": 0.6177237033843994, "learning_rate": 0.0001832990064257074, "entropy": 0.7767251085489988, "num_tokens": 14099063.0, "mean_token_accuracy": 0.8296999506652355, "epoch": 3.168, "step": 1980 }, { "loss": 0.7608, "grad_norm": 0.6110320091247559, "learning_rate": 0.00018310735102548972, "entropy": 0.8422384619712829, "num_tokens": 14169364.0, "mean_token_accuracy": 0.8171885862946511, "epoch": 3.184, "step": 1990 }, { "loss": 0.7619, "grad_norm": 0.6991633176803589, "learning_rate": 0.00018291470359626537, "entropy": 0.8386176811531186, "num_tokens": 14240791.0, "mean_token_accuracy": 0.8168883271515369, "epoch": 3.2, "step": 2000 }, { "loss": 0.8458, "grad_norm": 0.812531054019928, "learning_rate": 0.0001827210664376124, "entropy": 0.9236140789464116, "num_tokens": 14312233.0, "mean_token_accuracy": 0.7984275981783867, "epoch": 3.216, "step": 2010 }, { "loss": 0.8171, "grad_norm": 0.6983267664909363, "learning_rate": 0.00018252644186092298, "entropy": 0.9056729540228844, "num_tokens": 14384615.0, "mean_token_accuracy": 0.8071101978421211, "epoch": 3.232, "step": 2020 }, { "loss": 0.7983, "grad_norm": 0.7964588403701782, "learning_rate": 0.00018233083218937576, "entropy": 0.8779303507879377, "num_tokens": 14455767.0, "mean_token_accuracy": 0.809503474086523, "epoch": 3.248, "step": 2030 }, { "loss": 0.7672, "grad_norm": 0.7307837605476379, "learning_rate": 0.00018213423975790822, "entropy": 0.8466080158948899, "num_tokens": 14525052.0, "mean_token_accuracy": 0.8171820491552353, "epoch": 3.2640000000000002, "step": 2040 }, { "loss": 0.7947, "grad_norm": 0.7651835680007935, "learning_rate": 0.00018193666691318874, "entropy": 0.8759767279028893, "num_tokens": 14598311.0, "mean_token_accuracy": 0.8076938077807426, "epoch": 3.2800000000000002, "step": 2050 }, { "loss": 0.693, "grad_norm": 0.6537678241729736, "learning_rate": 0.00018173811601358866, "entropy": 0.775746912881732, "num_tokens": 14670585.0, "mean_token_accuracy": 0.8322148218750953, "epoch": 3.296, "step": 2060 }, { "loss": 0.75, "grad_norm": 0.7205032706260681, "learning_rate": 0.00018153858942915408, "entropy": 0.8315419342368842, "num_tokens": 14740899.0, "mean_token_accuracy": 0.8212967157363892, "epoch": 3.312, "step": 2070 }, { "loss": 0.7565, "grad_norm": 0.670572817325592, "learning_rate": 0.00018133808954157749, "entropy": 0.8450141604989767, "num_tokens": 14811232.0, "mean_token_accuracy": 0.819087627530098, "epoch": 3.328, "step": 2080 }, { "loss": 0.765, "grad_norm": 0.7283110618591309, "learning_rate": 0.00018113661874416957, "entropy": 0.8436155445873738, "num_tokens": 14880583.0, "mean_token_accuracy": 0.8132267624139786, "epoch": 3.344, "step": 2090 }, { "loss": 0.731, "grad_norm": 0.7579659223556519, "learning_rate": 0.00018093417944183034, "entropy": 0.8083607746288181, "num_tokens": 14948032.0, "mean_token_accuracy": 0.8245218083262443, "epoch": 3.36, "step": 2100 }, { "loss": 0.7631, "grad_norm": 0.8077874183654785, "learning_rate": 0.00018073077405102072, "entropy": 0.8481433935463428, "num_tokens": 15022707.0, "mean_token_accuracy": 0.8150244675576687, "epoch": 3.376, "step": 2110 }, { "loss": 0.7267, "grad_norm": 0.6470391750335693, "learning_rate": 0.0001805264049997334, "entropy": 0.8057174738496542, "num_tokens": 15095305.0, "mean_token_accuracy": 0.8232112221419812, "epoch": 3.392, "step": 2120 }, { "loss": 0.781, "grad_norm": 0.7371610403060913, "learning_rate": 0.00018032107472746412, "entropy": 0.8465482715517283, "num_tokens": 15166765.0, "mean_token_accuracy": 0.8166712678968906, "epoch": 3.408, "step": 2130 }, { "loss": 0.7452, "grad_norm": 0.7293988466262817, "learning_rate": 0.00018011478568518246, "entropy": 0.8173820916563272, "num_tokens": 15237156.0, "mean_token_accuracy": 0.8225811094045639, "epoch": 3.424, "step": 2140 }, { "loss": 0.7931, "grad_norm": 0.7543926239013672, "learning_rate": 0.0001799075403353025, "entropy": 0.8723974214866758, "num_tokens": 15308311.0, "mean_token_accuracy": 0.8110020823776722, "epoch": 3.44, "step": 2150 }, { "loss": 0.7863, "grad_norm": 0.7671536207199097, "learning_rate": 0.00017969934115165352, "entropy": 0.8596087738871574, "num_tokens": 15379395.0, "mean_token_accuracy": 0.8105585679411889, "epoch": 3.456, "step": 2160 }, { "loss": 0.7663, "grad_norm": 0.7112234830856323, "learning_rate": 0.00017949019061945046, "entropy": 0.8404063623398542, "num_tokens": 15449064.0, "mean_token_accuracy": 0.8161742366850376, "epoch": 3.472, "step": 2170 }, { "loss": 0.7836, "grad_norm": 0.6681432127952576, "learning_rate": 0.00017928009123526425, "entropy": 0.8629410825669765, "num_tokens": 15517095.0, "mean_token_accuracy": 0.8152158319950104, "epoch": 3.488, "step": 2180 }, { "loss": 0.783, "grad_norm": 0.656926155090332, "learning_rate": 0.00017906904550699194, "entropy": 0.8666257444769144, "num_tokens": 15590878.0, "mean_token_accuracy": 0.8132797740399837, "epoch": 3.504, "step": 2190 }, { "loss": 0.8065, "grad_norm": 0.6785566806793213, "learning_rate": 0.00017885705595382682, "entropy": 0.8881759837269783, "num_tokens": 15659377.0, "mean_token_accuracy": 0.8050676561892033, "epoch": 3.52, "step": 2200 }, { "loss": 0.7748, "grad_norm": 0.6876655220985413, "learning_rate": 0.00017864412510622843, "entropy": 0.8655701884999871, "num_tokens": 15733272.0, "mean_token_accuracy": 0.8117704160511494, "epoch": 3.536, "step": 2210 }, { "loss": 0.804, "grad_norm": 0.773967444896698, "learning_rate": 0.00017843025550589228, "entropy": 0.8780977323651313, "num_tokens": 15805233.0, "mean_token_accuracy": 0.8101526908576488, "epoch": 3.552, "step": 2220 }, { "loss": 0.766, "grad_norm": 0.8744268417358398, "learning_rate": 0.0001782154497057194, "entropy": 0.8369497753679752, "num_tokens": 15875629.0, "mean_token_accuracy": 0.8179697826504707, "epoch": 3.568, "step": 2230 }, { "loss": 0.8265, "grad_norm": 0.7182800769805908, "learning_rate": 0.00017799971026978608, "entropy": 0.8962526481598616, "num_tokens": 15946020.0, "mean_token_accuracy": 0.8018964633345604, "epoch": 3.584, "step": 2240 }, { "loss": 0.7448, "grad_norm": 0.8214159607887268, "learning_rate": 0.00017778303977331305, "entropy": 0.8260738991200924, "num_tokens": 16017717.0, "mean_token_accuracy": 0.819572264701128, "epoch": 3.6, "step": 2250 }, { "loss": 0.8046, "grad_norm": 0.7237060070037842, "learning_rate": 0.00017756544080263495, "entropy": 0.8852400667965412, "num_tokens": 16085024.0, "mean_token_accuracy": 0.8068661950528622, "epoch": 3.616, "step": 2260 }, { "loss": 0.7666, "grad_norm": 0.6629997491836548, "learning_rate": 0.00017734691595516934, "entropy": 0.8444820886477828, "num_tokens": 16155472.0, "mean_token_accuracy": 0.8162441961467266, "epoch": 3.632, "step": 2270 }, { "loss": 0.9041, "grad_norm": 0.7487677931785583, "learning_rate": 0.00017712746783938563, "entropy": 0.9649901568889618, "num_tokens": 16226871.0, "mean_token_accuracy": 0.7905729107558728, "epoch": 3.648, "step": 2280 }, { "loss": 0.7573, "grad_norm": 0.7176128029823303, "learning_rate": 0.00017690709907477412, "entropy": 0.8290225496515632, "num_tokens": 16299541.0, "mean_token_accuracy": 0.818456943333149, "epoch": 3.664, "step": 2290 }, { "loss": 0.743, "grad_norm": 0.7029726505279541, "learning_rate": 0.00017668581229181456, "entropy": 0.8121594816446305, "num_tokens": 16371492.0, "mean_token_accuracy": 0.8230444081127644, "epoch": 3.68, "step": 2300 }, { "loss": 0.8107, "grad_norm": 0.86060631275177, "learning_rate": 0.00017646361013194488, "entropy": 0.8850367331877351, "num_tokens": 16442710.0, "mean_token_accuracy": 0.8074446856975556, "epoch": 3.6959999999999997, "step": 2310 }, { "loss": 0.7706, "grad_norm": 0.8046766519546509, "learning_rate": 0.00017624049524752954, "entropy": 0.8499442409723997, "num_tokens": 16512131.0, "mean_token_accuracy": 0.8143888004124165, "epoch": 3.7119999999999997, "step": 2320 }, { "loss": 0.7919, "grad_norm": 0.6990330815315247, "learning_rate": 0.00017601647030182806, "entropy": 0.8634312754496932, "num_tokens": 16584287.0, "mean_token_accuracy": 0.8127248801290989, "epoch": 3.7279999999999998, "step": 2330 }, { "loss": 0.7862, "grad_norm": 0.7886999249458313, "learning_rate": 0.00017579153796896298, "entropy": 0.85138991009444, "num_tokens": 16653160.0, "mean_token_accuracy": 0.8142315901815891, "epoch": 3.7439999999999998, "step": 2340 }, { "loss": 0.7215, "grad_norm": 0.8504064679145813, "learning_rate": 0.00017556570093388806, "entropy": 0.8016205428168177, "num_tokens": 16720677.0, "mean_token_accuracy": 0.8230935603380203, "epoch": 3.76, "step": 2350 }, { "loss": 0.7919, "grad_norm": 0.6830351948738098, "learning_rate": 0.00017533896189235636, "entropy": 0.8867231639102101, "num_tokens": 16788771.0, "mean_token_accuracy": 0.8097460068762302, "epoch": 3.776, "step": 2360 }, { "loss": 0.7886, "grad_norm": 0.7679236531257629, "learning_rate": 0.00017511132355088783, "entropy": 0.844105114787817, "num_tokens": 16855464.0, "mean_token_accuracy": 0.8134540401399135, "epoch": 3.792, "step": 2370 }, { "loss": 0.7747, "grad_norm": 0.8023110032081604, "learning_rate": 0.0001748827886267372, "entropy": 0.8556949567049742, "num_tokens": 16925321.0, "mean_token_accuracy": 0.8127300873398781, "epoch": 3.808, "step": 2380 }, { "loss": 0.838, "grad_norm": 0.7937261462211609, "learning_rate": 0.0001746533598478613, "entropy": 0.8925119020044804, "num_tokens": 16999589.0, "mean_token_accuracy": 0.8020697735249996, "epoch": 3.824, "step": 2390 }, { "loss": 0.7605, "grad_norm": 0.7578120827674866, "learning_rate": 0.00017442303995288678, "entropy": 0.8391670389100909, "num_tokens": 17074030.0, "mean_token_accuracy": 0.8168841011822223, "epoch": 3.84, "step": 2400 }, { "loss": 0.7414, "grad_norm": 0.799427330493927, "learning_rate": 0.00017419183169107728, "entropy": 0.8241954291239381, "num_tokens": 17146241.0, "mean_token_accuracy": 0.8191890604794025, "epoch": 3.856, "step": 2410 }, { "loss": 0.7981, "grad_norm": 0.6652328968048096, "learning_rate": 0.0001739597378223006, "entropy": 0.8926442207768559, "num_tokens": 17217006.0, "mean_token_accuracy": 0.8100546926259995, "epoch": 3.872, "step": 2420 }, { "loss": 0.7331, "grad_norm": 0.726668119430542, "learning_rate": 0.00017372676111699576, "entropy": 0.8043451055884361, "num_tokens": 17286998.0, "mean_token_accuracy": 0.8223776213824749, "epoch": 3.888, "step": 2430 }, { "loss": 0.7706, "grad_norm": 0.7493301630020142, "learning_rate": 0.00017349290435614, "entropy": 0.8444451358169317, "num_tokens": 17358103.0, "mean_token_accuracy": 0.8147278741002083, "epoch": 3.904, "step": 2440 }, { "loss": 0.8582, "grad_norm": 0.9194010496139526, "learning_rate": 0.0001732581703312155, "entropy": 0.92505933791399, "num_tokens": 17431540.0, "mean_token_accuracy": 0.7952933654189109, "epoch": 3.92, "step": 2450 }, { "loss": 0.8141, "grad_norm": 0.8147196769714355, "learning_rate": 0.00017302256184417608, "entropy": 0.8854448474943638, "num_tokens": 17500514.0, "mean_token_accuracy": 0.8034609884023667, "epoch": 3.936, "step": 2460 }, { "loss": 0.7381, "grad_norm": 0.7713481783866882, "learning_rate": 0.00017278608170741383, "entropy": 0.8149165739305317, "num_tokens": 17572041.0, "mean_token_accuracy": 0.8233816765248776, "epoch": 3.952, "step": 2470 }, { "loss": 0.7965, "grad_norm": 0.6995890140533447, "learning_rate": 0.00017254873274372544, "entropy": 0.8742475273087621, "num_tokens": 17644776.0, "mean_token_accuracy": 0.8078553505241871, "epoch": 3.968, "step": 2480 }, { "loss": 0.8228, "grad_norm": 0.745158851146698, "learning_rate": 0.0001723105177862785, "entropy": 0.8932803479023278, "num_tokens": 17715872.0, "mean_token_accuracy": 0.8029817044734955, "epoch": 3.984, "step": 2490 }, { "loss": 0.7964, "grad_norm": 0.6700334548950195, "learning_rate": 0.00017207143967857777, "entropy": 0.856347457319498, "num_tokens": 17786888.0, "mean_token_accuracy": 0.8130462281405926, "epoch": 4.0, "step": 2500 }, { "eval_loss": 1.895753026008606, "eval_runtime": 53.8365, "eval_samples_per_second": 4.179, "eval_steps_per_second": 4.179, "eval_entropy": 1.3692996825112236, "eval_num_tokens": 17786888.0, "eval_mean_token_accuracy": 0.6342869651317596, "epoch": 4.0, "step": 2500 }, { "loss": 0.6065, "grad_norm": 0.8600279092788696, "learning_rate": 0.0001718315012744312, "entropy": 0.7013209199532866, "num_tokens": 17856909.0, "mean_token_accuracy": 0.8537710405886173, "epoch": 4.016, "step": 2510 }, { "loss": 0.5941, "grad_norm": 0.9425161480903625, "learning_rate": 0.00017159070543791582, "entropy": 0.7158705178648234, "num_tokens": 17928998.0, "mean_token_accuracy": 0.8561614014208316, "epoch": 4.032, "step": 2520 }, { "loss": 0.5547, "grad_norm": 0.7796197533607483, "learning_rate": 0.00017134905504334364, "entropy": 0.664517630636692, "num_tokens": 17996595.0, "mean_token_accuracy": 0.8621847502887249, "epoch": 4.048, "step": 2530 }, { "loss": 0.5959, "grad_norm": 0.9111642241477966, "learning_rate": 0.0001711065529752272, "entropy": 0.6948346728459001, "num_tokens": 18066756.0, "mean_token_accuracy": 0.8552937671542168, "epoch": 4.064, "step": 2540 }, { "loss": 0.5971, "grad_norm": 0.9173292517662048, "learning_rate": 0.00017086320212824537, "entropy": 0.6907664563506841, "num_tokens": 18137732.0, "mean_token_accuracy": 0.8559239439666271, "epoch": 4.08, "step": 2550 }, { "loss": 0.6156, "grad_norm": 0.8057093024253845, "learning_rate": 0.0001706190054072085, "entropy": 0.7114027110859752, "num_tokens": 18213084.0, "mean_token_accuracy": 0.8506132751703263, "epoch": 4.096, "step": 2560 }, { "loss": 0.5725, "grad_norm": 0.6977136135101318, "learning_rate": 0.000170373965727024, "entropy": 0.6711609566584229, "num_tokens": 18284661.0, "mean_token_accuracy": 0.8609179303050041, "epoch": 4.112, "step": 2570 }, { "loss": 0.6022, "grad_norm": 0.8515857458114624, "learning_rate": 0.00017012808601266137, "entropy": 0.710867534019053, "num_tokens": 18356699.0, "mean_token_accuracy": 0.8519293628633022, "epoch": 4.128, "step": 2580 }, { "loss": 0.6151, "grad_norm": 0.9551877975463867, "learning_rate": 0.0001698813691991174, "entropy": 0.7186268895864487, "num_tokens": 18428017.0, "mean_token_accuracy": 0.8500192128121853, "epoch": 4.144, "step": 2590 }, { "loss": 0.5998, "grad_norm": 0.9054930806159973, "learning_rate": 0.0001696338182313812, "entropy": 0.7155788550153375, "num_tokens": 18504738.0, "mean_token_accuracy": 0.8535547323524952, "epoch": 4.16, "step": 2600 }, { "loss": 0.5752, "grad_norm": 0.800977885723114, "learning_rate": 0.00016938543606439876, "entropy": 0.6714572484605015, "num_tokens": 18576944.0, "mean_token_accuracy": 0.8593325287103653, "epoch": 4.176, "step": 2610 }, { "loss": 0.6253, "grad_norm": 0.9490655064582825, "learning_rate": 0.0001691362256630379, "entropy": 0.7263286361470819, "num_tokens": 18646400.0, "mean_token_accuracy": 0.8469728611409664, "epoch": 4.192, "step": 2620 }, { "loss": 0.6196, "grad_norm": 0.7594051361083984, "learning_rate": 0.000168886190002053, "entropy": 0.7263486094772815, "num_tokens": 18717770.0, "mean_token_accuracy": 0.8494671188294888, "epoch": 4.208, "step": 2630 }, { "loss": 0.5928, "grad_norm": 0.8508740663528442, "learning_rate": 0.00016863533206604915, "entropy": 0.6983879346400499, "num_tokens": 18788959.0, "mean_token_accuracy": 0.8548591129481793, "epoch": 4.224, "step": 2640 }, { "loss": 0.5663, "grad_norm": 0.7377139329910278, "learning_rate": 0.0001683836548494468, "entropy": 0.6695951338857412, "num_tokens": 18862470.0, "mean_token_accuracy": 0.8594780787825584, "epoch": 4.24, "step": 2650 }, { "loss": 0.5799, "grad_norm": 0.8782995343208313, "learning_rate": 0.00016813116135644586, "entropy": 0.6822380177676678, "num_tokens": 18929233.0, "mean_token_accuracy": 0.858181843161583, "epoch": 4.256, "step": 2660 }, { "loss": 0.6115, "grad_norm": 0.6729841828346252, "learning_rate": 0.00016787785460098994, "entropy": 0.7198176568374037, "num_tokens": 19001007.0, "mean_token_accuracy": 0.8482233792543411, "epoch": 4.272, "step": 2670 }, { "loss": 0.592, "grad_norm": 0.8773292899131775, "learning_rate": 0.00016762373760673035, "entropy": 0.7007175114005804, "num_tokens": 19072098.0, "mean_token_accuracy": 0.85381373539567, "epoch": 4.288, "step": 2680 }, { "loss": 0.665, "grad_norm": 0.9225711822509766, "learning_rate": 0.00016736881340698994, "entropy": 0.7667893706820905, "num_tokens": 19141902.0, "mean_token_accuracy": 0.8400018438696861, "epoch": 4.304, "step": 2690 }, { "loss": 0.588, "grad_norm": 0.7835565209388733, "learning_rate": 0.00016711308504472702, "entropy": 0.6843621030449867, "num_tokens": 19212632.0, "mean_token_accuracy": 0.859191533178091, "epoch": 4.32, "step": 2700 }, { "loss": 0.6128, "grad_norm": 0.9108113050460815, "learning_rate": 0.00016685655557249887, "entropy": 0.6970704590901733, "num_tokens": 19282263.0, "mean_token_accuracy": 0.8525190979242325, "epoch": 4.336, "step": 2710 }, { "loss": 0.6164, "grad_norm": 0.8702614307403564, "learning_rate": 0.00016659922805242544, "entropy": 0.715700453799218, "num_tokens": 19355683.0, "mean_token_accuracy": 0.8466007210314274, "epoch": 4.352, "step": 2720 }, { "loss": 0.5301, "grad_norm": 0.7249788045883179, "learning_rate": 0.0001663411055561528, "entropy": 0.6174096701666713, "num_tokens": 19429071.0, "mean_token_accuracy": 0.8702598512172699, "epoch": 4.368, "step": 2730 }, { "loss": 0.6203, "grad_norm": 0.7559807896614075, "learning_rate": 0.0001660821911648163, "entropy": 0.7215574221685529, "num_tokens": 19496754.0, "mean_token_accuracy": 0.8482370212674141, "epoch": 4.384, "step": 2740 }, { "loss": 0.6122, "grad_norm": 0.8586194515228271, "learning_rate": 0.00016582248796900408, "entropy": 0.7097755916416645, "num_tokens": 19571654.0, "mean_token_accuracy": 0.8501580350100995, "epoch": 4.4, "step": 2750 }, { "loss": 0.5859, "grad_norm": 0.8493603467941284, "learning_rate": 0.00016556199906871987, "entropy": 0.6742020858451724, "num_tokens": 19643846.0, "mean_token_accuracy": 0.8590623654425145, "epoch": 4.416, "step": 2760 }, { "loss": 0.5931, "grad_norm": 0.9204057455062866, "learning_rate": 0.00016530072757334625, "entropy": 0.6836666258983314, "num_tokens": 19715550.0, "mean_token_accuracy": 0.8557057566940784, "epoch": 4.432, "step": 2770 }, { "loss": 0.608, "grad_norm": 0.713733434677124, "learning_rate": 0.0001650386766016073, "entropy": 0.7064328147098422, "num_tokens": 19785315.0, "mean_token_accuracy": 0.8526691488921643, "epoch": 4.448, "step": 2780 }, { "loss": 0.6407, "grad_norm": 0.7859025001525879, "learning_rate": 0.00016477584928153164, "entropy": 0.7297217072919011, "num_tokens": 19856422.0, "mean_token_accuracy": 0.8467439651489258, "epoch": 4.464, "step": 2790 }, { "loss": 0.6593, "grad_norm": 0.9457672834396362, "learning_rate": 0.0001645122487504147, "entropy": 0.7447050439193845, "num_tokens": 19926383.0, "mean_token_accuracy": 0.8418301861733198, "epoch": 4.48, "step": 2800 }, { "loss": 0.5812, "grad_norm": 0.8376542925834656, "learning_rate": 0.00016424787815478182, "entropy": 0.6817172725684941, "num_tokens": 19995621.0, "mean_token_accuracy": 0.8584077559411526, "epoch": 4.496, "step": 2810 }, { "loss": 0.6164, "grad_norm": 0.7870206236839294, "learning_rate": 0.00016398274065035015, "entropy": 0.7246623629704118, "num_tokens": 20069704.0, "mean_token_accuracy": 0.8500882290303707, "epoch": 4.5120000000000005, "step": 2820 }, { "loss": 0.628, "grad_norm": 0.9741735458374023, "learning_rate": 0.00016371683940199133, "entropy": 0.7153911519795656, "num_tokens": 20141806.0, "mean_token_accuracy": 0.849506089836359, "epoch": 4.5280000000000005, "step": 2830 }, { "loss": 0.6287, "grad_norm": 0.853892982006073, "learning_rate": 0.0001634501775836935, "entropy": 0.7306448003277183, "num_tokens": 20211466.0, "mean_token_accuracy": 0.8433212049305439, "epoch": 4.5440000000000005, "step": 2840 }, { "loss": 0.6338, "grad_norm": 0.776993989944458, "learning_rate": 0.00016318275837852363, "entropy": 0.742659255489707, "num_tokens": 20278910.0, "mean_token_accuracy": 0.8419442869722843, "epoch": 4.5600000000000005, "step": 2850 }, { "loss": 0.6209, "grad_norm": 0.9394012689590454, "learning_rate": 0.0001629145849785893, "entropy": 0.7120476359501481, "num_tokens": 20353300.0, "mean_token_accuracy": 0.8490571916103363, "epoch": 4.576, "step": 2860 }, { "loss": 0.5807, "grad_norm": 0.8927264213562012, "learning_rate": 0.00016264566058500076, "entropy": 0.6852920278906822, "num_tokens": 20425039.0, "mean_token_accuracy": 0.8551250107586383, "epoch": 4.592, "step": 2870 }, { "loss": 0.626, "grad_norm": 0.9679337739944458, "learning_rate": 0.00016237598840783263, "entropy": 0.7085732834413647, "num_tokens": 20496082.0, "mean_token_accuracy": 0.8492946833372116, "epoch": 4.608, "step": 2880 }, { "loss": 0.5596, "grad_norm": 0.8522034883499146, "learning_rate": 0.00016210557166608562, "entropy": 0.6630181092768908, "num_tokens": 20563064.0, "mean_token_accuracy": 0.8628097176551819, "epoch": 4.624, "step": 2890 }, { "loss": 0.6449, "grad_norm": 0.8708239793777466, "learning_rate": 0.00016183441358764812, "entropy": 0.7344071606174112, "num_tokens": 20637410.0, "mean_token_accuracy": 0.839364691823721, "epoch": 4.64, "step": 2900 }, { "loss": 0.6077, "grad_norm": 0.8663641214370728, "learning_rate": 0.00016156251740925755, "entropy": 0.7113010743632913, "num_tokens": 20709981.0, "mean_token_accuracy": 0.8464154809713363, "epoch": 4.656, "step": 2910 }, { "loss": 0.625, "grad_norm": 0.7439706921577454, "learning_rate": 0.00016128988637646204, "entropy": 0.7173363106325269, "num_tokens": 20780504.0, "mean_token_accuracy": 0.844101945310831, "epoch": 4.672, "step": 2920 }, { "loss": 0.6103, "grad_norm": 1.0111740827560425, "learning_rate": 0.00016101652374358116, "entropy": 0.7269579544663429, "num_tokens": 20849371.0, "mean_token_accuracy": 0.8479768209159374, "epoch": 4.688, "step": 2930 }, { "loss": 0.6781, "grad_norm": 0.8146656155586243, "learning_rate": 0.00016074243277366764, "entropy": 0.7614334810525178, "num_tokens": 20918901.0, "mean_token_accuracy": 0.8370049424469471, "epoch": 4.704, "step": 2940 }, { "loss": 0.6208, "grad_norm": 0.849168598651886, "learning_rate": 0.0001604676167384681, "entropy": 0.7172065624967218, "num_tokens": 20989322.0, "mean_token_accuracy": 0.8474414400756359, "epoch": 4.72, "step": 2950 }, { "loss": 0.5781, "grad_norm": 1.0044703483581543, "learning_rate": 0.00016019207891838395, "entropy": 0.6904253536835313, "num_tokens": 21057828.0, "mean_token_accuracy": 0.8542206771671772, "epoch": 4.736, "step": 2960 }, { "loss": 0.6432, "grad_norm": 0.9523283839225769, "learning_rate": 0.00015991582260243246, "entropy": 0.735679992288351, "num_tokens": 21132657.0, "mean_token_accuracy": 0.8427156813442707, "epoch": 4.752, "step": 2970 }, { "loss": 0.6666, "grad_norm": 0.8836421370506287, "learning_rate": 0.00015963885108820743, "entropy": 0.7626718487590551, "num_tokens": 21204931.0, "mean_token_accuracy": 0.835651733726263, "epoch": 4.768, "step": 2980 }, { "loss": 0.603, "grad_norm": 0.8655375242233276, "learning_rate": 0.00015936116768183959, "entropy": 0.692391081340611, "num_tokens": 21277764.0, "mean_token_accuracy": 0.8507428117096424, "epoch": 4.784, "step": 2990 }, { "loss": 0.6146, "grad_norm": 0.8820854425430298, "learning_rate": 0.00015908277569795745, "entropy": 0.7245607381686568, "num_tokens": 21348086.0, "mean_token_accuracy": 0.8458702847361564, "epoch": 4.8, "step": 3000 }, { "loss": 0.6231, "grad_norm": 0.8185605406761169, "learning_rate": 0.0001588036784596476, "entropy": 0.7096690386533737, "num_tokens": 21417198.0, "mean_token_accuracy": 0.8479459665715694, "epoch": 4.816, "step": 3010 }, { "loss": 0.5982, "grad_norm": 0.9467663168907166, "learning_rate": 0.00015852387929841513, "entropy": 0.7007859252393246, "num_tokens": 21488221.0, "mean_token_accuracy": 0.8508504040539264, "epoch": 4.832, "step": 3020 }, { "loss": 0.6126, "grad_norm": 0.7923316955566406, "learning_rate": 0.00015824338155414358, "entropy": 0.7072218367829919, "num_tokens": 21560355.0, "mean_token_accuracy": 0.8495206698775292, "epoch": 4.848, "step": 3030 }, { "loss": 0.6637, "grad_norm": 0.8898075222969055, "learning_rate": 0.00015796218857505546, "entropy": 0.7505927115678788, "num_tokens": 21631190.0, "mean_token_accuracy": 0.8387955226004123, "epoch": 4.864, "step": 3040 }, { "loss": 0.6317, "grad_norm": 0.8147240281105042, "learning_rate": 0.00015768030371767205, "entropy": 0.7208523044362665, "num_tokens": 21701523.0, "mean_token_accuracy": 0.8456745684146881, "epoch": 4.88, "step": 3050 }, { "loss": 0.5892, "grad_norm": 0.8324803113937378, "learning_rate": 0.00015739773034677339, "entropy": 0.6744862981140614, "num_tokens": 21772199.0, "mean_token_accuracy": 0.8562620833516121, "epoch": 4.896, "step": 3060 }, { "loss": 0.6592, "grad_norm": 0.9161856174468994, "learning_rate": 0.00015711447183535806, "entropy": 0.763653089851141, "num_tokens": 21842613.0, "mean_token_accuracy": 0.8374100357294083, "epoch": 4.912, "step": 3070 }, { "loss": 0.6836, "grad_norm": 0.8156411051750183, "learning_rate": 0.00015683053156460302, "entropy": 0.7553671417757869, "num_tokens": 21912124.0, "mean_token_accuracy": 0.8351289607584477, "epoch": 4.928, "step": 3080 }, { "loss": 0.6268, "grad_norm": 0.8647060394287109, "learning_rate": 0.00015654591292382322, "entropy": 0.7259074050933123, "num_tokens": 21984815.0, "mean_token_accuracy": 0.8446345880627633, "epoch": 4.944, "step": 3090 }, { "loss": 0.5949, "grad_norm": 0.8052799105644226, "learning_rate": 0.00015626061931043106, "entropy": 0.6884508535265923, "num_tokens": 22054355.0, "mean_token_accuracy": 0.8516699656844139, "epoch": 4.96, "step": 3100 }, { "loss": 0.646, "grad_norm": 0.8996228575706482, "learning_rate": 0.00015597465412989597, "entropy": 0.7440328940749168, "num_tokens": 22124201.0, "mean_token_accuracy": 0.840420650690794, "epoch": 4.976, "step": 3110 }, { "loss": 0.631, "grad_norm": 0.8454340100288391, "learning_rate": 0.00015568802079570358, "entropy": 0.731732152402401, "num_tokens": 22196608.0, "mean_token_accuracy": 0.844842928647995, "epoch": 4.992, "step": 3120 }, { "eval_loss": 2.023507833480835, "eval_runtime": 53.7914, "eval_samples_per_second": 4.183, "eval_steps_per_second": 4.183, "eval_entropy": 1.2990507414605883, "eval_num_tokens": 22233610.0, "eval_mean_token_accuracy": 0.6262136620945401, "epoch": 5.0, "step": 3125 }, { "loss": 0.5654, "grad_norm": 0.7261621952056885, "learning_rate": 0.00015540072272931518, "entropy": 0.6741341140121222, "num_tokens": 22271065.0, "mean_token_accuracy": 0.8656918816268444, "epoch": 5.008, "step": 3130 }, { "loss": 0.4675, "grad_norm": 0.9127357006072998, "learning_rate": 0.00015511276336012682, "entropy": 0.5709282426163554, "num_tokens": 22340450.0, "mean_token_accuracy": 0.8878632701933384, "epoch": 5.024, "step": 3140 }, { "loss": 0.4129, "grad_norm": 1.0269066095352173, "learning_rate": 0.00015482414612542816, "entropy": 0.5165249770507216, "num_tokens": 22414661.0, "mean_token_accuracy": 0.8991757407784462, "epoch": 5.04, "step": 3150 }, { "loss": 0.4111, "grad_norm": 0.9042404294013977, "learning_rate": 0.00015453487447036172, "entropy": 0.539594710431993, "num_tokens": 22483973.0, "mean_token_accuracy": 0.8982367627322674, "epoch": 5.056, "step": 3160 }, { "loss": 0.4628, "grad_norm": 0.9346851110458374, "learning_rate": 0.00015424495184788173, "entropy": 0.5479174485430122, "num_tokens": 22557960.0, "mean_token_accuracy": 0.8879766039550304, "epoch": 5.072, "step": 3170 }, { "loss": 0.4515, "grad_norm": 0.940280020236969, "learning_rate": 0.0001539543817187127, "entropy": 0.5730179216712713, "num_tokens": 22631904.0, "mean_token_accuracy": 0.8899232544004917, "epoch": 5.088, "step": 3180 }, { "loss": 0.4629, "grad_norm": 0.8844811916351318, "learning_rate": 0.00015366316755130829, "entropy": 0.567308340780437, "num_tokens": 22697894.0, "mean_token_accuracy": 0.8906447403132915, "epoch": 5.104, "step": 3190 }, { "loss": 0.394, "grad_norm": 0.9145937561988831, "learning_rate": 0.0001533713128218099, "entropy": 0.5068852055817843, "num_tokens": 22771050.0, "mean_token_accuracy": 0.9021118730306625, "epoch": 5.12, "step": 3200 }, { "loss": 0.4902, "grad_norm": 1.0215147733688354, "learning_rate": 0.0001530788210140051, "entropy": 0.5933262527920306, "num_tokens": 22839474.0, "mean_token_accuracy": 0.883255897462368, "epoch": 5.136, "step": 3210 }, { "loss": 0.4244, "grad_norm": 1.0089625120162964, "learning_rate": 0.00015278569561928614, "entropy": 0.5247660465538502, "num_tokens": 22913625.0, "mean_token_accuracy": 0.8956619575619698, "epoch": 5.152, "step": 3220 }, { "loss": 0.4653, "grad_norm": 0.8481457233428955, "learning_rate": 0.0001524919401366081, "entropy": 0.580933877453208, "num_tokens": 22984579.0, "mean_token_accuracy": 0.8867798656225204, "epoch": 5.168, "step": 3230 }, { "loss": 0.4804, "grad_norm": 1.1692826747894287, "learning_rate": 0.00015219755807244735, "entropy": 0.5918811490759254, "num_tokens": 23055692.0, "mean_token_accuracy": 0.8845623731613159, "epoch": 5.184, "step": 3240 }, { "loss": 0.4477, "grad_norm": 0.8568065762519836, "learning_rate": 0.00015190255294075951, "entropy": 0.5466854967176914, "num_tokens": 23129872.0, "mean_token_accuracy": 0.8907463282346726, "epoch": 5.2, "step": 3250 }, { "loss": 0.424, "grad_norm": 0.8651702404022217, "learning_rate": 0.00015160692826293772, "entropy": 0.5296070985496044, "num_tokens": 23201295.0, "mean_token_accuracy": 0.8952592253684998, "epoch": 5.216, "step": 3260 }, { "loss": 0.4431, "grad_norm": 0.8434457182884216, "learning_rate": 0.00015131068756777028, "entropy": 0.5530296273529529, "num_tokens": 23271944.0, "mean_token_accuracy": 0.8892045259475708, "epoch": 5.232, "step": 3270 }, { "loss": 0.4905, "grad_norm": 1.016168475151062, "learning_rate": 0.00015101383439139885, "entropy": 0.6148540241643786, "num_tokens": 23343369.0, "mean_token_accuracy": 0.8775349244475364, "epoch": 5.248, "step": 3280 }, { "loss": 0.4591, "grad_norm": 0.74690181016922, "learning_rate": 0.00015071637227727608, "entropy": 0.5617364960722625, "num_tokens": 23415420.0, "mean_token_accuracy": 0.8863817408680916, "epoch": 5.264, "step": 3290 }, { "loss": 0.445, "grad_norm": 0.9008899927139282, "learning_rate": 0.0001504183047761232, "entropy": 0.5419710614718497, "num_tokens": 23487624.0, "mean_token_accuracy": 0.8912027768790722, "epoch": 5.28, "step": 3300 }, { "loss": 0.4305, "grad_norm": 1.026107668876648, "learning_rate": 0.00015011963544588806, "entropy": 0.5422392937354743, "num_tokens": 23556796.0, "mean_token_accuracy": 0.8931495890021324, "epoch": 5.296, "step": 3310 }, { "loss": 0.4815, "grad_norm": 0.7754479050636292, "learning_rate": 0.00014982036785170211, "entropy": 0.5962111439555884, "num_tokens": 23624761.0, "mean_token_accuracy": 0.8800701349973679, "epoch": 5.312, "step": 3320 }, { "loss": 0.4873, "grad_norm": 1.1638911962509155, "learning_rate": 0.00014952050556583824, "entropy": 0.5943260421045125, "num_tokens": 23697411.0, "mean_token_accuracy": 0.8819492779672146, "epoch": 5.328, "step": 3330 }, { "loss": 0.4625, "grad_norm": 0.9679837226867676, "learning_rate": 0.00014922005216766793, "entropy": 0.575105587206781, "num_tokens": 23769926.0, "mean_token_accuracy": 0.8842086799442768, "epoch": 5.344, "step": 3340 }, { "loss": 0.4617, "grad_norm": 0.9896945953369141, "learning_rate": 0.00014891901124361868, "entropy": 0.5718484384939074, "num_tokens": 23839206.0, "mean_token_accuracy": 0.8856840580701828, "epoch": 5.36, "step": 3350 }, { "loss": 0.4841, "grad_norm": 0.9680900573730469, "learning_rate": 0.00014861738638713106, "entropy": 0.6036011801101268, "num_tokens": 23912417.0, "mean_token_accuracy": 0.8791848108172416, "epoch": 5.376, "step": 3360 }, { "loss": 0.462, "grad_norm": 0.88089919090271, "learning_rate": 0.00014831518119861597, "entropy": 0.5620745504274964, "num_tokens": 23983115.0, "mean_token_accuracy": 0.8861476197838783, "epoch": 5.392, "step": 3370 }, { "loss": 0.4372, "grad_norm": 0.8471999168395996, "learning_rate": 0.00014801239928541142, "entropy": 0.5359208431094885, "num_tokens": 24057358.0, "mean_token_accuracy": 0.8947220787405967, "epoch": 5.408, "step": 3380 }, { "loss": 0.4605, "grad_norm": 0.9373394846916199, "learning_rate": 0.0001477090442617397, "entropy": 0.5680675610899926, "num_tokens": 24128868.0, "mean_token_accuracy": 0.8858709551393986, "epoch": 5.424, "step": 3390 }, { "loss": 0.4455, "grad_norm": 0.8329901099205017, "learning_rate": 0.00014740511974866425, "entropy": 0.5457164883613587, "num_tokens": 24201126.0, "mean_token_accuracy": 0.8884629659354687, "epoch": 5.44, "step": 3400 }, { "loss": 0.4482, "grad_norm": 0.8197970986366272, "learning_rate": 0.0001471006293740461, "entropy": 0.5778754077851772, "num_tokens": 24269573.0, "mean_token_accuracy": 0.8853156208992005, "epoch": 5.456, "step": 3410 }, { "loss": 0.4825, "grad_norm": 0.853316068649292, "learning_rate": 0.00014679557677250112, "entropy": 0.5823379795998335, "num_tokens": 24341536.0, "mean_token_accuracy": 0.8856781981885433, "epoch": 5.4719999999999995, "step": 3420 }, { "loss": 0.4371, "grad_norm": 0.8373925089836121, "learning_rate": 0.00014648996558535606, "entropy": 0.5256961174309254, "num_tokens": 24412868.0, "mean_token_accuracy": 0.8929714098572731, "epoch": 5.4879999999999995, "step": 3430 }, { "loss": 0.4502, "grad_norm": 0.8919670581817627, "learning_rate": 0.00014618379946060552, "entropy": 0.5522337751463056, "num_tokens": 24489433.0, "mean_token_accuracy": 0.8885155335068703, "epoch": 5.504, "step": 3440 }, { "loss": 0.4894, "grad_norm": 0.8846027851104736, "learning_rate": 0.00014587708205286815, "entropy": 0.5992745257914066, "num_tokens": 24564741.0, "mean_token_accuracy": 0.8783063404262066, "epoch": 5.52, "step": 3450 }, { "loss": 0.517, "grad_norm": 1.086386799812317, "learning_rate": 0.0001455698170233431, "entropy": 0.6178754404187202, "num_tokens": 24635767.0, "mean_token_accuracy": 0.8740144841372967, "epoch": 5.536, "step": 3460 }, { "loss": 0.4955, "grad_norm": 1.1269433498382568, "learning_rate": 0.00014526200803976637, "entropy": 0.6003208708018064, "num_tokens": 24708347.0, "mean_token_accuracy": 0.8766027115285396, "epoch": 5.552, "step": 3470 }, { "loss": 0.4322, "grad_norm": 0.9048835635185242, "learning_rate": 0.000144953658776367, "entropy": 0.5393293729983271, "num_tokens": 24779091.0, "mean_token_accuracy": 0.892086160928011, "epoch": 5.568, "step": 3480 }, { "loss": 0.4636, "grad_norm": 1.0130865573883057, "learning_rate": 0.00014464477291382315, "entropy": 0.5619299493730068, "num_tokens": 24856837.0, "mean_token_accuracy": 0.8865033619105815, "epoch": 5.584, "step": 3490 }, { "loss": 0.4773, "grad_norm": 1.0342962741851807, "learning_rate": 0.00014433535413921821, "entropy": 0.584529060125351, "num_tokens": 24925012.0, "mean_token_accuracy": 0.8826412722468376, "epoch": 5.6, "step": 3500 }, { "loss": 0.5045, "grad_norm": 1.0701674222946167, "learning_rate": 0.00014402540614599687, "entropy": 0.6082589998841286, "num_tokens": 24997613.0, "mean_token_accuracy": 0.8738044142723084, "epoch": 5.616, "step": 3510 }, { "loss": 0.5231, "grad_norm": 1.0346108675003052, "learning_rate": 0.0001437149326339208, "entropy": 0.6407520338892937, "num_tokens": 25065793.0, "mean_token_accuracy": 0.8714756190776825, "epoch": 5.632, "step": 3520 }, { "loss": 0.5037, "grad_norm": 1.062588095664978, "learning_rate": 0.00014340393730902484, "entropy": 0.6168096417561173, "num_tokens": 25134646.0, "mean_token_accuracy": 0.8722797758877278, "epoch": 5.648, "step": 3530 }, { "loss": 0.4856, "grad_norm": 0.8903110027313232, "learning_rate": 0.0001430924238835724, "entropy": 0.590385424066335, "num_tokens": 25207526.0, "mean_token_accuracy": 0.8815938279032707, "epoch": 5.664, "step": 3540 }, { "loss": 0.4628, "grad_norm": 0.9246307611465454, "learning_rate": 0.00014278039607601136, "entropy": 0.5803060375154019, "num_tokens": 25278678.0, "mean_token_accuracy": 0.8839930236339569, "epoch": 5.68, "step": 3550 }, { "loss": 0.5044, "grad_norm": 1.0896925926208496, "learning_rate": 0.00014246785761092974, "entropy": 0.6183232348412275, "num_tokens": 25347472.0, "mean_token_accuracy": 0.8736830309033394, "epoch": 5.696, "step": 3560 }, { "loss": 0.5059, "grad_norm": 1.1501187086105347, "learning_rate": 0.0001421548122190109, "entropy": 0.607684375718236, "num_tokens": 25416791.0, "mean_token_accuracy": 0.8759103573858738, "epoch": 5.712, "step": 3570 }, { "loss": 0.4922, "grad_norm": 0.935501217842102, "learning_rate": 0.0001418412636369895, "entropy": 0.5978634035214782, "num_tokens": 25484611.0, "mean_token_accuracy": 0.8796527273952961, "epoch": 5.728, "step": 3580 }, { "loss": 0.4882, "grad_norm": 1.0129141807556152, "learning_rate": 0.0001415272156076064, "entropy": 0.5977130083367228, "num_tokens": 25553387.0, "mean_token_accuracy": 0.8780643150210381, "epoch": 5.744, "step": 3590 }, { "loss": 0.509, "grad_norm": 0.9960727095603943, "learning_rate": 0.00014121267187956445, "entropy": 0.6134402383118868, "num_tokens": 25623778.0, "mean_token_accuracy": 0.8737864255905151, "epoch": 5.76, "step": 3600 }, { "loss": 0.4578, "grad_norm": 1.0559104681015015, "learning_rate": 0.00014089763620748339, "entropy": 0.563372618984431, "num_tokens": 25693928.0, "mean_token_accuracy": 0.8870187282562256, "epoch": 5.776, "step": 3610 }, { "loss": 0.525, "grad_norm": 0.916033923625946, "learning_rate": 0.0001405821123518551, "entropy": 0.6502167999744415, "num_tokens": 25764192.0, "mean_token_accuracy": 0.8711183421313763, "epoch": 5.792, "step": 3620 }, { "loss": 0.5284, "grad_norm": 0.9909388422966003, "learning_rate": 0.0001402661040789989, "entropy": 0.6288280609995127, "num_tokens": 25838150.0, "mean_token_accuracy": 0.8703570686280727, "epoch": 5.808, "step": 3630 }, { "loss": 0.5206, "grad_norm": 1.0801376104354858, "learning_rate": 0.00013994961516101642, "entropy": 0.6285689912736416, "num_tokens": 25906619.0, "mean_token_accuracy": 0.8717172846198082, "epoch": 5.824, "step": 3640 }, { "loss": 0.5035, "grad_norm": 0.8041856288909912, "learning_rate": 0.00013963264937574653, "entropy": 0.6169568607583642, "num_tokens": 25978892.0, "mean_token_accuracy": 0.8756810665130615, "epoch": 5.84, "step": 3650 }, { "loss": 0.513, "grad_norm": 1.0533690452575684, "learning_rate": 0.0001393152105067205, "entropy": 0.6247089951299131, "num_tokens": 26051161.0, "mean_token_accuracy": 0.8725937850773334, "epoch": 5.856, "step": 3660 }, { "loss": 0.5424, "grad_norm": 1.0128294229507446, "learning_rate": 0.00013899730234311644, "entropy": 0.6458030100911856, "num_tokens": 26124861.0, "mean_token_accuracy": 0.8648265175521374, "epoch": 5.872, "step": 3670 }, { "loss": 0.5026, "grad_norm": 1.1381916999816895, "learning_rate": 0.00013867892867971458, "entropy": 0.6243725307285786, "num_tokens": 26193787.0, "mean_token_accuracy": 0.8721223786473274, "epoch": 5.888, "step": 3680 }, { "loss": 0.4958, "grad_norm": 0.9508771300315857, "learning_rate": 0.0001383600933168514, "entropy": 0.6094806173816323, "num_tokens": 26266947.0, "mean_token_accuracy": 0.8744397521018982, "epoch": 5.904, "step": 3690 }, { "loss": 0.5692, "grad_norm": 1.083992838859558, "learning_rate": 0.00013804080006037478, "entropy": 0.6682732639834285, "num_tokens": 26336615.0, "mean_token_accuracy": 0.8598686315119266, "epoch": 5.92, "step": 3700 }, { "loss": 0.5442, "grad_norm": 0.9117934107780457, "learning_rate": 0.0001377210527215982, "entropy": 0.650467436015606, "num_tokens": 26406167.0, "mean_token_accuracy": 0.8646365746855735, "epoch": 5.936, "step": 3710 }, { "loss": 0.5087, "grad_norm": 1.1013089418411255, "learning_rate": 0.0001374008551172555, "entropy": 0.6175769362598658, "num_tokens": 26476135.0, "mean_token_accuracy": 0.8744233258068561, "epoch": 5.952, "step": 3720 }, { "loss": 0.5426, "grad_norm": 1.14009690284729, "learning_rate": 0.00013708021106945514, "entropy": 0.6512466743588448, "num_tokens": 26542086.0, "mean_token_accuracy": 0.8673823356628418, "epoch": 5.968, "step": 3730 }, { "loss": 0.525, "grad_norm": 1.2493031024932861, "learning_rate": 0.00013675912440563458, "entropy": 0.6328467240557074, "num_tokens": 26611541.0, "mean_token_accuracy": 0.8700577892363072, "epoch": 5.984, "step": 3740 }, { "loss": 0.5445, "grad_norm": 0.9933606386184692, "learning_rate": 0.00013643759895851494, "entropy": 0.6508631907403469, "num_tokens": 26680332.0, "mean_token_accuracy": 0.8659440115094185, "epoch": 6.0, "step": 3750 }, { "eval_loss": 2.1597042083740234, "eval_runtime": 53.7247, "eval_samples_per_second": 4.188, "eval_steps_per_second": 4.188, "eval_entropy": 1.2236345574590894, "eval_num_tokens": 26680332.0, "eval_mean_token_accuracy": 0.619241547981898, "epoch": 6.0, "step": 3750 }, { "loss": 0.3947, "grad_norm": 1.1351908445358276, "learning_rate": 0.00013611563856605463, "entropy": 0.507540424540639, "num_tokens": 26749534.0, "mean_token_accuracy": 0.9086824953556061, "epoch": 6.016, "step": 3760 }, { "loss": 0.3236, "grad_norm": 0.8625683784484863, "learning_rate": 0.00013579324707140413, "entropy": 0.4453432267531753, "num_tokens": 26822005.0, "mean_token_accuracy": 0.9232410036027432, "epoch": 6.032, "step": 3770 }, { "loss": 0.3495, "grad_norm": 1.0252923965454102, "learning_rate": 0.00013547042832285974, "entropy": 0.4442704081535339, "num_tokens": 26894664.0, "mean_token_accuracy": 0.9189668878912925, "epoch": 6.048, "step": 3780 }, { "loss": 0.3413, "grad_norm": 0.9415987730026245, "learning_rate": 0.00013514718617381778, "entropy": 0.46286946404725315, "num_tokens": 26966852.0, "mean_token_accuracy": 0.9167165383696556, "epoch": 6.064, "step": 3790 }, { "loss": 0.3645, "grad_norm": 1.1017154455184937, "learning_rate": 0.00013482352448272868, "entropy": 0.4707667143084109, "num_tokens": 27036342.0, "mean_token_accuracy": 0.9132362395524979, "epoch": 6.08, "step": 3800 }, { "loss": 0.3265, "grad_norm": 0.7547138333320618, "learning_rate": 0.00013449944711305066, "entropy": 0.4366528897546232, "num_tokens": 27105952.0, "mean_token_accuracy": 0.920817120373249, "epoch": 6.096, "step": 3810 }, { "loss": 0.3475, "grad_norm": 0.9691864252090454, "learning_rate": 0.0001341749579332039, "entropy": 0.4646099326200783, "num_tokens": 27177000.0, "mean_token_accuracy": 0.9176120407879352, "epoch": 6.112, "step": 3820 }, { "loss": 0.3543, "grad_norm": 1.0419487953186035, "learning_rate": 0.00013385006081652424, "entropy": 0.4646885816939175, "num_tokens": 27249771.0, "mean_token_accuracy": 0.9135935790836811, "epoch": 6.128, "step": 3830 }, { "loss": 0.3191, "grad_norm": 0.8572609424591064, "learning_rate": 0.00013352475964121686, "entropy": 0.43566130194813013, "num_tokens": 27321765.0, "mean_token_accuracy": 0.9212069340050221, "epoch": 6.144, "step": 3840 }, { "loss": 0.3554, "grad_norm": 1.015175461769104, "learning_rate": 0.00013319905829031016, "entropy": 0.4676606528460979, "num_tokens": 27389576.0, "mean_token_accuracy": 0.9150214172899723, "epoch": 6.16, "step": 3850 }, { "loss": 0.3643, "grad_norm": 0.9731613993644714, "learning_rate": 0.0001328729606516093, "entropy": 0.4739726860076189, "num_tokens": 27458987.0, "mean_token_accuracy": 0.9142082013189793, "epoch": 6.176, "step": 3860 }, { "loss": 0.3793, "grad_norm": 1.135664463043213, "learning_rate": 0.00013254647061764982, "entropy": 0.4980067117139697, "num_tokens": 27525550.0, "mean_token_accuracy": 0.9076109625399112, "epoch": 6.192, "step": 3870 }, { "loss": 0.3668, "grad_norm": 0.9112149477005005, "learning_rate": 0.00013221959208565114, "entropy": 0.4712543160654604, "num_tokens": 27599750.0, "mean_token_accuracy": 0.910884364694357, "epoch": 6.208, "step": 3880 }, { "loss": 0.3215, "grad_norm": 1.0918331146240234, "learning_rate": 0.0001318923289574701, "entropy": 0.4320028844289482, "num_tokens": 27671187.0, "mean_token_accuracy": 0.9220032453536987, "epoch": 6.224, "step": 3890 }, { "loss": 0.3683, "grad_norm": 0.940580427646637, "learning_rate": 0.0001315646851395543, "entropy": 0.4778835457749665, "num_tokens": 27742081.0, "mean_token_accuracy": 0.9086015738546849, "epoch": 6.24, "step": 3900 }, { "loss": 0.3588, "grad_norm": 1.0839608907699585, "learning_rate": 0.00013123666454289566, "entropy": 0.46964468210935595, "num_tokens": 27814774.0, "mean_token_accuracy": 0.912013228237629, "epoch": 6.256, "step": 3910 }, { "loss": 0.3521, "grad_norm": 0.900403618812561, "learning_rate": 0.00013090827108298348, "entropy": 0.47151055820286275, "num_tokens": 27886541.0, "mean_token_accuracy": 0.9124030962586402, "epoch": 6.272, "step": 3920 }, { "loss": 0.3598, "grad_norm": 1.0008821487426758, "learning_rate": 0.00013057950867975785, "entropy": 0.47035413663834336, "num_tokens": 27956260.0, "mean_token_accuracy": 0.9149011373519897, "epoch": 6.288, "step": 3930 }, { "loss": 0.3607, "grad_norm": 1.121002197265625, "learning_rate": 0.00013025038125756284, "entropy": 0.4691207459196448, "num_tokens": 28026909.0, "mean_token_accuracy": 0.9134053491055966, "epoch": 6.304, "step": 3940 }, { "loss": 0.3504, "grad_norm": 1.0420325994491577, "learning_rate": 0.0001299208927450996, "entropy": 0.4478804627433419, "num_tokens": 28099146.0, "mean_token_accuracy": 0.9154062435030937, "epoch": 6.32, "step": 3950 }, { "loss": 0.3601, "grad_norm": 1.0676225423812866, "learning_rate": 0.0001295910470753797, "entropy": 0.4762770028784871, "num_tokens": 28170333.0, "mean_token_accuracy": 0.9119017936289311, "epoch": 6.336, "step": 3960 }, { "loss": 0.3282, "grad_norm": 0.9221089482307434, "learning_rate": 0.0001292608481856777, "entropy": 0.4365522609092295, "num_tokens": 28243642.0, "mean_token_accuracy": 0.9186208121478557, "epoch": 6.352, "step": 3970 }, { "loss": 0.3435, "grad_norm": 0.9660590291023254, "learning_rate": 0.0001289303000174847, "entropy": 0.448877714574337, "num_tokens": 28315700.0, "mean_token_accuracy": 0.9166012026369572, "epoch": 6.368, "step": 3980 }, { "loss": 0.3627, "grad_norm": 1.0125195980072021, "learning_rate": 0.00012859940651646098, "entropy": 0.47722272109240294, "num_tokens": 28386754.0, "mean_token_accuracy": 0.9132031716406346, "epoch": 6.384, "step": 3990 }, { "loss": 0.3332, "grad_norm": 1.094364047050476, "learning_rate": 0.0001282681716323888, "entropy": 0.44014298133552077, "num_tokens": 28458738.0, "mean_token_accuracy": 0.9186070390045643, "epoch": 6.4, "step": 4000 }, { "loss": 0.3346, "grad_norm": 0.8572192192077637, "learning_rate": 0.00012793659931912565, "entropy": 0.4415401941165328, "num_tokens": 28531963.0, "mean_token_accuracy": 0.9172732464969158, "epoch": 6.416, "step": 4010 }, { "loss": 0.3741, "grad_norm": 1.0554254055023193, "learning_rate": 0.00012760469353455665, "entropy": 0.48848841469734905, "num_tokens": 28604538.0, "mean_token_accuracy": 0.9080187804996968, "epoch": 6.432, "step": 4020 }, { "loss": 0.3607, "grad_norm": 0.9965901970863342, "learning_rate": 0.00012727245824054753, "entropy": 0.4705518776550889, "num_tokens": 28675033.0, "mean_token_accuracy": 0.9097922071814537, "epoch": 6.448, "step": 4030 }, { "loss": 0.3792, "grad_norm": 0.9006433486938477, "learning_rate": 0.00012693989740289729, "entropy": 0.4865629017353058, "num_tokens": 28745266.0, "mean_token_accuracy": 0.9091149859130383, "epoch": 6.464, "step": 4040 }, { "loss": 0.3689, "grad_norm": 0.9792384505271912, "learning_rate": 0.00012660701499129083, "entropy": 0.480984403565526, "num_tokens": 28811965.0, "mean_token_accuracy": 0.9097073063254356, "epoch": 6.48, "step": 4050 }, { "loss": 0.3889, "grad_norm": 0.9604381322860718, "learning_rate": 0.00012627381497925163, "entropy": 0.5089773237705231, "num_tokens": 28884150.0, "mean_token_accuracy": 0.903744374960661, "epoch": 6.496, "step": 4060 }, { "loss": 0.3601, "grad_norm": 1.0571720600128174, "learning_rate": 0.0001259403013440942, "entropy": 0.4644645845517516, "num_tokens": 28956413.0, "mean_token_accuracy": 0.9132918052375316, "epoch": 6.5120000000000005, "step": 4070 }, { "loss": 0.3682, "grad_norm": 1.0090655088424683, "learning_rate": 0.00012560647806687678, "entropy": 0.4941337088122964, "num_tokens": 29028849.0, "mean_token_accuracy": 0.9070493340492248, "epoch": 6.5280000000000005, "step": 4080 }, { "loss": 0.3688, "grad_norm": 1.0308855772018433, "learning_rate": 0.00012527234913235362, "entropy": 0.47687761913985016, "num_tokens": 29100192.0, "mean_token_accuracy": 0.9101373247802258, "epoch": 6.5440000000000005, "step": 4090 }, { "loss": 0.3632, "grad_norm": 0.997096598148346, "learning_rate": 0.0001249379185289276, "entropy": 0.4790775926783681, "num_tokens": 29172134.0, "mean_token_accuracy": 0.9102026455104351, "epoch": 6.5600000000000005, "step": 4100 }, { "loss": 0.3668, "grad_norm": 1.1396162509918213, "learning_rate": 0.00012460319024860248, "entropy": 0.48015540270134804, "num_tokens": 29244775.0, "mean_token_accuracy": 0.9097504615783691, "epoch": 6.576, "step": 4110 }, { "loss": 0.3919, "grad_norm": 1.0722675323486328, "learning_rate": 0.0001242681682869353, "entropy": 0.5142561631277204, "num_tokens": 29315812.0, "mean_token_accuracy": 0.9033015862107276, "epoch": 6.592, "step": 4120 }, { "loss": 0.3503, "grad_norm": 0.8198496103286743, "learning_rate": 0.00012393285664298877, "entropy": 0.46417886260896923, "num_tokens": 29387158.0, "mean_token_accuracy": 0.911801540851593, "epoch": 6.608, "step": 4130 }, { "loss": 0.4061, "grad_norm": 0.9145491123199463, "learning_rate": 0.00012359725931928333, "entropy": 0.5196443384513258, "num_tokens": 29452660.0, "mean_token_accuracy": 0.9015356115996838, "epoch": 6.624, "step": 4140 }, { "loss": 0.3675, "grad_norm": 1.0900770425796509, "learning_rate": 0.00012326138032174965, "entropy": 0.48102738671004774, "num_tokens": 29524408.0, "mean_token_accuracy": 0.9091583035886288, "epoch": 6.64, "step": 4150 }, { "loss": 0.388, "grad_norm": 1.0717415809631348, "learning_rate": 0.0001229252236596805, "entropy": 0.49053167132660747, "num_tokens": 29595260.0, "mean_token_accuracy": 0.9047121495008469, "epoch": 6.656, "step": 4160 }, { "loss": 0.3674, "grad_norm": 1.0290180444717407, "learning_rate": 0.0001225887933456832, "entropy": 0.4733533734455705, "num_tokens": 29670589.0, "mean_token_accuracy": 0.9102878101170063, "epoch": 6.672, "step": 4170 }, { "loss": 0.3529, "grad_norm": 1.0597587823867798, "learning_rate": 0.00012225209339563145, "entropy": 0.4645724995061755, "num_tokens": 29738578.0, "mean_token_accuracy": 0.9136429138481617, "epoch": 6.688, "step": 4180 }, { "loss": 0.3715, "grad_norm": 1.0108861923217773, "learning_rate": 0.00012191512782861762, "entropy": 0.4792745865881443, "num_tokens": 29810139.0, "mean_token_accuracy": 0.9078541591763496, "epoch": 6.704, "step": 4190 }, { "loss": 0.3653, "grad_norm": 0.8933523893356323, "learning_rate": 0.00012157790066690464, "entropy": 0.46566064320504663, "num_tokens": 29880753.0, "mean_token_accuracy": 0.9104361660778523, "epoch": 6.72, "step": 4200 }, { "loss": 0.3895, "grad_norm": 1.0159423351287842, "learning_rate": 0.00012124041593587798, "entropy": 0.4971291688270867, "num_tokens": 29952773.0, "mean_token_accuracy": 0.9039761230349541, "epoch": 6.736, "step": 4210 }, { "loss": 0.3908, "grad_norm": 0.8928624391555786, "learning_rate": 0.0001209026776639977, "entropy": 0.5062906684353947, "num_tokens": 30022692.0, "mean_token_accuracy": 0.9036431312561035, "epoch": 6.752, "step": 4220 }, { "loss": 0.3935, "grad_norm": 1.2116057872772217, "learning_rate": 0.0001205646898827503, "entropy": 0.5079310759902, "num_tokens": 30093459.0, "mean_token_accuracy": 0.9026992864906788, "epoch": 6.768, "step": 4230 }, { "loss": 0.3885, "grad_norm": 0.9644590616226196, "learning_rate": 0.00012022645662660054, "entropy": 0.5007971756160259, "num_tokens": 30161759.0, "mean_token_accuracy": 0.9056496627628803, "epoch": 6.784, "step": 4240 }, { "loss": 0.3593, "grad_norm": 1.2600727081298828, "learning_rate": 0.00011988798193294344, "entropy": 0.4744190940633416, "num_tokens": 30233027.0, "mean_token_accuracy": 0.9106418497860431, "epoch": 6.8, "step": 4250 }, { "loss": 0.3845, "grad_norm": 1.000135064125061, "learning_rate": 0.00011954926984205592, "entropy": 0.49182934742420914, "num_tokens": 30305539.0, "mean_token_accuracy": 0.9044167138636112, "epoch": 6.816, "step": 4260 }, { "loss": 0.4002, "grad_norm": 1.1069928407669067, "learning_rate": 0.00011921032439704867, "entropy": 0.5152767574414611, "num_tokens": 30376457.0, "mean_token_accuracy": 0.9024541571736335, "epoch": 6.832, "step": 4270 }, { "loss": 0.3762, "grad_norm": 1.0780972242355347, "learning_rate": 0.00011887114964381783, "entropy": 0.47134133633226155, "num_tokens": 30443130.0, "mean_token_accuracy": 0.9109449982643127, "epoch": 6.848, "step": 4280 }, { "loss": 0.3621, "grad_norm": 1.0384007692337036, "learning_rate": 0.00011853174963099682, "entropy": 0.47412992659956216, "num_tokens": 30512781.0, "mean_token_accuracy": 0.9104038722813129, "epoch": 6.864, "step": 4290 }, { "loss": 0.4273, "grad_norm": 0.9921269416809082, "learning_rate": 0.00011819212840990778, "entropy": 0.5268450791016221, "num_tokens": 30588107.0, "mean_token_accuracy": 0.8968553464859724, "epoch": 6.88, "step": 4300 }, { "loss": 0.3909, "grad_norm": 0.9813525676727295, "learning_rate": 0.00011785229003451348, "entropy": 0.4912795711308718, "num_tokens": 30663425.0, "mean_token_accuracy": 0.9038021437823772, "epoch": 6.896, "step": 4310 }, { "loss": 0.3894, "grad_norm": 1.0695106983184814, "learning_rate": 0.00011751223856136878, "entropy": 0.4952791491523385, "num_tokens": 30736267.0, "mean_token_accuracy": 0.9049302615225315, "epoch": 6.912, "step": 4320 }, { "loss": 0.3753, "grad_norm": 1.1270382404327393, "learning_rate": 0.00011717197804957207, "entropy": 0.48176986668258903, "num_tokens": 30809577.0, "mean_token_accuracy": 0.9078494250774384, "epoch": 6.928, "step": 4330 }, { "loss": 0.376, "grad_norm": 0.9611650705337524, "learning_rate": 0.00011683151256071725, "entropy": 0.48404958862811326, "num_tokens": 30878952.0, "mean_token_accuracy": 0.9097036592662334, "epoch": 6.944, "step": 4340 }, { "loss": 0.3972, "grad_norm": 1.1812978982925415, "learning_rate": 0.00011649084615884471, "entropy": 0.5077037469483912, "num_tokens": 30951015.0, "mean_token_accuracy": 0.9033686682581902, "epoch": 6.96, "step": 4350 }, { "loss": 0.4018, "grad_norm": 1.1753220558166504, "learning_rate": 0.00011614998291039326, "entropy": 0.5067099524661899, "num_tokens": 31022118.0, "mean_token_accuracy": 0.9013751477003098, "epoch": 6.976, "step": 4360 }, { "loss": 0.4265, "grad_norm": 1.04413640499115, "learning_rate": 0.0001158089268841513, "entropy": 0.5334932506084442, "num_tokens": 31091341.0, "mean_token_accuracy": 0.8971994817256927, "epoch": 6.992, "step": 4370 }, { "eval_loss": 2.330570697784424, "eval_runtime": 53.2203, "eval_samples_per_second": 4.228, "eval_steps_per_second": 4.228, "eval_entropy": 1.1098890497949387, "eval_num_tokens": 31127054.0, "eval_mean_token_accuracy": 0.6140021105607351, "epoch": 7.0, "step": 4375 }, { "loss": 0.3383, "grad_norm": 0.9320855736732483, "learning_rate": 0.00011546768215120847, "entropy": 0.46375819463282825, "num_tokens": 31161923.0, "mean_token_accuracy": 0.9188552089035511, "epoch": 7.008, "step": 4380 }, { "loss": 0.2887, "grad_norm": 0.9524214863777161, "learning_rate": 0.00011512625278490683, "entropy": 0.3875926799140871, "num_tokens": 31234135.0, "mean_token_accuracy": 0.9336316235363483, "epoch": 7.024, "step": 4390 }, { "loss": 0.2824, "grad_norm": 0.8379424214363098, "learning_rate": 0.00011478464286079245, "entropy": 0.4003670670092106, "num_tokens": 31304319.0, "mean_token_accuracy": 0.9355235919356346, "epoch": 7.04, "step": 4400 }, { "loss": 0.2456, "grad_norm": 0.9706165194511414, "learning_rate": 0.00011444285645656665, "entropy": 0.3429591671563685, "num_tokens": 31376918.0, "mean_token_accuracy": 0.9432424649596214, "epoch": 7.056, "step": 4410 }, { "loss": 0.2548, "grad_norm": 0.7785767316818237, "learning_rate": 0.00011410089765203724, "entropy": 0.36323242392390964, "num_tokens": 31448396.0, "mean_token_accuracy": 0.9406179443001748, "epoch": 7.072, "step": 4420 }, { "loss": 0.3045, "grad_norm": 1.095278024673462, "learning_rate": 0.00011375877052907013, "entropy": 0.40881806388497355, "num_tokens": 31521035.0, "mean_token_accuracy": 0.9299851886928081, "epoch": 7.088, "step": 4430 }, { "loss": 0.2408, "grad_norm": 0.9054039120674133, "learning_rate": 0.00011341647917154024, "entropy": 0.3513195881620049, "num_tokens": 31593524.0, "mean_token_accuracy": 0.943135941773653, "epoch": 7.104, "step": 4440 }, { "loss": 0.2629, "grad_norm": 0.9614740610122681, "learning_rate": 0.00011307402766528293, "entropy": 0.3655347550287843, "num_tokens": 31666003.0, "mean_token_accuracy": 0.9381780624389648, "epoch": 7.12, "step": 4450 }, { "loss": 0.2536, "grad_norm": 0.7834696769714355, "learning_rate": 0.00011273142009804522, "entropy": 0.3473891716450453, "num_tokens": 31735400.0, "mean_token_accuracy": 0.940393590927124, "epoch": 7.136, "step": 4460 }, { "loss": 0.242, "grad_norm": 1.0196019411087036, "learning_rate": 0.00011238866055943706, "entropy": 0.3489194389432669, "num_tokens": 31805891.0, "mean_token_accuracy": 0.9433179296553135, "epoch": 7.152, "step": 4470 }, { "loss": 0.266, "grad_norm": 0.7691382169723511, "learning_rate": 0.00011204575314088233, "entropy": 0.3676401344127953, "num_tokens": 31875990.0, "mean_token_accuracy": 0.938730213046074, "epoch": 7.168, "step": 4480 }, { "loss": 0.2913, "grad_norm": 1.0972436666488647, "learning_rate": 0.00011170270193557016, "entropy": 0.401006649620831, "num_tokens": 31948104.0, "mean_token_accuracy": 0.9308591552078724, "epoch": 7.184, "step": 4490 }, { "loss": 0.271, "grad_norm": 0.928560733795166, "learning_rate": 0.00011135951103840603, "entropy": 0.37095291148871185, "num_tokens": 32020448.0, "mean_token_accuracy": 0.9377502493560315, "epoch": 7.2, "step": 4500 }, { "loss": 0.2423, "grad_norm": 0.8929839134216309, "learning_rate": 0.00011101618454596287, "entropy": 0.34759438401088116, "num_tokens": 32091204.0, "mean_token_accuracy": 0.9434906929731369, "epoch": 7.216, "step": 4510 }, { "loss": 0.2803, "grad_norm": 0.908519983291626, "learning_rate": 0.00011067272655643222, "entropy": 0.386752575263381, "num_tokens": 32159957.0, "mean_token_accuracy": 0.9348809912800788, "epoch": 7.232, "step": 4520 }, { "loss": 0.2498, "grad_norm": 0.9027432203292847, "learning_rate": 0.00011032914116957513, "entropy": 0.3536661508493125, "num_tokens": 32232626.0, "mean_token_accuracy": 0.941464664787054, "epoch": 7.248, "step": 4530 }, { "loss": 0.2451, "grad_norm": 0.9318743944168091, "learning_rate": 0.00010998543248667352, "entropy": 0.34123657466843726, "num_tokens": 32306366.0, "mean_token_accuracy": 0.9429686009883881, "epoch": 7.264, "step": 4540 }, { "loss": 0.2556, "grad_norm": 0.9799351692199707, "learning_rate": 0.00010964160461048096, "entropy": 0.35888168662786485, "num_tokens": 32378150.0, "mean_token_accuracy": 0.9399545796215534, "epoch": 7.28, "step": 4550 }, { "loss": 0.2834, "grad_norm": 1.1736223697662354, "learning_rate": 0.00010929766164517384, "entropy": 0.39516890943050387, "num_tokens": 32449514.0, "mean_token_accuracy": 0.9332543216645718, "epoch": 7.296, "step": 4560 }, { "loss": 0.2699, "grad_norm": 0.9763228297233582, "learning_rate": 0.0001089536076963023, "entropy": 0.3721382154151797, "num_tokens": 32521108.0, "mean_token_accuracy": 0.9355851218104363, "epoch": 7.312, "step": 4570 }, { "loss": 0.278, "grad_norm": 0.9528865218162537, "learning_rate": 0.00010860944687074129, "entropy": 0.3834563828073442, "num_tokens": 32591443.0, "mean_token_accuracy": 0.9349353685975075, "epoch": 7.328, "step": 4580 }, { "loss": 0.282, "grad_norm": 1.3110307455062866, "learning_rate": 0.00010826518327664148, "entropy": 0.38018117249011996, "num_tokens": 32662166.0, "mean_token_accuracy": 0.9353329785168171, "epoch": 7.344, "step": 4590 }, { "loss": 0.2739, "grad_norm": 1.1898859739303589, "learning_rate": 0.0001079208210233803, "entropy": 0.3840799957513809, "num_tokens": 32734120.0, "mean_token_accuracy": 0.9349162131547928, "epoch": 7.36, "step": 4600 }, { "loss": 0.2811, "grad_norm": 1.1253063678741455, "learning_rate": 0.00010757636422151287, "entropy": 0.38191785793751476, "num_tokens": 32806839.0, "mean_token_accuracy": 0.9348918169736862, "epoch": 7.376, "step": 4610 }, { "loss": 0.2691, "grad_norm": 1.0744221210479736, "learning_rate": 0.00010723181698272282, "entropy": 0.3794839221984148, "num_tokens": 32879244.0, "mean_token_accuracy": 0.9349598817527294, "epoch": 7.392, "step": 4620 }, { "loss": 0.2628, "grad_norm": 0.9702807068824768, "learning_rate": 0.00010688718341977336, "entropy": 0.3710989004932344, "num_tokens": 32951641.0, "mean_token_accuracy": 0.9375138603150844, "epoch": 7.408, "step": 4630 }, { "loss": 0.2727, "grad_norm": 1.0916621685028076, "learning_rate": 0.00010654246764645812, "entropy": 0.374803014844656, "num_tokens": 33021336.0, "mean_token_accuracy": 0.9344482824206353, "epoch": 7.424, "step": 4640 }, { "loss": 0.2761, "grad_norm": 1.200566291809082, "learning_rate": 0.00010619767377755203, "entropy": 0.38532257787883284, "num_tokens": 33092079.0, "mean_token_accuracy": 0.9347227938473225, "epoch": 7.44, "step": 4650 }, { "loss": 0.2966, "grad_norm": 1.0548347234725952, "learning_rate": 0.00010585280592876233, "entropy": 0.401569415256381, "num_tokens": 33165386.0, "mean_token_accuracy": 0.9307530455291271, "epoch": 7.456, "step": 4660 }, { "loss": 0.2823, "grad_norm": 0.9163881540298462, "learning_rate": 0.00010550786821667918, "entropy": 0.3958779005333781, "num_tokens": 33235582.0, "mean_token_accuracy": 0.9341799102723598, "epoch": 7.4719999999999995, "step": 4670 }, { "loss": 0.2622, "grad_norm": 0.7987439632415771, "learning_rate": 0.00010516286475872672, "entropy": 0.3667552155442536, "num_tokens": 33308026.0, "mean_token_accuracy": 0.9369193151593208, "epoch": 7.4879999999999995, "step": 4680 }, { "loss": 0.2693, "grad_norm": 1.0379583835601807, "learning_rate": 0.000104817799673114, "entropy": 0.37949036844074724, "num_tokens": 33379941.0, "mean_token_accuracy": 0.9354175627231598, "epoch": 7.504, "step": 4690 }, { "loss": 0.2837, "grad_norm": 1.1165646314620972, "learning_rate": 0.00010447267707878552, "entropy": 0.38817160604521633, "num_tokens": 33449413.0, "mean_token_accuracy": 0.9332479029893875, "epoch": 7.52, "step": 4700 }, { "loss": 0.2838, "grad_norm": 1.4692283868789673, "learning_rate": 0.00010412750109537243, "entropy": 0.39286461789160965, "num_tokens": 33519038.0, "mean_token_accuracy": 0.9335977986454964, "epoch": 7.536, "step": 4710 }, { "loss": 0.2948, "grad_norm": 1.0642696619033813, "learning_rate": 0.000103782275843143, "entropy": 0.40992468390613795, "num_tokens": 33588310.0, "mean_token_accuracy": 0.9305080980062485, "epoch": 7.552, "step": 4720 }, { "loss": 0.3017, "grad_norm": 0.7679217457771301, "learning_rate": 0.00010343700544295374, "entropy": 0.41145033929497005, "num_tokens": 33661679.0, "mean_token_accuracy": 0.9287241578102112, "epoch": 7.568, "step": 4730 }, { "loss": 0.277, "grad_norm": 0.8832826018333435, "learning_rate": 0.00010309169401619998, "entropy": 0.3885214670561254, "num_tokens": 33732036.0, "mean_token_accuracy": 0.9333964847028255, "epoch": 7.584, "step": 4740 }, { "loss": 0.3261, "grad_norm": 0.9321479797363281, "learning_rate": 0.00010274634568476687, "entropy": 0.43886260148137807, "num_tokens": 33802606.0, "mean_token_accuracy": 0.9225052051246166, "epoch": 7.6, "step": 4750 }, { "loss": 0.2899, "grad_norm": 1.0162326097488403, "learning_rate": 0.00010240096457097997, "entropy": 0.40483922939747574, "num_tokens": 33869791.0, "mean_token_accuracy": 0.9298799574375153, "epoch": 7.616, "step": 4760 }, { "loss": 0.3026, "grad_norm": 1.0396180152893066, "learning_rate": 0.00010205555479755624, "entropy": 0.40616279244422915, "num_tokens": 33943491.0, "mean_token_accuracy": 0.9273484729230403, "epoch": 7.632, "step": 4770 }, { "loss": 0.3097, "grad_norm": 1.2665165662765503, "learning_rate": 0.00010171012048755472, "entropy": 0.4197121734730899, "num_tokens": 34014309.0, "mean_token_accuracy": 0.9244455620646477, "epoch": 7.648, "step": 4780 }, { "loss": 0.2712, "grad_norm": 0.9822998046875, "learning_rate": 0.00010136466576432732, "entropy": 0.37735338560305537, "num_tokens": 34084780.0, "mean_token_accuracy": 0.9350693315267563, "epoch": 7.664, "step": 4790 }, { "loss": 0.3012, "grad_norm": 1.0876035690307617, "learning_rate": 0.00010101919475146966, "entropy": 0.41120583154261114, "num_tokens": 34154565.0, "mean_token_accuracy": 0.9287385255098343, "epoch": 7.68, "step": 4800 }, { "loss": 0.2901, "grad_norm": 1.0259525775909424, "learning_rate": 0.00010067371157277172, "entropy": 0.3989484773948789, "num_tokens": 34228476.0, "mean_token_accuracy": 0.9293934546411038, "epoch": 7.696, "step": 4810 }, { "loss": 0.2879, "grad_norm": 0.9387515783309937, "learning_rate": 0.00010032822035216875, "entropy": 0.3991957766003907, "num_tokens": 34295993.0, "mean_token_accuracy": 0.9303789146244525, "epoch": 7.712, "step": 4820 }, { "loss": 0.2859, "grad_norm": 1.2203270196914673, "learning_rate": 9.998272521369204e-05, "entropy": 0.3818904057145119, "num_tokens": 34366911.0, "mean_token_accuracy": 0.9327978789806366, "epoch": 7.728, "step": 4830 }, { "loss": 0.2782, "grad_norm": 0.9826841354370117, "learning_rate": 9.963723028141958e-05, "entropy": 0.3865166328847408, "num_tokens": 34437467.0, "mean_token_accuracy": 0.9339363284409046, "epoch": 7.744, "step": 4840 }, { "loss": 0.3052, "grad_norm": 1.218998670578003, "learning_rate": 9.929173967942693e-05, "entropy": 0.41630224157124757, "num_tokens": 34507311.0, "mean_token_accuracy": 0.927312783151865, "epoch": 7.76, "step": 4850 }, { "loss": 0.2854, "grad_norm": 1.163989782333374, "learning_rate": 9.894625753173796e-05, "entropy": 0.3943789599463344, "num_tokens": 34581921.0, "mean_token_accuracy": 0.930002325028181, "epoch": 7.776, "step": 4860 }, { "loss": 0.2765, "grad_norm": 1.0578101873397827, "learning_rate": 9.860078796227556e-05, "entropy": 0.37997521720826627, "num_tokens": 34654061.0, "mean_token_accuracy": 0.9343456543982029, "epoch": 7.792, "step": 4870 }, { "loss": 0.3248, "grad_norm": 1.0614850521087646, "learning_rate": 9.825533509481259e-05, "entropy": 0.42891493774950507, "num_tokens": 34726852.0, "mean_token_accuracy": 0.9228919960558415, "epoch": 7.808, "step": 4880 }, { "loss": 0.2735, "grad_norm": 1.0059537887573242, "learning_rate": 9.790990305292247e-05, "entropy": 0.3826503403484821, "num_tokens": 34797045.0, "mean_token_accuracy": 0.9314724989235401, "epoch": 7.824, "step": 4890 }, { "loss": 0.2968, "grad_norm": 1.0807831287384033, "learning_rate": 9.756449595993004e-05, "entropy": 0.40056526130065323, "num_tokens": 34866304.0, "mean_token_accuracy": 0.9300453126430511, "epoch": 7.84, "step": 4900 }, { "loss": 0.2938, "grad_norm": 1.0693719387054443, "learning_rate": 9.721911793886232e-05, "entropy": 0.4011099323630333, "num_tokens": 34935086.0, "mean_token_accuracy": 0.9288134753704071, "epoch": 7.856, "step": 4910 }, { "loss": 0.3169, "grad_norm": 0.898857057094574, "learning_rate": 9.687377311239938e-05, "entropy": 0.43236136697232724, "num_tokens": 35000912.0, "mean_token_accuracy": 0.9236554890871048, "epoch": 7.872, "step": 4920 }, { "loss": 0.3008, "grad_norm": 1.1391286849975586, "learning_rate": 9.652846560282494e-05, "entropy": 0.41000478006899355, "num_tokens": 35072646.0, "mean_token_accuracy": 0.9280422836542129, "epoch": 7.888, "step": 4930 }, { "loss": 0.2915, "grad_norm": 1.0925270318984985, "learning_rate": 9.618319953197738e-05, "entropy": 0.3952603991143405, "num_tokens": 35143999.0, "mean_token_accuracy": 0.9309671342372894, "epoch": 7.904, "step": 4940 }, { "loss": 0.2764, "grad_norm": 0.9488377571105957, "learning_rate": 9.583797902120036e-05, "entropy": 0.38776318952441213, "num_tokens": 35215248.0, "mean_token_accuracy": 0.9321672603487968, "epoch": 7.92, "step": 4950 }, { "loss": 0.2898, "grad_norm": 1.1328189373016357, "learning_rate": 9.549280819129377e-05, "entropy": 0.39868045747280123, "num_tokens": 35285667.0, "mean_token_accuracy": 0.9301923528313637, "epoch": 7.936, "step": 4960 }, { "loss": 0.2755, "grad_norm": 1.136831521987915, "learning_rate": 9.514769116246445e-05, "entropy": 0.36694235671311615, "num_tokens": 35360550.0, "mean_token_accuracy": 0.9344906225800514, "epoch": 7.952, "step": 4970 }, { "loss": 0.2958, "grad_norm": 0.949995756149292, "learning_rate": 9.480263205427696e-05, "entropy": 0.40033630281686783, "num_tokens": 35431744.0, "mean_token_accuracy": 0.9272466823458672, "epoch": 7.968, "step": 4980 }, { "loss": 0.2982, "grad_norm": 0.9519734978675842, "learning_rate": 9.445763498560463e-05, "entropy": 0.41145028462633493, "num_tokens": 35503344.0, "mean_token_accuracy": 0.9293198801577092, "epoch": 7.984, "step": 4990 }, { "loss": 0.2684, "grad_norm": 1.056819200515747, "learning_rate": 9.411270407458009e-05, "entropy": 0.36817682478576896, "num_tokens": 35573776.0, "mean_token_accuracy": 0.9344622194766998, "epoch": 8.0, "step": 5000 }, { "eval_loss": 2.5042002201080322, "eval_runtime": 53.2139, "eval_samples_per_second": 4.228, "eval_steps_per_second": 4.228, "eval_entropy": 1.0041650705867344, "eval_num_tokens": 35573776.0, "eval_mean_token_accuracy": 0.610490095085568, "epoch": 8.0, "step": 5000 }, { "loss": 0.1923, "grad_norm": 0.9085320830345154, "learning_rate": 9.376784343854637e-05, "entropy": 0.2981388337910175, "num_tokens": 35646992.0, "mean_token_accuracy": 0.9583885133266449, "epoch": 8.016, "step": 5010 }, { "loss": 0.1977, "grad_norm": 0.8003173470497131, "learning_rate": 9.342305719400755e-05, "entropy": 0.2993398167192936, "num_tokens": 35714064.0, "mean_token_accuracy": 0.9557507887482644, "epoch": 8.032, "step": 5020 }, { "loss": 0.2044, "grad_norm": 0.9359211921691895, "learning_rate": 9.307834945657984e-05, "entropy": 0.3068748265504837, "num_tokens": 35782300.0, "mean_token_accuracy": 0.9551200956106186, "epoch": 8.048, "step": 5030 }, { "loss": 0.1912, "grad_norm": 0.9900560975074768, "learning_rate": 9.273372434094219e-05, "entropy": 0.28805918116122486, "num_tokens": 35855469.0, "mean_token_accuracy": 0.9561170756816864, "epoch": 8.064, "step": 5040 }, { "loss": 0.192, "grad_norm": 0.9500458240509033, "learning_rate": 9.238918596078746e-05, "entropy": 0.29490622635930774, "num_tokens": 35926727.0, "mean_token_accuracy": 0.957545743137598, "epoch": 8.08, "step": 5050 }, { "loss": 0.2276, "grad_norm": 1.0550724267959595, "learning_rate": 9.204473842877313e-05, "entropy": 0.3265444653108716, "num_tokens": 35994931.0, "mean_token_accuracy": 0.951187041401863, "epoch": 8.096, "step": 5060 }, { "loss": 0.1777, "grad_norm": 0.8349865674972534, "learning_rate": 9.170038585647219e-05, "entropy": 0.2711963947862387, "num_tokens": 36069597.0, "mean_token_accuracy": 0.9602036297321319, "epoch": 8.112, "step": 5070 }, { "loss": 0.2112, "grad_norm": 1.1272614002227783, "learning_rate": 9.135613235432413e-05, "entropy": 0.31780841331928966, "num_tokens": 36138637.0, "mean_token_accuracy": 0.9510883264243603, "epoch": 8.128, "step": 5080 }, { "loss": 0.1893, "grad_norm": 1.0895860195159912, "learning_rate": 9.101198203158595e-05, "entropy": 0.28185115857049825, "num_tokens": 36213893.0, "mean_token_accuracy": 0.9576184302568436, "epoch": 8.144, "step": 5090 }, { "loss": 0.1942, "grad_norm": 0.7432615160942078, "learning_rate": 9.066793899628293e-05, "entropy": 0.2891276630572975, "num_tokens": 36287367.0, "mean_token_accuracy": 0.956569205224514, "epoch": 8.16, "step": 5100 }, { "loss": 0.1875, "grad_norm": 0.831279456615448, "learning_rate": 9.03240073551598e-05, "entropy": 0.28564814319834114, "num_tokens": 36359227.0, "mean_token_accuracy": 0.9579950869083405, "epoch": 8.176, "step": 5110 }, { "loss": 0.1997, "grad_norm": 0.8252593874931335, "learning_rate": 8.998019121363148e-05, "entropy": 0.2945885207504034, "num_tokens": 36432178.0, "mean_token_accuracy": 0.9543778121471405, "epoch": 8.192, "step": 5120 }, { "loss": 0.2205, "grad_norm": 0.9013761878013611, "learning_rate": 8.963649467573433e-05, "entropy": 0.3207083295099437, "num_tokens": 36501377.0, "mean_token_accuracy": 0.9505049519240856, "epoch": 8.208, "step": 5130 }, { "loss": 0.1867, "grad_norm": 0.8803057670593262, "learning_rate": 8.929292184407692e-05, "entropy": 0.27388147534802554, "num_tokens": 36574310.0, "mean_token_accuracy": 0.9577967099845409, "epoch": 8.224, "step": 5140 }, { "loss": 0.2151, "grad_norm": 0.9320727586746216, "learning_rate": 8.894947681979127e-05, "entropy": 0.32154301581904293, "num_tokens": 36644972.0, "mean_token_accuracy": 0.9509343810379505, "epoch": 8.24, "step": 5150 }, { "loss": 0.1973, "grad_norm": 1.028472900390625, "learning_rate": 8.860616370248374e-05, "entropy": 0.2982589267194271, "num_tokens": 36716075.0, "mean_token_accuracy": 0.9557504668831825, "epoch": 8.256, "step": 5160 }, { "loss": 0.189, "grad_norm": 0.8370059728622437, "learning_rate": 8.826298659018615e-05, "entropy": 0.28127606231719254, "num_tokens": 36789071.0, "mean_token_accuracy": 0.9562889978289604, "epoch": 8.272, "step": 5170 }, { "loss": 0.2224, "grad_norm": 0.9848848581314087, "learning_rate": 8.791994957930692e-05, "entropy": 0.33124771546572446, "num_tokens": 36857318.0, "mean_token_accuracy": 0.9496401883661747, "epoch": 8.288, "step": 5180 }, { "loss": 0.2029, "grad_norm": 1.034847378730774, "learning_rate": 8.757705676458207e-05, "entropy": 0.2941365322098136, "num_tokens": 36933013.0, "mean_token_accuracy": 0.9555138386785984, "epoch": 8.304, "step": 5190 }, { "loss": 0.2236, "grad_norm": 0.9423278570175171, "learning_rate": 8.723431223902642e-05, "entropy": 0.3222027115523815, "num_tokens": 37005687.0, "mean_token_accuracy": 0.9499928623437881, "epoch": 8.32, "step": 5200 }, { "loss": 0.2074, "grad_norm": 0.9669883251190186, "learning_rate": 8.689172009388466e-05, "entropy": 0.30945723317563534, "num_tokens": 37077900.0, "mean_token_accuracy": 0.9514043360948563, "epoch": 8.336, "step": 5210 }, { "loss": 0.1924, "grad_norm": 1.0017062425613403, "learning_rate": 8.654928441858264e-05, "entropy": 0.2860132520087063, "num_tokens": 37147902.0, "mean_token_accuracy": 0.9562473930418491, "epoch": 8.352, "step": 5220 }, { "loss": 0.2081, "grad_norm": 0.9740207195281982, "learning_rate": 8.620700930067837e-05, "entropy": 0.30963305858895185, "num_tokens": 37221509.0, "mean_token_accuracy": 0.9523635923862457, "epoch": 8.368, "step": 5230 }, { "loss": 0.2228, "grad_norm": 1.1588295698165894, "learning_rate": 8.586489882581345e-05, "entropy": 0.3244482036679983, "num_tokens": 37292896.0, "mean_token_accuracy": 0.949735163897276, "epoch": 8.384, "step": 5240 }, { "loss": 0.217, "grad_norm": 0.8069917559623718, "learning_rate": 8.552295707766405e-05, "entropy": 0.31814685724675656, "num_tokens": 37362892.0, "mean_token_accuracy": 0.9523125126957893, "epoch": 8.4, "step": 5250 }, { "loss": 0.2262, "grad_norm": 0.9578646421432495, "learning_rate": 8.518118813789237e-05, "entropy": 0.3224184068851173, "num_tokens": 37430160.0, "mean_token_accuracy": 0.949615902453661, "epoch": 8.416, "step": 5260 }, { "loss": 0.2113, "grad_norm": 1.0544745922088623, "learning_rate": 8.483959608609789e-05, "entropy": 0.3111849991604686, "num_tokens": 37502342.0, "mean_token_accuracy": 0.952064162492752, "epoch": 8.432, "step": 5270 }, { "loss": 0.2238, "grad_norm": 0.8992865681648254, "learning_rate": 8.449818499976855e-05, "entropy": 0.3239798369817436, "num_tokens": 37575319.0, "mean_token_accuracy": 0.9480805143713951, "epoch": 8.448, "step": 5280 }, { "loss": 0.2213, "grad_norm": 1.0319328308105469, "learning_rate": 8.415695895423217e-05, "entropy": 0.3235674677416682, "num_tokens": 37644956.0, "mean_token_accuracy": 0.9499170504510402, "epoch": 8.464, "step": 5290 }, { "loss": 0.2159, "grad_norm": 0.8126994967460632, "learning_rate": 8.381592202260784e-05, "entropy": 0.31322045912966134, "num_tokens": 37717858.0, "mean_token_accuracy": 0.9505295008420944, "epoch": 8.48, "step": 5300 }, { "loss": 0.2389, "grad_norm": 0.9714730978012085, "learning_rate": 8.347507827575718e-05, "entropy": 0.31535128662362694, "num_tokens": 37789461.0, "mean_token_accuracy": 0.9478254236280919, "epoch": 8.496, "step": 5310 }, { "loss": 0.209, "grad_norm": 0.8814937472343445, "learning_rate": 8.313443178223588e-05, "entropy": 0.30966101977974175, "num_tokens": 37862196.0, "mean_token_accuracy": 0.9509022668004036, "epoch": 8.512, "step": 5320 }, { "loss": 0.2012, "grad_norm": 0.8697338104248047, "learning_rate": 8.2793986608245e-05, "entropy": 0.3039259472861886, "num_tokens": 37934239.0, "mean_token_accuracy": 0.9527431644499302, "epoch": 8.528, "step": 5330 }, { "loss": 0.2346, "grad_norm": 1.0329784154891968, "learning_rate": 8.24537468175826e-05, "entropy": 0.336016805563122, "num_tokens": 38006441.0, "mean_token_accuracy": 0.9450462855398655, "epoch": 8.544, "step": 5340 }, { "loss": 0.2196, "grad_norm": 0.979934811592102, "learning_rate": 8.211371647159508e-05, "entropy": 0.32440577950328586, "num_tokens": 38073385.0, "mean_token_accuracy": 0.9495778903365135, "epoch": 8.56, "step": 5350 }, { "loss": 0.1962, "grad_norm": 0.9665189981460571, "learning_rate": 8.177389962912874e-05, "entropy": 0.2927178157493472, "num_tokens": 38144840.0, "mean_token_accuracy": 0.9539746806025505, "epoch": 8.576, "step": 5360 }, { "loss": 0.2137, "grad_norm": 0.8954610824584961, "learning_rate": 8.143430034648139e-05, "entropy": 0.31386541817337277, "num_tokens": 38215823.0, "mean_token_accuracy": 0.950902970135212, "epoch": 8.592, "step": 5370 }, { "loss": 0.2321, "grad_norm": 1.091084361076355, "learning_rate": 8.109492267735385e-05, "entropy": 0.33996669538319113, "num_tokens": 38283538.0, "mean_token_accuracy": 0.9460224941372871, "epoch": 8.608, "step": 5380 }, { "loss": 0.2063, "grad_norm": 0.8174199461936951, "learning_rate": 8.07557706728017e-05, "entropy": 0.3058034451678395, "num_tokens": 38356372.0, "mean_token_accuracy": 0.9534699149429798, "epoch": 8.624, "step": 5390 }, { "loss": 0.2044, "grad_norm": 0.7819794416427612, "learning_rate": 8.041684838118664e-05, "entropy": 0.3054806312546134, "num_tokens": 38427910.0, "mean_token_accuracy": 0.9532774895429611, "epoch": 8.64, "step": 5400 }, { "loss": 0.236, "grad_norm": 1.099134922027588, "learning_rate": 8.007815984812858e-05, "entropy": 0.33885709652677176, "num_tokens": 38498173.0, "mean_token_accuracy": 0.9452091291546821, "epoch": 8.656, "step": 5410 }, { "loss": 0.1984, "grad_norm": 0.8229247331619263, "learning_rate": 7.973970911645691e-05, "entropy": 0.30156353693455457, "num_tokens": 38570629.0, "mean_token_accuracy": 0.9550521992146969, "epoch": 8.672, "step": 5420 }, { "loss": 0.2308, "grad_norm": 1.129019021987915, "learning_rate": 7.94015002261626e-05, "entropy": 0.3288916707038879, "num_tokens": 38642879.0, "mean_token_accuracy": 0.9468361325562, "epoch": 8.688, "step": 5430 }, { "loss": 0.2171, "grad_norm": 0.841923177242279, "learning_rate": 7.906353721434976e-05, "entropy": 0.3163255458697677, "num_tokens": 38713048.0, "mean_token_accuracy": 0.9497400291264058, "epoch": 8.704, "step": 5440 }, { "loss": 0.2285, "grad_norm": 1.1277023553848267, "learning_rate": 7.87258241151875e-05, "entropy": 0.34124059332534673, "num_tokens": 38781774.0, "mean_token_accuracy": 0.947848080098629, "epoch": 8.72, "step": 5450 }, { "loss": 0.2192, "grad_norm": 1.0023715496063232, "learning_rate": 7.838836495986189e-05, "entropy": 0.32045974950306116, "num_tokens": 38855087.0, "mean_token_accuracy": 0.9500632904469967, "epoch": 8.736, "step": 5460 }, { "loss": 0.218, "grad_norm": 1.137165904045105, "learning_rate": 7.805116377652759e-05, "entropy": 0.3166652027517557, "num_tokens": 38925079.0, "mean_token_accuracy": 0.9495291888713837, "epoch": 8.752, "step": 5470 }, { "loss": 0.2223, "grad_norm": 1.122688889503479, "learning_rate": 7.77142245902601e-05, "entropy": 0.3226370580494404, "num_tokens": 38992376.0, "mean_token_accuracy": 0.9500689409673214, "epoch": 8.768, "step": 5480 }, { "loss": 0.2093, "grad_norm": 0.9929949045181274, "learning_rate": 7.737755142300734e-05, "entropy": 0.2983867183327675, "num_tokens": 39065736.0, "mean_token_accuracy": 0.9512861520051956, "epoch": 8.784, "step": 5490 }, { "loss": 0.2159, "grad_norm": 0.8959084153175354, "learning_rate": 7.704114829354205e-05, "entropy": 0.31323315892368553, "num_tokens": 39134579.0, "mean_token_accuracy": 0.9508936807513237, "epoch": 8.8, "step": 5500 }, { "loss": 0.2235, "grad_norm": 1.0284905433654785, "learning_rate": 7.670501921741345e-05, "entropy": 0.3290384978055954, "num_tokens": 39204620.0, "mean_token_accuracy": 0.9467869654297829, "epoch": 8.816, "step": 5510 }, { "loss": 0.222, "grad_norm": 1.0128976106643677, "learning_rate": 7.636916820689947e-05, "entropy": 0.3234254792332649, "num_tokens": 39276150.0, "mean_token_accuracy": 0.9491306327283382, "epoch": 8.832, "step": 5520 }, { "loss": 0.2489, "grad_norm": 1.1969118118286133, "learning_rate": 7.603359927095898e-05, "entropy": 0.3530355732887983, "num_tokens": 39346056.0, "mean_token_accuracy": 0.9432303093373775, "epoch": 8.848, "step": 5530 }, { "loss": 0.2228, "grad_norm": 1.0612092018127441, "learning_rate": 7.569831641518361e-05, "entropy": 0.31966694379225374, "num_tokens": 39419131.0, "mean_token_accuracy": 0.9486171402037143, "epoch": 8.864, "step": 5540 }, { "loss": 0.1972, "grad_norm": 1.2570899724960327, "learning_rate": 7.536332364175031e-05, "entropy": 0.2964139534160495, "num_tokens": 39491030.0, "mean_token_accuracy": 0.9536322988569736, "epoch": 8.88, "step": 5550 }, { "loss": 0.2085, "grad_norm": 0.8758759498596191, "learning_rate": 7.502862494937328e-05, "entropy": 0.3173206441104412, "num_tokens": 39561528.0, "mean_token_accuracy": 0.9507977932691574, "epoch": 8.896, "step": 5560 }, { "loss": 0.2403, "grad_norm": 0.8718555569648743, "learning_rate": 7.469422433325641e-05, "entropy": 0.33940853346139194, "num_tokens": 39631942.0, "mean_token_accuracy": 0.9450227491557598, "epoch": 8.912, "step": 5570 }, { "loss": 0.2284, "grad_norm": 1.1153777837753296, "learning_rate": 7.436012578504555e-05, "entropy": 0.331297882180661, "num_tokens": 39703647.0, "mean_token_accuracy": 0.947604501992464, "epoch": 8.928, "step": 5580 }, { "loss": 0.2276, "grad_norm": 1.012484073638916, "learning_rate": 7.402633329278077e-05, "entropy": 0.32911767391487956, "num_tokens": 39773595.0, "mean_token_accuracy": 0.9476138241589069, "epoch": 8.943999999999999, "step": 5590 }, { "loss": 0.2328, "grad_norm": 0.9098719358444214, "learning_rate": 7.369285084084897e-05, "entropy": 0.3285402927547693, "num_tokens": 39845327.0, "mean_token_accuracy": 0.9472571425139904, "epoch": 8.96, "step": 5600 }, { "loss": 0.2266, "grad_norm": 0.7948988080024719, "learning_rate": 7.335968240993605e-05, "entropy": 0.32143015316687523, "num_tokens": 39918045.0, "mean_token_accuracy": 0.9491382904350758, "epoch": 8.975999999999999, "step": 5610 }, { "loss": 0.2171, "grad_norm": 1.0216199159622192, "learning_rate": 7.302683197697965e-05, "entropy": 0.3146923606283963, "num_tokens": 39985930.0, "mean_token_accuracy": 0.9506479643285275, "epoch": 8.992, "step": 5620 }, { "eval_loss": 2.6925711631774902, "eval_runtime": 53.2464, "eval_samples_per_second": 4.226, "eval_steps_per_second": 4.226, "eval_entropy": 0.9193782540162404, "eval_num_tokens": 40020498.0, "eval_mean_token_accuracy": 0.6071438705921173, "epoch": 9.0, "step": 5625 }, { "loss": 0.1817, "grad_norm": 0.7968447208404541, "learning_rate": 7.269430351512143e-05, "entropy": 0.2834266349673271, "num_tokens": 40057814.0, "mean_token_accuracy": 0.9604593835771084, "epoch": 9.008, "step": 5630 }, { "loss": 0.155, "grad_norm": 0.8618186712265015, "learning_rate": 7.236210099365992e-05, "entropy": 0.24600234422832729, "num_tokens": 40126990.0, "mean_token_accuracy": 0.965995016694069, "epoch": 9.024, "step": 5640 }, { "loss": 0.1634, "grad_norm": 0.707120954990387, "learning_rate": 7.203022837800286e-05, "entropy": 0.2663976957090199, "num_tokens": 40196937.0, "mean_token_accuracy": 0.9648015633225441, "epoch": 9.04, "step": 5650 }, { "loss": 0.1521, "grad_norm": 1.0786882638931274, "learning_rate": 7.169868962962003e-05, "entropy": 0.24705568039789796, "num_tokens": 40268869.0, "mean_token_accuracy": 0.9676280461251736, "epoch": 9.056, "step": 5660 }, { "loss": 0.1481, "grad_norm": 0.6875161528587341, "learning_rate": 7.136748870599602e-05, "entropy": 0.23586258850991726, "num_tokens": 40337561.0, "mean_token_accuracy": 0.9684364266693593, "epoch": 9.072, "step": 5670 }, { "loss": 0.169, "grad_norm": 1.011060118675232, "learning_rate": 7.103662956058277e-05, "entropy": 0.25747689977288246, "num_tokens": 40406016.0, "mean_token_accuracy": 0.9634551033377647, "epoch": 9.088, "step": 5680 }, { "loss": 0.1564, "grad_norm": 0.6770326495170593, "learning_rate": 7.07061161427526e-05, "entropy": 0.24060514494776725, "num_tokens": 40480218.0, "mean_token_accuracy": 0.9677465595304966, "epoch": 9.104, "step": 5690 }, { "loss": 0.161, "grad_norm": 0.8777767419815063, "learning_rate": 7.037595239775093e-05, "entropy": 0.24993779128417373, "num_tokens": 40553187.0, "mean_token_accuracy": 0.965890035778284, "epoch": 9.12, "step": 5700 }, { "loss": 0.15, "grad_norm": 0.8308086395263672, "learning_rate": 7.004614226664925e-05, "entropy": 0.2371044062077999, "num_tokens": 40626361.0, "mean_token_accuracy": 0.9685675866901875, "epoch": 9.136, "step": 5710 }, { "loss": 0.1472, "grad_norm": 0.9495595097541809, "learning_rate": 6.971668968629812e-05, "entropy": 0.23877446223050355, "num_tokens": 40697226.0, "mean_token_accuracy": 0.9681690536439419, "epoch": 9.152, "step": 5720 }, { "loss": 0.1533, "grad_norm": 0.7855075597763062, "learning_rate": 6.938759858928e-05, "entropy": 0.23875852376222612, "num_tokens": 40771555.0, "mean_token_accuracy": 0.9679764688014985, "epoch": 9.168, "step": 5730 }, { "loss": 0.1612, "grad_norm": 0.6989420056343079, "learning_rate": 6.905887290386253e-05, "entropy": 0.25270219054073095, "num_tokens": 40845471.0, "mean_token_accuracy": 0.9651485405862331, "epoch": 9.184, "step": 5740 }, { "loss": 0.1561, "grad_norm": 0.7295164465904236, "learning_rate": 6.873051655395147e-05, "entropy": 0.24261162793263794, "num_tokens": 40917202.0, "mean_token_accuracy": 0.9677018158137798, "epoch": 9.2, "step": 5750 }, { "loss": 0.1559, "grad_norm": 0.9656840562820435, "learning_rate": 6.840253345904392e-05, "entropy": 0.24460366796702146, "num_tokens": 40990267.0, "mean_token_accuracy": 0.9666223213076591, "epoch": 9.216, "step": 5760 }, { "loss": 0.1619, "grad_norm": 0.7864900231361389, "learning_rate": 6.807492753418161e-05, "entropy": 0.25977253885939716, "num_tokens": 41060680.0, "mean_token_accuracy": 0.9650142326951027, "epoch": 9.232, "step": 5770 }, { "loss": 0.1606, "grad_norm": 0.8831656575202942, "learning_rate": 6.774770268990403e-05, "entropy": 0.24828892489895224, "num_tokens": 41130574.0, "mean_token_accuracy": 0.9662511855363846, "epoch": 9.248, "step": 5780 }, { "loss": 0.1617, "grad_norm": 0.8227595090866089, "learning_rate": 6.742086283220186e-05, "entropy": 0.25471440562978387, "num_tokens": 41202104.0, "mean_token_accuracy": 0.9642134122550488, "epoch": 9.264, "step": 5790 }, { "loss": 0.1743, "grad_norm": 0.9235018491744995, "learning_rate": 6.709441186247027e-05, "entropy": 0.25891048507764935, "num_tokens": 41272381.0, "mean_token_accuracy": 0.9634511433541775, "epoch": 9.28, "step": 5800 }, { "loss": 0.1571, "grad_norm": 0.7930293083190918, "learning_rate": 6.676835367746243e-05, "entropy": 0.23769985176622868, "num_tokens": 41344603.0, "mean_token_accuracy": 0.9672488898038865, "epoch": 9.296, "step": 5810 }, { "loss": 0.1568, "grad_norm": 0.8937450647354126, "learning_rate": 6.644269216924289e-05, "entropy": 0.24787413524463772, "num_tokens": 41414128.0, "mean_token_accuracy": 0.9667388997972012, "epoch": 9.312, "step": 5820 }, { "loss": 0.1688, "grad_norm": 0.7893115282058716, "learning_rate": 6.611743122514125e-05, "entropy": 0.25856088735163213, "num_tokens": 41484160.0, "mean_token_accuracy": 0.9632179424166679, "epoch": 9.328, "step": 5830 }, { "loss": 0.1655, "grad_norm": 0.8730454444885254, "learning_rate": 6.579257472770561e-05, "entropy": 0.25609133280813695, "num_tokens": 41555924.0, "mean_token_accuracy": 0.9649411410093307, "epoch": 9.344, "step": 5840 }, { "loss": 0.1582, "grad_norm": 0.8747796416282654, "learning_rate": 6.546812655465637e-05, "entropy": 0.2408789069391787, "num_tokens": 41627058.0, "mean_token_accuracy": 0.9653325662016868, "epoch": 9.36, "step": 5850 }, { "loss": 0.1688, "grad_norm": 0.8404150009155273, "learning_rate": 6.514409057883985e-05, "entropy": 0.2700158623978496, "num_tokens": 41698779.0, "mean_token_accuracy": 0.9623691022396088, "epoch": 9.376, "step": 5860 }, { "loss": 0.1683, "grad_norm": 0.7696067094802856, "learning_rate": 6.482047066818207e-05, "entropy": 0.25225405166856946, "num_tokens": 41768481.0, "mean_token_accuracy": 0.965115375071764, "epoch": 9.392, "step": 5870 }, { "loss": 0.1666, "grad_norm": 0.7780594229698181, "learning_rate": 6.449727068564266e-05, "entropy": 0.2548931485041976, "num_tokens": 41840657.0, "mean_token_accuracy": 0.9643183685839176, "epoch": 9.408, "step": 5880 }, { "loss": 0.1554, "grad_norm": 0.6530511379241943, "learning_rate": 6.41744944891686e-05, "entropy": 0.23985642651095987, "num_tokens": 41912287.0, "mean_token_accuracy": 0.9662373088300228, "epoch": 9.424, "step": 5890 }, { "loss": 0.1639, "grad_norm": 0.8008093237876892, "learning_rate": 6.385214593164834e-05, "entropy": 0.2573473766446114, "num_tokens": 41983327.0, "mean_token_accuracy": 0.9638742327690124, "epoch": 9.44, "step": 5900 }, { "loss": 0.1644, "grad_norm": 0.9024697542190552, "learning_rate": 6.353022886086565e-05, "entropy": 0.25696071414276955, "num_tokens": 42055915.0, "mean_token_accuracy": 0.9650357320904732, "epoch": 9.456, "step": 5910 }, { "loss": 0.1735, "grad_norm": 0.8074780702590942, "learning_rate": 6.320874711945382e-05, "entropy": 0.2696662923321128, "num_tokens": 42127037.0, "mean_token_accuracy": 0.9620686806738377, "epoch": 9.472, "step": 5920 }, { "loss": 0.1531, "grad_norm": 0.8950142860412598, "learning_rate": 6.288770454484973e-05, "entropy": 0.24099779883399605, "num_tokens": 42198153.0, "mean_token_accuracy": 0.9675011284649372, "epoch": 9.488, "step": 5930 }, { "loss": 0.1979, "grad_norm": 0.9752508401870728, "learning_rate": 6.256710496924793e-05, "entropy": 0.29840408228337767, "num_tokens": 42266295.0, "mean_token_accuracy": 0.9564948178827762, "epoch": 9.504, "step": 5940 }, { "loss": 0.1747, "grad_norm": 0.9533753991127014, "learning_rate": 6.224695221955528e-05, "entropy": 0.27019109539687636, "num_tokens": 42339812.0, "mean_token_accuracy": 0.9629211343824864, "epoch": 9.52, "step": 5950 }, { "loss": 0.1739, "grad_norm": 0.658411979675293, "learning_rate": 6.192725011734477e-05, "entropy": 0.2570402139332145, "num_tokens": 42410133.0, "mean_token_accuracy": 0.9631010890007019, "epoch": 9.536, "step": 5960 }, { "loss": 0.1725, "grad_norm": 0.7558372020721436, "learning_rate": 6.160800247881019e-05, "entropy": 0.26847462030127645, "num_tokens": 42477763.0, "mean_token_accuracy": 0.9631271466612816, "epoch": 9.552, "step": 5970 }, { "loss": 0.1732, "grad_norm": 0.7868661284446716, "learning_rate": 6.12892131147206e-05, "entropy": 0.261898516304791, "num_tokens": 42550710.0, "mean_token_accuracy": 0.9635617524385452, "epoch": 9.568, "step": 5980 }, { "loss": 0.1724, "grad_norm": 0.7022451758384705, "learning_rate": 6.097088583037467e-05, "entropy": 0.26516395015642047, "num_tokens": 42620644.0, "mean_token_accuracy": 0.9625461868941784, "epoch": 9.584, "step": 5990 }, { "loss": 0.1681, "grad_norm": 1.0162158012390137, "learning_rate": 6.065302442555543e-05, "entropy": 0.25498022614046933, "num_tokens": 42692776.0, "mean_token_accuracy": 0.9638829886913299, "epoch": 9.6, "step": 6000 }, { "loss": 0.1606, "grad_norm": 0.9396739602088928, "learning_rate": 6.0335632694484786e-05, "entropy": 0.25690388344228265, "num_tokens": 42761413.0, "mean_token_accuracy": 0.9656979449093341, "epoch": 9.616, "step": 6010 }, { "loss": 0.1738, "grad_norm": 1.0537030696868896, "learning_rate": 6.001871442577833e-05, "entropy": 0.2666920253075659, "num_tokens": 42832433.0, "mean_token_accuracy": 0.9615103788673878, "epoch": 9.632, "step": 6020 }, { "loss": 0.1682, "grad_norm": 0.7915371060371399, "learning_rate": 5.970227340240002e-05, "entropy": 0.26054652519524096, "num_tokens": 42905579.0, "mean_token_accuracy": 0.9649281807243824, "epoch": 9.648, "step": 6030 }, { "loss": 0.1591, "grad_norm": 0.7845244407653809, "learning_rate": 5.938631340161711e-05, "entropy": 0.250272882822901, "num_tokens": 42976821.0, "mean_token_accuracy": 0.9660152614116668, "epoch": 9.664, "step": 6040 }, { "loss": 0.1691, "grad_norm": 0.8098534345626831, "learning_rate": 5.9070838194954995e-05, "entropy": 0.25881309872493147, "num_tokens": 43050817.0, "mean_token_accuracy": 0.9644887626171113, "epoch": 9.68, "step": 6050 }, { "loss": 0.1622, "grad_norm": 0.816795289516449, "learning_rate": 5.8755851548152205e-05, "entropy": 0.2446274903602898, "num_tokens": 43121627.0, "mean_token_accuracy": 0.9650425039231777, "epoch": 9.696, "step": 6060 }, { "loss": 0.1721, "grad_norm": 1.0384894609451294, "learning_rate": 5.844135722111555e-05, "entropy": 0.26637672940269114, "num_tokens": 43195719.0, "mean_token_accuracy": 0.9627684839069843, "epoch": 9.712, "step": 6070 }, { "loss": 0.178, "grad_norm": 0.859541654586792, "learning_rate": 5.8127358967875025e-05, "entropy": 0.2715363884344697, "num_tokens": 43261492.0, "mean_token_accuracy": 0.9619791120290756, "epoch": 9.728, "step": 6080 }, { "loss": 0.1789, "grad_norm": 0.9277631044387817, "learning_rate": 5.7813860536539274e-05, "entropy": 0.2708555068820715, "num_tokens": 43332286.0, "mean_token_accuracy": 0.9608107179403305, "epoch": 9.744, "step": 6090 }, { "loss": 0.152, "grad_norm": 0.7766569256782532, "learning_rate": 5.7500865669250626e-05, "entropy": 0.23946294621564448, "num_tokens": 43408710.0, "mean_token_accuracy": 0.9676922507584095, "epoch": 9.76, "step": 6100 }, { "loss": 0.158, "grad_norm": 0.7725386619567871, "learning_rate": 5.718837810214046e-05, "entropy": 0.24342647939920425, "num_tokens": 43481169.0, "mean_token_accuracy": 0.9665091663599015, "epoch": 9.776, "step": 6110 }, { "loss": 0.1672, "grad_norm": 0.7234213948249817, "learning_rate": 5.687640156528483e-05, "entropy": 0.2518201654776931, "num_tokens": 43553694.0, "mean_token_accuracy": 0.9636642657220363, "epoch": 9.792, "step": 6120 }, { "loss": 0.1674, "grad_norm": 0.9298166632652283, "learning_rate": 5.65649397826596e-05, "entropy": 0.257735395245254, "num_tokens": 43621804.0, "mean_token_accuracy": 0.9640507340431214, "epoch": 9.808, "step": 6130 }, { "loss": 0.1729, "grad_norm": 0.9965875148773193, "learning_rate": 5.6253996472096195e-05, "entropy": 0.2640981442295015, "num_tokens": 43691839.0, "mean_token_accuracy": 0.9621976710855961, "epoch": 9.824, "step": 6140 }, { "loss": 0.1586, "grad_norm": 0.7626583576202393, "learning_rate": 5.594357534523723e-05, "entropy": 0.24685132470913232, "num_tokens": 43762114.0, "mean_token_accuracy": 0.9649995803833008, "epoch": 9.84, "step": 6150 }, { "loss": 0.1692, "grad_norm": 0.9121729135513306, "learning_rate": 5.563368010749208e-05, "entropy": 0.2647077966481447, "num_tokens": 43830364.0, "mean_token_accuracy": 0.9636400885879993, "epoch": 9.856, "step": 6160 }, { "loss": 0.1724, "grad_norm": 0.8674713373184204, "learning_rate": 5.532431445799284e-05, "entropy": 0.26577149322256444, "num_tokens": 43901768.0, "mean_token_accuracy": 0.9628065168857575, "epoch": 9.872, "step": 6170 }, { "loss": 0.1609, "grad_norm": 0.9042540192604065, "learning_rate": 5.501548208955003e-05, "entropy": 0.25197874922305347, "num_tokens": 43973633.0, "mean_token_accuracy": 0.9655407838523388, "epoch": 9.888, "step": 6180 }, { "loss": 0.1699, "grad_norm": 1.1059504747390747, "learning_rate": 5.470718668860848e-05, "entropy": 0.2651668076403439, "num_tokens": 44042363.0, "mean_token_accuracy": 0.9621016822755337, "epoch": 9.904, "step": 6190 }, { "loss": 0.1566, "grad_norm": 0.7089834213256836, "learning_rate": 5.439943193520342e-05, "entropy": 0.25018488001078365, "num_tokens": 44114542.0, "mean_token_accuracy": 0.9660967275500297, "epoch": 9.92, "step": 6200 }, { "loss": 0.1639, "grad_norm": 0.7003989219665527, "learning_rate": 5.409222150291651e-05, "entropy": 0.26170137082226574, "num_tokens": 44182628.0, "mean_token_accuracy": 0.9629987984895706, "epoch": 9.936, "step": 6210 }, { "loss": 0.1727, "grad_norm": 1.0769059658050537, "learning_rate": 5.378555905883209e-05, "entropy": 0.26657488849014044, "num_tokens": 44254153.0, "mean_token_accuracy": 0.9631704069674015, "epoch": 9.952, "step": 6220 }, { "loss": 0.1677, "grad_norm": 0.8431476354598999, "learning_rate": 5.347944826349323e-05, "entropy": 0.2599874511361122, "num_tokens": 44325612.0, "mean_token_accuracy": 0.9641674496233463, "epoch": 9.968, "step": 6230 }, { "loss": 0.1542, "grad_norm": 0.9448006749153137, "learning_rate": 5.3173892770858116e-05, "entropy": 0.23997493041679263, "num_tokens": 44396904.0, "mean_token_accuracy": 0.9672681398689746, "epoch": 9.984, "step": 6240 }, { "loss": 0.175, "grad_norm": 0.9028638601303101, "learning_rate": 5.28688962282565e-05, "entropy": 0.2642019227147102, "num_tokens": 44467220.0, "mean_token_accuracy": 0.961811276525259, "epoch": 10.0, "step": 6250 }, { "eval_loss": 2.8563432693481445, "eval_runtime": 53.156, "eval_samples_per_second": 4.233, "eval_steps_per_second": 4.233, "eval_entropy": 0.866648942232132, "eval_num_tokens": 44467220.0, "eval_mean_token_accuracy": 0.6050341996881697, "epoch": 10.0, "step": 6250 }, { "loss": 0.1239, "grad_norm": 0.6332527995109558, "learning_rate": 5.256446227634604e-05, "entropy": 0.2157553312368691, "num_tokens": 44535849.0, "mean_token_accuracy": 0.9760536558926105, "epoch": 10.016, "step": 6260 }, { "loss": 0.1205, "grad_norm": 0.8870267868041992, "learning_rate": 5.2260594549069e-05, "entropy": 0.20029621049761773, "num_tokens": 44607316.0, "mean_token_accuracy": 0.9753897979855537, "epoch": 10.032, "step": 6270 }, { "loss": 0.1213, "grad_norm": 0.7666226625442505, "learning_rate": 5.195729667360871e-05, "entropy": 0.20257386625744403, "num_tokens": 44677070.0, "mean_token_accuracy": 0.9761847510933876, "epoch": 10.048, "step": 6280 }, { "loss": 0.1392, "grad_norm": 0.6441207528114319, "learning_rate": 5.1654572270346356e-05, "entropy": 0.22155026979744435, "num_tokens": 44748158.0, "mean_token_accuracy": 0.9725708521902561, "epoch": 10.064, "step": 6290 }, { "loss": 0.1186, "grad_norm": 0.6694724559783936, "learning_rate": 5.135242495281771e-05, "entropy": 0.19015136314556003, "num_tokens": 44820225.0, "mean_token_accuracy": 0.9769647605717182, "epoch": 10.08, "step": 6300 }, { "loss": 0.1317, "grad_norm": 0.621321976184845, "learning_rate": 5.1050858327670136e-05, "entropy": 0.21349630267359315, "num_tokens": 44893530.0, "mean_token_accuracy": 0.973505049943924, "epoch": 10.096, "step": 6310 }, { "loss": 0.1208, "grad_norm": 0.7306549549102783, "learning_rate": 5.0749875994619356e-05, "entropy": 0.19401828572154045, "num_tokens": 44966348.0, "mean_token_accuracy": 0.9757913880050182, "epoch": 10.112, "step": 6320 }, { "loss": 0.1385, "grad_norm": 0.690692663192749, "learning_rate": 5.044948154640656e-05, "entropy": 0.22190411519259215, "num_tokens": 45034886.0, "mean_token_accuracy": 0.9721606239676476, "epoch": 10.128, "step": 6330 }, { "loss": 0.121, "grad_norm": 0.7270233035087585, "learning_rate": 5.0149678568755545e-05, "entropy": 0.19890884049236773, "num_tokens": 45103440.0, "mean_token_accuracy": 0.9752137348055839, "epoch": 10.144, "step": 6340 }, { "loss": 0.1291, "grad_norm": 0.6128872632980347, "learning_rate": 4.985047064032987e-05, "entropy": 0.20917901964858174, "num_tokens": 45175806.0, "mean_token_accuracy": 0.9743299268186092, "epoch": 10.16, "step": 6350 }, { "loss": 0.1336, "grad_norm": 0.7374527454376221, "learning_rate": 4.955186133269023e-05, "entropy": 0.22092988500371574, "num_tokens": 45245006.0, "mean_token_accuracy": 0.971930580586195, "epoch": 10.176, "step": 6360 }, { "loss": 0.1218, "grad_norm": 0.6083544492721558, "learning_rate": 4.925385421025167e-05, "entropy": 0.2019744067452848, "num_tokens": 45314972.0, "mean_token_accuracy": 0.974312373995781, "epoch": 10.192, "step": 6370 }, { "loss": 0.1296, "grad_norm": 0.6664021611213684, "learning_rate": 4.895645283024116e-05, "entropy": 0.21060689948499203, "num_tokens": 45384237.0, "mean_token_accuracy": 0.9731297813355922, "epoch": 10.208, "step": 6380 }, { "loss": 0.128, "grad_norm": 0.8613024353981018, "learning_rate": 4.8659660742655024e-05, "entropy": 0.2190774405375123, "num_tokens": 45454743.0, "mean_token_accuracy": 0.9734842479228973, "epoch": 10.224, "step": 6390 }, { "loss": 0.1237, "grad_norm": 0.871457040309906, "learning_rate": 4.8363481490216754e-05, "entropy": 0.2010152987204492, "num_tokens": 45527965.0, "mean_token_accuracy": 0.9749523274600506, "epoch": 10.24, "step": 6400 }, { "loss": 0.1337, "grad_norm": 0.5624682903289795, "learning_rate": 4.809744705924049e-05, "entropy": 0.21898201797157527, "num_tokens": 45600028.0, "mean_token_accuracy": 0.972916466742754, "epoch": 10.256, "step": 6410 }, { "loss": 0.1242, "grad_norm": 0.7315216660499573, "learning_rate": 4.7802441927552686e-05, "entropy": 0.1996945803053677, "num_tokens": 45668132.0, "mean_token_accuracy": 0.9751198820769786, "epoch": 10.272, "step": 6420 }, { "loss": 0.128, "grad_norm": 0.7099934816360474, "learning_rate": 4.7508059863391906e-05, "entropy": 0.20152195487171412, "num_tokens": 45739572.0, "mean_token_accuracy": 0.9744913943111897, "epoch": 10.288, "step": 6430 }, { "loss": 0.1294, "grad_norm": 0.7074333429336548, "learning_rate": 4.7214304380713883e-05, "entropy": 0.20873078322038055, "num_tokens": 45813258.0, "mean_token_accuracy": 0.9739388301968575, "epoch": 10.304, "step": 6440 }, { "loss": 0.1289, "grad_norm": 0.4944125711917877, "learning_rate": 4.6921178985994896e-05, "entropy": 0.2143176795914769, "num_tokens": 45886162.0, "mean_token_accuracy": 0.974186547845602, "epoch": 10.32, "step": 6450 }, { "loss": 0.14, "grad_norm": 0.647855818271637, "learning_rate": 4.662868717819008e-05, "entropy": 0.22046252256259322, "num_tokens": 45957356.0, "mean_token_accuracy": 0.9719342090189457, "epoch": 10.336, "step": 6460 }, { "loss": 0.1409, "grad_norm": 0.6705102920532227, "learning_rate": 4.633683244869172e-05, "entropy": 0.22202565195038915, "num_tokens": 46031001.0, "mean_token_accuracy": 0.9715973980724811, "epoch": 10.352, "step": 6470 }, { "loss": 0.1166, "grad_norm": 0.6515005230903625, "learning_rate": 4.604561828128733e-05, "entropy": 0.1849020540714264, "num_tokens": 46101742.0, "mean_token_accuracy": 0.9770025290548802, "epoch": 10.368, "step": 6480 }, { "loss": 0.1299, "grad_norm": 0.6584956645965576, "learning_rate": 4.5755048152118304e-05, "entropy": 0.20367154031991958, "num_tokens": 46174027.0, "mean_token_accuracy": 0.9733322270214557, "epoch": 10.384, "step": 6490 }, { "loss": 0.1249, "grad_norm": 0.7282118797302246, "learning_rate": 4.5465125529638305e-05, "entropy": 0.19983167136088015, "num_tokens": 46248922.0, "mean_token_accuracy": 0.9750345937907696, "epoch": 10.4, "step": 6500 }, { "loss": 0.1515, "grad_norm": 0.7358629703521729, "learning_rate": 4.517585387457187e-05, "entropy": 0.2463570193387568, "num_tokens": 46315620.0, "mean_token_accuracy": 0.968318247795105, "epoch": 10.416, "step": 6510 }, { "loss": 0.122, "grad_norm": 0.7602038979530334, "learning_rate": 4.488723663987321e-05, "entropy": 0.1966056229546666, "num_tokens": 46387837.0, "mean_token_accuracy": 0.9759966738522052, "epoch": 10.432, "step": 6520 }, { "loss": 0.119, "grad_norm": 0.5757632851600647, "learning_rate": 4.4599277270684824e-05, "entropy": 0.1981935636140406, "num_tokens": 46462781.0, "mean_token_accuracy": 0.9765027850866318, "epoch": 10.448, "step": 6530 }, { "loss": 0.1293, "grad_norm": 0.792890727519989, "learning_rate": 4.431197920429645e-05, "entropy": 0.2117100368719548, "num_tokens": 46537321.0, "mean_token_accuracy": 0.973727760463953, "epoch": 10.464, "step": 6540 }, { "loss": 0.1272, "grad_norm": 0.5456752777099609, "learning_rate": 4.4025345870104054e-05, "entropy": 0.20415150783956051, "num_tokens": 46606915.0, "mean_token_accuracy": 0.9747604489326477, "epoch": 10.48, "step": 6550 }, { "loss": 0.1382, "grad_norm": 0.7836380004882812, "learning_rate": 4.3739380689568955e-05, "entropy": 0.21043188832700252, "num_tokens": 46678340.0, "mean_token_accuracy": 0.9719588063657284, "epoch": 10.496, "step": 6560 }, { "loss": 0.1397, "grad_norm": 0.6452347636222839, "learning_rate": 4.345408707617681e-05, "entropy": 0.23038826417177916, "num_tokens": 46746565.0, "mean_token_accuracy": 0.9707624129951, "epoch": 10.512, "step": 6570 }, { "loss": 0.1306, "grad_norm": 0.6284129619598389, "learning_rate": 4.316946843539701e-05, "entropy": 0.21005241535604, "num_tokens": 46816752.0, "mean_token_accuracy": 0.973749927431345, "epoch": 10.528, "step": 6580 }, { "loss": 0.1252, "grad_norm": 0.8489777445793152, "learning_rate": 4.2885528164642e-05, "entropy": 0.20542811863124372, "num_tokens": 46887785.0, "mean_token_accuracy": 0.974838101118803, "epoch": 10.544, "step": 6590 }, { "loss": 0.1336, "grad_norm": 0.7341023087501526, "learning_rate": 4.2602269653226644e-05, "entropy": 0.2093356036581099, "num_tokens": 46960581.0, "mean_token_accuracy": 0.9741512671113014, "epoch": 10.56, "step": 6600 }, { "loss": 0.1166, "grad_norm": 0.5979950428009033, "learning_rate": 4.231969628232797e-05, "entropy": 0.19281008960679175, "num_tokens": 47033114.0, "mean_token_accuracy": 0.9761400036513805, "epoch": 10.576, "step": 6610 }, { "loss": 0.1347, "grad_norm": 0.7809708118438721, "learning_rate": 4.2037811424944574e-05, "entropy": 0.2198480661958456, "num_tokens": 47103398.0, "mean_token_accuracy": 0.972527951747179, "epoch": 10.592, "step": 6620 }, { "loss": 0.1242, "grad_norm": 0.5542659759521484, "learning_rate": 4.1756618445856475e-05, "entropy": 0.2000423938035965, "num_tokens": 47177975.0, "mean_token_accuracy": 0.9748056583106518, "epoch": 10.608, "step": 6630 }, { "loss": 0.125, "grad_norm": 0.7994014620780945, "learning_rate": 4.147612070158491e-05, "entropy": 0.2061722468584776, "num_tokens": 47254506.0, "mean_token_accuracy": 0.975242943316698, "epoch": 10.624, "step": 6640 }, { "loss": 0.1297, "grad_norm": 0.5730145573616028, "learning_rate": 4.119632154035241e-05, "entropy": 0.21104849921539426, "num_tokens": 47324809.0, "mean_token_accuracy": 0.9733332604169845, "epoch": 10.64, "step": 6650 }, { "loss": 0.1347, "grad_norm": 0.8271772861480713, "learning_rate": 4.091722430204256e-05, "entropy": 0.21633169213309883, "num_tokens": 47393574.0, "mean_token_accuracy": 0.9725298747420311, "epoch": 10.656, "step": 6660 }, { "loss": 0.1278, "grad_norm": 0.7446340918540955, "learning_rate": 4.063883231816044e-05, "entropy": 0.2033039097674191, "num_tokens": 47466230.0, "mean_token_accuracy": 0.9743968114256859, "epoch": 10.672, "step": 6670 }, { "loss": 0.1279, "grad_norm": 0.7032793760299683, "learning_rate": 4.03611489117926e-05, "entropy": 0.20807351395487786, "num_tokens": 47535277.0, "mean_token_accuracy": 0.9739071294665337, "epoch": 10.688, "step": 6680 }, { "loss": 0.1271, "grad_norm": 0.6926349401473999, "learning_rate": 4.008417739756753e-05, "entropy": 0.2175192498601973, "num_tokens": 47603486.0, "mean_token_accuracy": 0.9730976164340973, "epoch": 10.704, "step": 6690 }, { "loss": 0.1367, "grad_norm": 0.7771379351615906, "learning_rate": 3.980792108161605e-05, "entropy": 0.22200953382998706, "num_tokens": 47670926.0, "mean_token_accuracy": 0.9713366881012917, "epoch": 10.72, "step": 6700 }, { "loss": 0.1304, "grad_norm": 0.7071493268013, "learning_rate": 3.953238326153193e-05, "entropy": 0.2104584775865078, "num_tokens": 47740618.0, "mean_token_accuracy": 0.97333719804883, "epoch": 10.736, "step": 6710 }, { "loss": 0.1274, "grad_norm": 0.6477629542350769, "learning_rate": 3.925756722633237e-05, "entropy": 0.21221939409151674, "num_tokens": 47815475.0, "mean_token_accuracy": 0.9739153109490871, "epoch": 10.752, "step": 6720 }, { "loss": 0.1304, "grad_norm": 0.9050307869911194, "learning_rate": 3.8983476256418874e-05, "entropy": 0.20829010517336427, "num_tokens": 47886743.0, "mean_token_accuracy": 0.9738272845745086, "epoch": 10.768, "step": 6730 }, { "loss": 0.1218, "grad_norm": 0.8196435570716858, "learning_rate": 3.871011362353798e-05, "entropy": 0.19618914872407914, "num_tokens": 47961578.0, "mean_token_accuracy": 0.9754014074802398, "epoch": 10.784, "step": 6740 }, { "loss": 0.1228, "grad_norm": 0.6681481003761292, "learning_rate": 3.843748259074244e-05, "entropy": 0.2035204851999879, "num_tokens": 48035292.0, "mean_token_accuracy": 0.9749187581241131, "epoch": 10.8, "step": 6750 }, { "loss": 0.1328, "grad_norm": 0.6521177291870117, "learning_rate": 3.81655864123519e-05, "entropy": 0.21425552265718578, "num_tokens": 48104317.0, "mean_token_accuracy": 0.9734170585870743, "epoch": 10.816, "step": 6760 }, { "loss": 0.1419, "grad_norm": 0.7844504714012146, "learning_rate": 3.78944283339144e-05, "entropy": 0.22933667246252298, "num_tokens": 48172291.0, "mean_token_accuracy": 0.970756521821022, "epoch": 10.832, "step": 6770 }, { "loss": 0.1326, "grad_norm": 0.557014524936676, "learning_rate": 3.7624011592167396e-05, "entropy": 0.21116115297190846, "num_tokens": 48242054.0, "mean_token_accuracy": 0.9723976865410805, "epoch": 10.848, "step": 6780 }, { "loss": 0.1257, "grad_norm": 0.754261314868927, "learning_rate": 3.735433941499924e-05, "entropy": 0.203738493937999, "num_tokens": 48313155.0, "mean_token_accuracy": 0.9749681808054447, "epoch": 10.864, "step": 6790 }, { "loss": 0.1271, "grad_norm": 0.7218680381774902, "learning_rate": 3.7085415021410706e-05, "entropy": 0.20570443160831928, "num_tokens": 48383519.0, "mean_token_accuracy": 0.9738583870232105, "epoch": 10.88, "step": 6800 }, { "loss": 0.1384, "grad_norm": 0.6295368075370789, "learning_rate": 3.6817241621476384e-05, "entropy": 0.23252024864777923, "num_tokens": 48453307.0, "mean_token_accuracy": 0.9719019345939159, "epoch": 10.896, "step": 6810 }, { "loss": 0.1611, "grad_norm": 0.9336260557174683, "learning_rate": 3.654982241630652e-05, "entropy": 0.23568118829280138, "num_tokens": 48526921.0, "mean_token_accuracy": 0.9684044919908047, "epoch": 10.912, "step": 6820 }, { "loss": 0.1296, "grad_norm": 0.7827471494674683, "learning_rate": 3.628316059800868e-05, "entropy": 0.2157411332242191, "num_tokens": 48598143.0, "mean_token_accuracy": 0.972834387421608, "epoch": 10.928, "step": 6830 }, { "loss": 0.1245, "grad_norm": 0.7971508502960205, "learning_rate": 3.6017259349649854e-05, "entropy": 0.1979653806425631, "num_tokens": 48670073.0, "mean_token_accuracy": 0.974965862929821, "epoch": 10.943999999999999, "step": 6840 }, { "loss": 0.1367, "grad_norm": 0.7692677974700928, "learning_rate": 3.57521218452182e-05, "entropy": 0.21803633542731404, "num_tokens": 48738324.0, "mean_token_accuracy": 0.9724878318607807, "epoch": 10.96, "step": 6850 }, { "loss": 0.1383, "grad_norm": 0.6828943490982056, "learning_rate": 3.548775124958532e-05, "entropy": 0.22128824079409243, "num_tokens": 48805838.0, "mean_token_accuracy": 0.9710485093295574, "epoch": 10.975999999999999, "step": 6860 }, { "loss": 0.1391, "grad_norm": 0.8902535438537598, "learning_rate": 3.522415071846844e-05, "entropy": 0.2197513627819717, "num_tokens": 48877813.0, "mean_token_accuracy": 0.971608704328537, "epoch": 10.992, "step": 6870 }, { "eval_loss": 3.0090174674987793, "eval_runtime": 53.8985, "eval_samples_per_second": 4.175, "eval_steps_per_second": 4.175, "eval_entropy": 0.8023658769660525, "eval_num_tokens": 48913942.0, "eval_mean_token_accuracy": 0.6042358354727427, "epoch": 11.0, "step": 6875 }, { "loss": 0.1068, "grad_norm": 0.5849855542182922, "learning_rate": 3.496132339839271e-05, "entropy": 0.18352556303143502, "num_tokens": 48950058.0, "mean_token_accuracy": 0.9793416492640972, "epoch": 11.008, "step": 6880 }, { "loss": 0.1219, "grad_norm": 0.6256635785102844, "learning_rate": 3.469927242665375e-05, "entropy": 0.206114200130105, "num_tokens": 49017266.0, "mean_token_accuracy": 0.9771608673036098, "epoch": 11.024, "step": 6890 }, { "loss": 0.0977, "grad_norm": 0.46146005392074585, "learning_rate": 3.443800093128011e-05, "entropy": 0.16538381036370992, "num_tokens": 49090084.0, "mean_token_accuracy": 0.9821187578141689, "epoch": 11.04, "step": 6900 }, { "loss": 0.1074, "grad_norm": 0.7215375304222107, "learning_rate": 3.4177512030995926e-05, "entropy": 0.1829201366752386, "num_tokens": 49159124.0, "mean_token_accuracy": 0.978987704217434, "epoch": 11.056, "step": 6910 }, { "loss": 0.099, "grad_norm": 0.5193976759910583, "learning_rate": 3.3917808835183706e-05, "entropy": 0.1686112449504435, "num_tokens": 49231136.0, "mean_token_accuracy": 0.9818178720772266, "epoch": 11.072, "step": 6920 }, { "loss": 0.0994, "grad_norm": 0.5599414110183716, "learning_rate": 3.365889444384721e-05, "entropy": 0.16642968403175473, "num_tokens": 49302151.0, "mean_token_accuracy": 0.9816991679370404, "epoch": 11.088, "step": 6930 }, { "loss": 0.1142, "grad_norm": 0.7030749917030334, "learning_rate": 3.340077194757456e-05, "entropy": 0.18803519140928984, "num_tokens": 49375371.0, "mean_token_accuracy": 0.9782954022288323, "epoch": 11.104, "step": 6940 }, { "loss": 0.0997, "grad_norm": 0.6080374121665955, "learning_rate": 3.314344442750116e-05, "entropy": 0.16656863475218414, "num_tokens": 49446522.0, "mean_token_accuracy": 0.9813706450164318, "epoch": 11.12, "step": 6950 }, { "loss": 0.1148, "grad_norm": 0.5571662187576294, "learning_rate": 3.288691495527301e-05, "entropy": 0.19810480657033622, "num_tokens": 49514699.0, "mean_token_accuracy": 0.9776599563658237, "epoch": 11.136, "step": 6960 }, { "loss": 0.1045, "grad_norm": 0.49021604657173157, "learning_rate": 3.263118659301008e-05, "entropy": 0.1782640275079757, "num_tokens": 49587901.0, "mean_token_accuracy": 0.9803149081766606, "epoch": 11.152, "step": 6970 }, { "loss": 0.1003, "grad_norm": 0.6382109522819519, "learning_rate": 3.237626239326965e-05, "entropy": 0.1723961304873228, "num_tokens": 49661544.0, "mean_token_accuracy": 0.9812774859368801, "epoch": 11.168, "step": 6980 }, { "loss": 0.107, "grad_norm": 0.5811018347740173, "learning_rate": 3.2122145399010074e-05, "entropy": 0.18397795790806412, "num_tokens": 49728448.0, "mean_token_accuracy": 0.9788592837750911, "epoch": 11.184, "step": 6990 }, { "loss": 0.1134, "grad_norm": 0.5804753303527832, "learning_rate": 3.1868838643554166e-05, "entropy": 0.197007495444268, "num_tokens": 49794435.0, "mean_token_accuracy": 0.9773269660770894, "epoch": 11.2, "step": 7000 }, { "loss": 0.1012, "grad_norm": 0.6094357371330261, "learning_rate": 3.161634515055323e-05, "entropy": 0.16826781537383795, "num_tokens": 49864765.0, "mean_token_accuracy": 0.9806108541786671, "epoch": 11.216, "step": 7010 }, { "loss": 0.1067, "grad_norm": 0.6471855640411377, "learning_rate": 3.1364667933950845e-05, "entropy": 0.18037505801767112, "num_tokens": 49937789.0, "mean_token_accuracy": 0.9803285896778107, "epoch": 11.232, "step": 7020 }, { "loss": 0.0955, "grad_norm": 0.5202105045318604, "learning_rate": 3.1113809997946996e-05, "entropy": 0.15841048760339618, "num_tokens": 50011162.0, "mean_token_accuracy": 0.9822677031159401, "epoch": 11.248, "step": 7030 }, { "loss": 0.1036, "grad_norm": 0.716526985168457, "learning_rate": 3.08637743369621e-05, "entropy": 0.1733441966585815, "num_tokens": 50081774.0, "mean_token_accuracy": 0.9805869229137898, "epoch": 11.264, "step": 7040 }, { "loss": 0.1017, "grad_norm": 0.4948079288005829, "learning_rate": 3.061456393560129e-05, "entropy": 0.17455019410699607, "num_tokens": 50154831.0, "mean_token_accuracy": 0.9809325553476811, "epoch": 11.28, "step": 7050 }, { "loss": 0.0969, "grad_norm": 0.53554767370224, "learning_rate": 3.0366181768618817e-05, "entropy": 0.1630277507007122, "num_tokens": 50225830.0, "mean_token_accuracy": 0.9821226269006729, "epoch": 11.296, "step": 7060 }, { "loss": 0.1006, "grad_norm": 0.8505144715309143, "learning_rate": 3.0118630800882596e-05, "entropy": 0.16263017570599914, "num_tokens": 50296425.0, "mean_token_accuracy": 0.9811368383467197, "epoch": 11.312, "step": 7070 }, { "loss": 0.108, "grad_norm": 0.6110939383506775, "learning_rate": 2.9871913987338673e-05, "entropy": 0.17711443724110723, "num_tokens": 50365318.0, "mean_token_accuracy": 0.9793041288852692, "epoch": 11.328, "step": 7080 }, { "loss": 0.1069, "grad_norm": 0.4753178358078003, "learning_rate": 2.9626034272976056e-05, "entropy": 0.18435784978792072, "num_tokens": 50436698.0, "mean_token_accuracy": 0.97963672503829, "epoch": 11.344, "step": 7090 }, { "loss": 0.1101, "grad_norm": 0.8963127136230469, "learning_rate": 2.9380994592791545e-05, "entropy": 0.17894844841212035, "num_tokens": 50508017.0, "mean_token_accuracy": 0.979377930611372, "epoch": 11.36, "step": 7100 }, { "loss": 0.1002, "grad_norm": 0.6266742944717407, "learning_rate": 2.913679787175465e-05, "entropy": 0.16800236902199686, "num_tokens": 50583212.0, "mean_token_accuracy": 0.9817425340414048, "epoch": 11.376, "step": 7110 }, { "loss": 0.1025, "grad_norm": 0.5600296854972839, "learning_rate": 2.889344702477278e-05, "entropy": 0.17211269345134497, "num_tokens": 50655769.0, "mean_token_accuracy": 0.9806753464043141, "epoch": 11.392, "step": 7120 }, { "loss": 0.1025, "grad_norm": 0.5361045598983765, "learning_rate": 2.865094495665638e-05, "entropy": 0.1741516502574086, "num_tokens": 50728632.0, "mean_token_accuracy": 0.9794694639742374, "epoch": 11.408, "step": 7130 }, { "loss": 0.1017, "grad_norm": 0.5622146129608154, "learning_rate": 2.84092945620842e-05, "entropy": 0.16967379879206418, "num_tokens": 50803852.0, "mean_token_accuracy": 0.9812305927276611, "epoch": 11.424, "step": 7140 }, { "loss": 0.1143, "grad_norm": 0.5616720914840698, "learning_rate": 2.8168498725568837e-05, "entropy": 0.19283345742151142, "num_tokens": 50874490.0, "mean_token_accuracy": 0.9781708754599094, "epoch": 11.44, "step": 7150 }, { "loss": 0.1115, "grad_norm": 0.9333820939064026, "learning_rate": 2.7928560321422237e-05, "entropy": 0.1841776268556714, "num_tokens": 50945521.0, "mean_token_accuracy": 0.9796050064265728, "epoch": 11.456, "step": 7160 }, { "loss": 0.1035, "grad_norm": 0.5893402695655823, "learning_rate": 2.7689482213721517e-05, "entropy": 0.1747731989249587, "num_tokens": 51016965.0, "mean_token_accuracy": 0.9803075574338436, "epoch": 11.472, "step": 7170 }, { "loss": 0.1134, "grad_norm": 0.6170578598976135, "learning_rate": 2.745126725627458e-05, "entropy": 0.1899030352011323, "num_tokens": 51086502.0, "mean_token_accuracy": 0.9783800221979618, "epoch": 11.488, "step": 7180 }, { "loss": 0.1075, "grad_norm": 0.6724886298179626, "learning_rate": 2.7213918292586173e-05, "entropy": 0.17713073138147592, "num_tokens": 51159636.0, "mean_token_accuracy": 0.9807966135442256, "epoch": 11.504, "step": 7190 }, { "loss": 0.108, "grad_norm": 0.6363348960876465, "learning_rate": 2.6977438155823933e-05, "entropy": 0.1811823179014027, "num_tokens": 51228032.0, "mean_token_accuracy": 0.9797785244882107, "epoch": 11.52, "step": 7200 }, { "loss": 0.111, "grad_norm": 0.71759033203125, "learning_rate": 2.67418296687845e-05, "entropy": 0.18422073973342776, "num_tokens": 51296486.0, "mean_token_accuracy": 0.9785069808363914, "epoch": 11.536, "step": 7210 }, { "loss": 0.1076, "grad_norm": 0.5558964014053345, "learning_rate": 2.650709564386e-05, "entropy": 0.18222952368669212, "num_tokens": 51367430.0, "mean_token_accuracy": 0.9793256342411041, "epoch": 11.552, "step": 7220 }, { "loss": 0.1077, "grad_norm": 0.5821406841278076, "learning_rate": 2.6273238883004246e-05, "entropy": 0.181116136116907, "num_tokens": 51436002.0, "mean_token_accuracy": 0.9789303138852119, "epoch": 11.568, "step": 7230 }, { "loss": 0.1253, "grad_norm": 0.6732730865478516, "learning_rate": 2.6040262177699426e-05, "entropy": 0.19822444436140357, "num_tokens": 51508002.0, "mean_token_accuracy": 0.9768066488206386, "epoch": 11.584, "step": 7240 }, { "loss": 0.1034, "grad_norm": 0.5156833529472351, "learning_rate": 2.580816830892272e-05, "entropy": 0.17577411858364939, "num_tokens": 51581037.0, "mean_token_accuracy": 0.9805229149758816, "epoch": 11.6, "step": 7250 }, { "loss": 0.1079, "grad_norm": 0.5528436303138733, "learning_rate": 2.557696004711323e-05, "entropy": 0.17719884365797042, "num_tokens": 51650806.0, "mean_token_accuracy": 0.9792453184723854, "epoch": 11.616, "step": 7260 }, { "loss": 0.1082, "grad_norm": 0.8260487914085388, "learning_rate": 2.5346640152138723e-05, "entropy": 0.18247934188693762, "num_tokens": 51722362.0, "mean_token_accuracy": 0.9794173203408718, "epoch": 11.632, "step": 7270 }, { "loss": 0.1016, "grad_norm": 0.7104451060295105, "learning_rate": 2.511721137326284e-05, "entropy": 0.1656028908677399, "num_tokens": 51796380.0, "mean_token_accuracy": 0.9812566667795182, "epoch": 11.648, "step": 7280 }, { "loss": 0.1048, "grad_norm": 0.5455500483512878, "learning_rate": 2.4888676449112182e-05, "entropy": 0.17055389578454197, "num_tokens": 51868152.0, "mean_token_accuracy": 0.9803950227797031, "epoch": 11.664, "step": 7290 }, { "loss": 0.1026, "grad_norm": 0.5882729887962341, "learning_rate": 2.466103810764364e-05, "entropy": 0.17382117751985787, "num_tokens": 51940786.0, "mean_token_accuracy": 0.9810817994177341, "epoch": 11.68, "step": 7300 }, { "loss": 0.1027, "grad_norm": 0.5812434554100037, "learning_rate": 2.4434299066111953e-05, "entropy": 0.17270538122393192, "num_tokens": 52012737.0, "mean_token_accuracy": 0.9804746553301811, "epoch": 11.696, "step": 7310 }, { "loss": 0.1028, "grad_norm": 0.49763479828834534, "learning_rate": 2.4208462031037072e-05, "entropy": 0.17209364427253604, "num_tokens": 52086407.0, "mean_token_accuracy": 0.980265936255455, "epoch": 11.712, "step": 7320 }, { "loss": 0.1116, "grad_norm": 0.7350765466690063, "learning_rate": 2.398352969817196e-05, "entropy": 0.1886303871870041, "num_tokens": 52153748.0, "mean_token_accuracy": 0.977981960773468, "epoch": 11.728, "step": 7330 }, { "loss": 0.1027, "grad_norm": 0.6190215349197388, "learning_rate": 2.3759504752470463e-05, "entropy": 0.1742776150815189, "num_tokens": 52223334.0, "mean_token_accuracy": 0.980067191272974, "epoch": 11.744, "step": 7340 }, { "loss": 0.109, "grad_norm": 0.5890244841575623, "learning_rate": 2.353638986805513e-05, "entropy": 0.1852172072045505, "num_tokens": 52295182.0, "mean_token_accuracy": 0.9792810574173927, "epoch": 11.76, "step": 7350 }, { "loss": 0.1064, "grad_norm": 0.5243596434593201, "learning_rate": 2.3314187708185452e-05, "entropy": 0.1774914358742535, "num_tokens": 52367405.0, "mean_token_accuracy": 0.9798739768564702, "epoch": 11.776, "step": 7360 }, { "loss": 0.1022, "grad_norm": 0.4872676730155945, "learning_rate": 2.3092900925225903e-05, "entropy": 0.17283066119998694, "num_tokens": 52438892.0, "mean_token_accuracy": 0.9806529499590397, "epoch": 11.792, "step": 7370 }, { "loss": 0.1002, "grad_norm": 0.6441254019737244, "learning_rate": 2.287253216061438e-05, "entropy": 0.16436945265159011, "num_tokens": 52510840.0, "mean_token_accuracy": 0.9813690960407258, "epoch": 11.808, "step": 7380 }, { "loss": 0.118, "grad_norm": 0.5675173401832581, "learning_rate": 2.2653084044830687e-05, "entropy": 0.19531303457915783, "num_tokens": 52581964.0, "mean_token_accuracy": 0.977670143544674, "epoch": 11.824, "step": 7390 }, { "loss": 0.1064, "grad_norm": 0.6464403867721558, "learning_rate": 2.2434559197365034e-05, "entropy": 0.1793277831748128, "num_tokens": 52653772.0, "mean_token_accuracy": 0.9793172635138034, "epoch": 11.84, "step": 7400 }, { "loss": 0.1026, "grad_norm": 0.5074166655540466, "learning_rate": 2.2216960226686957e-05, "entropy": 0.17227189820259808, "num_tokens": 52727074.0, "mean_token_accuracy": 0.9803804717957973, "epoch": 11.856, "step": 7410 }, { "loss": 0.1264, "grad_norm": 0.4302932024002075, "learning_rate": 2.200028973021395e-05, "entropy": 0.19774739649146794, "num_tokens": 52796697.0, "mean_token_accuracy": 0.9752276577055454, "epoch": 11.872, "step": 7420 }, { "loss": 0.1125, "grad_norm": 0.48672178387641907, "learning_rate": 2.1784550294280616e-05, "entropy": 0.18949801493436097, "num_tokens": 52866931.0, "mean_token_accuracy": 0.9780422315001488, "epoch": 11.888, "step": 7430 }, { "loss": 0.1073, "grad_norm": 0.5931698083877563, "learning_rate": 2.1569744494107723e-05, "entropy": 0.1792034028097987, "num_tokens": 52937273.0, "mean_token_accuracy": 0.978913563489914, "epoch": 11.904, "step": 7440 }, { "loss": 0.1073, "grad_norm": 0.5881561636924744, "learning_rate": 2.1355874893771567e-05, "entropy": 0.17747059678658844, "num_tokens": 53009570.0, "mean_token_accuracy": 0.9794997930526733, "epoch": 11.92, "step": 7450 }, { "loss": 0.1081, "grad_norm": 0.5375831723213196, "learning_rate": 2.1142944046173207e-05, "entropy": 0.1834828875027597, "num_tokens": 53079129.0, "mean_token_accuracy": 0.979352767765522, "epoch": 11.936, "step": 7460 }, { "loss": 0.1058, "grad_norm": 0.7035475373268127, "learning_rate": 2.0930954493008115e-05, "entropy": 0.17621439220383764, "num_tokens": 53151102.0, "mean_token_accuracy": 0.9802208960056304, "epoch": 11.952, "step": 7470 }, { "loss": 0.1021, "grad_norm": 0.7972382307052612, "learning_rate": 2.0719908764735795e-05, "entropy": 0.1723351100459695, "num_tokens": 53222563.0, "mean_token_accuracy": 0.980517354607582, "epoch": 11.968, "step": 7480 }, { "loss": 0.1085, "grad_norm": 0.686572790145874, "learning_rate": 2.0509809380549537e-05, "entropy": 0.18022875115275383, "num_tokens": 53289876.0, "mean_token_accuracy": 0.9791302554309368, "epoch": 11.984, "step": 7490 }, { "loss": 0.1087, "grad_norm": 0.7836893796920776, "learning_rate": 2.0300658848346487e-05, "entropy": 0.1836314565502107, "num_tokens": 53360664.0, "mean_token_accuracy": 0.9787776321172714, "epoch": 12.0, "step": 7500 }, { "eval_loss": 3.1667251586914062, "eval_runtime": 53.9064, "eval_samples_per_second": 4.174, "eval_steps_per_second": 4.174, "eval_entropy": 0.7561789662308163, "eval_num_tokens": 53360664.0, "eval_mean_token_accuracy": 0.6035612714290619, "epoch": 12.0, "step": 7500 }, { "loss": 0.0882, "grad_norm": 0.5549907088279724, "learning_rate": 2.0092459664697517e-05, "entropy": 0.16086191297508776, "num_tokens": 53432739.0, "mean_token_accuracy": 0.9842534877359868, "epoch": 12.016, "step": 7510 }, { "loss": 0.0962, "grad_norm": 0.4880717098712921, "learning_rate": 1.9885214314817568e-05, "entropy": 0.16504003247246146, "num_tokens": 53501848.0, "mean_token_accuracy": 0.9829154796898365, "epoch": 12.032, "step": 7520 }, { "loss": 0.093, "grad_norm": 0.4875560402870178, "learning_rate": 1.9678925272535887e-05, "entropy": 0.15956798340193928, "num_tokens": 53570350.0, "mean_token_accuracy": 0.9826718807220459, "epoch": 12.048, "step": 7530 }, { "loss": 0.0938, "grad_norm": 0.31622323393821716, "learning_rate": 1.947359500026663e-05, "entropy": 0.16362849986180664, "num_tokens": 53642824.0, "mean_token_accuracy": 0.9836931116878986, "epoch": 12.064, "step": 7540 }, { "loss": 0.0873, "grad_norm": 0.43750253319740295, "learning_rate": 1.926922594897932e-05, "entropy": 0.15534009635448456, "num_tokens": 53714162.0, "mean_token_accuracy": 0.9834329225122929, "epoch": 12.08, "step": 7550 }, { "loss": 0.0921, "grad_norm": 0.35489797592163086, "learning_rate": 1.9065820558169644e-05, "entropy": 0.15326882498338817, "num_tokens": 53784324.0, "mean_token_accuracy": 0.9838205099105835, "epoch": 12.096, "step": 7560 }, { "loss": 0.0901, "grad_norm": 0.5639646053314209, "learning_rate": 1.8863381255830436e-05, "entropy": 0.15162807204760612, "num_tokens": 53854345.0, "mean_token_accuracy": 0.984326408803463, "epoch": 12.112, "step": 7570 }, { "loss": 0.0926, "grad_norm": 0.43230634927749634, "learning_rate": 1.8661910458422514e-05, "entropy": 0.15461639519780873, "num_tokens": 53926282.0, "mean_token_accuracy": 0.983488316833973, "epoch": 12.128, "step": 7580 }, { "loss": 0.0882, "grad_norm": 0.4113493859767914, "learning_rate": 1.846141057084594e-05, "entropy": 0.15016764723695813, "num_tokens": 53999251.0, "mean_token_accuracy": 0.9844455532729626, "epoch": 12.144, "step": 7590 }, { "loss": 0.0955, "grad_norm": 0.4295278787612915, "learning_rate": 1.8261883986411343e-05, "entropy": 0.16442636353895068, "num_tokens": 54070341.0, "mean_token_accuracy": 0.9828539147973061, "epoch": 12.16, "step": 7600 }, { "loss": 0.0896, "grad_norm": 0.36599329113960266, "learning_rate": 1.8063333086811272e-05, "entropy": 0.14966885969042779, "num_tokens": 54141284.0, "mean_token_accuracy": 0.9836175635457038, "epoch": 12.176, "step": 7610 }, { "loss": 0.0934, "grad_norm": 0.45671212673187256, "learning_rate": 1.7865760242091823e-05, "entropy": 0.15814985772594808, "num_tokens": 54212836.0, "mean_token_accuracy": 0.9834645815193653, "epoch": 12.192, "step": 7620 }, { "loss": 0.0933, "grad_norm": 0.4735674262046814, "learning_rate": 1.7669167810624256e-05, "entropy": 0.1542667276225984, "num_tokens": 54284816.0, "mean_token_accuracy": 0.9837598778307438, "epoch": 12.208, "step": 7630 }, { "loss": 0.1002, "grad_norm": 0.4404248297214508, "learning_rate": 1.747355813907704e-05, "entropy": 0.17326725954189898, "num_tokens": 54353483.0, "mean_token_accuracy": 0.9819205164909363, "epoch": 12.224, "step": 7640 }, { "loss": 0.0929, "grad_norm": 0.48079532384872437, "learning_rate": 1.7278933562387622e-05, "entropy": 0.15326191121712326, "num_tokens": 54424707.0, "mean_token_accuracy": 0.9834895506501198, "epoch": 12.24, "step": 7650 }, { "loss": 0.093, "grad_norm": 0.4942658245563507, "learning_rate": 1.7085296403734673e-05, "entropy": 0.1612951885908842, "num_tokens": 54496220.0, "mean_token_accuracy": 0.9834933511912822, "epoch": 12.256, "step": 7660 }, { "loss": 0.085, "grad_norm": 0.40816792845726013, "learning_rate": 1.6892648974510328e-05, "entropy": 0.13999008475802838, "num_tokens": 54571193.0, "mean_token_accuracy": 0.9847295343875885, "epoch": 12.272, "step": 7670 }, { "loss": 0.0941, "grad_norm": 0.4192746579647064, "learning_rate": 1.67009935742926e-05, "entropy": 0.16656063878908753, "num_tokens": 54640297.0, "mean_token_accuracy": 0.982462902367115, "epoch": 12.288, "step": 7680 }, { "loss": 0.0898, "grad_norm": 0.5430303812026978, "learning_rate": 1.651033249081797e-05, "entropy": 0.15198782617226242, "num_tokens": 54711105.0, "mean_token_accuracy": 0.9834337346255779, "epoch": 12.304, "step": 7690 }, { "loss": 0.0927, "grad_norm": 0.4365152418613434, "learning_rate": 1.632066799995401e-05, "entropy": 0.15395681224763394, "num_tokens": 54783948.0, "mean_token_accuracy": 0.9837517060339451, "epoch": 12.32, "step": 7700 }, { "loss": 0.0958, "grad_norm": 0.3381444811820984, "learning_rate": 1.6132002365672227e-05, "entropy": 0.1610792596824467, "num_tokens": 54856581.0, "mean_token_accuracy": 0.9828747831285, "epoch": 12.336, "step": 7710 }, { "loss": 0.0925, "grad_norm": 0.4189335107803345, "learning_rate": 1.5944337840021063e-05, "entropy": 0.15986438244581222, "num_tokens": 54927900.0, "mean_token_accuracy": 0.9829278640449047, "epoch": 12.352, "step": 7720 }, { "loss": 0.0898, "grad_norm": 0.5801244974136353, "learning_rate": 1.5757676663099076e-05, "entropy": 0.15716084130108357, "num_tokens": 55000213.0, "mean_token_accuracy": 0.9836381793022155, "epoch": 12.368, "step": 7730 }, { "loss": 0.0892, "grad_norm": 0.5472594499588013, "learning_rate": 1.5572021063028054e-05, "entropy": 0.15130675472319127, "num_tokens": 55072480.0, "mean_token_accuracy": 0.9838834188878536, "epoch": 12.384, "step": 7740 }, { "loss": 0.0907, "grad_norm": 0.5565981864929199, "learning_rate": 1.538737325592652e-05, "entropy": 0.15513508589938282, "num_tokens": 55144595.0, "mean_token_accuracy": 0.9841105192899704, "epoch": 12.4, "step": 7750 }, { "loss": 0.0904, "grad_norm": 0.3367442786693573, "learning_rate": 1.5203735445883282e-05, "entropy": 0.15252459235489368, "num_tokens": 55217093.0, "mean_token_accuracy": 0.9837716959416867, "epoch": 12.416, "step": 7760 }, { "loss": 0.0939, "grad_norm": 0.43458524346351624, "learning_rate": 1.5021109824931034e-05, "entropy": 0.1650460788514465, "num_tokens": 55284372.0, "mean_token_accuracy": 0.9820706158876419, "epoch": 12.432, "step": 7770 }, { "loss": 0.0871, "grad_norm": 0.4024854004383087, "learning_rate": 1.4839498573020339e-05, "entropy": 0.14761667069979012, "num_tokens": 55357256.0, "mean_token_accuracy": 0.9845916740596294, "epoch": 12.448, "step": 7780 }, { "loss": 0.0893, "grad_norm": 0.45392951369285583, "learning_rate": 1.4658903857993489e-05, "entropy": 0.1495253335684538, "num_tokens": 55429814.0, "mean_token_accuracy": 0.9843823350965977, "epoch": 12.464, "step": 7790 }, { "loss": 0.0927, "grad_norm": 0.3834194839000702, "learning_rate": 1.4479327835558654e-05, "entropy": 0.1640132769010961, "num_tokens": 55500672.0, "mean_token_accuracy": 0.9821198776364326, "epoch": 12.48, "step": 7800 }, { "loss": 0.0971, "grad_norm": 0.6415119767189026, "learning_rate": 1.4300772649264138e-05, "entropy": 0.1669124247506261, "num_tokens": 55570088.0, "mean_token_accuracy": 0.9823191277682781, "epoch": 12.496, "step": 7810 }, { "loss": 0.0952, "grad_norm": 0.6738594770431519, "learning_rate": 1.4123240430472828e-05, "entropy": 0.16212726458907128, "num_tokens": 55638902.0, "mean_token_accuracy": 0.9821942381560802, "epoch": 12.512, "step": 7820 }, { "loss": 0.0916, "grad_norm": 0.41530320048332214, "learning_rate": 1.3946733298336778e-05, "entropy": 0.15399048114195465, "num_tokens": 55709016.0, "mean_token_accuracy": 0.9833643361926079, "epoch": 12.528, "step": 7830 }, { "loss": 0.0852, "grad_norm": 0.5057052373886108, "learning_rate": 1.3771253359771818e-05, "entropy": 0.1466932299081236, "num_tokens": 55782178.0, "mean_token_accuracy": 0.9848661877214908, "epoch": 12.544, "step": 7840 }, { "loss": 0.0888, "grad_norm": 0.6285877227783203, "learning_rate": 1.3596802709432466e-05, "entropy": 0.14846890671178697, "num_tokens": 55852392.0, "mean_token_accuracy": 0.9842716619372368, "epoch": 12.56, "step": 7850 }, { "loss": 0.0909, "grad_norm": 0.46850839257240295, "learning_rate": 1.3423383429686954e-05, "entropy": 0.15417468920350075, "num_tokens": 55923955.0, "mean_token_accuracy": 0.9833245389163494, "epoch": 12.576, "step": 7860 }, { "loss": 0.0867, "grad_norm": 0.6758456230163574, "learning_rate": 1.3250997590592251e-05, "entropy": 0.150390401808545, "num_tokens": 55996428.0, "mean_token_accuracy": 0.984145438671112, "epoch": 12.592, "step": 7870 }, { "loss": 0.0929, "grad_norm": 0.49797308444976807, "learning_rate": 1.3079647249869554e-05, "entropy": 0.15639365501701832, "num_tokens": 56066338.0, "mean_token_accuracy": 0.9833997257053853, "epoch": 12.608, "step": 7880 }, { "loss": 0.0909, "grad_norm": 0.3179808557033539, "learning_rate": 1.2909334452879518e-05, "entropy": 0.15142184090800584, "num_tokens": 56136312.0, "mean_token_accuracy": 0.9837806254625321, "epoch": 12.624, "step": 7890 }, { "loss": 0.0916, "grad_norm": 0.5887900590896606, "learning_rate": 1.2740061232597977e-05, "entropy": 0.1574403473176062, "num_tokens": 56206595.0, "mean_token_accuracy": 0.9830654293298722, "epoch": 12.64, "step": 7900 }, { "loss": 0.0884, "grad_norm": 0.32210978865623474, "learning_rate": 1.2571829609591568e-05, "entropy": 0.14550409968942404, "num_tokens": 56278261.0, "mean_token_accuracy": 0.9839170910418034, "epoch": 12.656, "step": 7910 }, { "loss": 0.1161, "grad_norm": 0.3749602138996124, "learning_rate": 1.2404641591993772e-05, "entropy": 0.1843244494870305, "num_tokens": 56348988.0, "mean_token_accuracy": 0.979022791236639, "epoch": 12.672, "step": 7920 }, { "loss": 0.0913, "grad_norm": 0.5172518491744995, "learning_rate": 1.2238499175480788e-05, "entropy": 0.1558773159980774, "num_tokens": 56419596.0, "mean_token_accuracy": 0.9833558730781078, "epoch": 12.688, "step": 7930 }, { "loss": 0.09, "grad_norm": 0.6708750128746033, "learning_rate": 1.2073404343247752e-05, "entropy": 0.15603746068663896, "num_tokens": 56491743.0, "mean_token_accuracy": 0.9838562846183777, "epoch": 12.704, "step": 7940 }, { "loss": 0.0962, "grad_norm": 0.39717695116996765, "learning_rate": 1.1909359065985126e-05, "entropy": 0.16046544937416912, "num_tokens": 56560538.0, "mean_token_accuracy": 0.9827483147382736, "epoch": 12.72, "step": 7950 }, { "loss": 0.0966, "grad_norm": 0.5046579241752625, "learning_rate": 1.174636530185509e-05, "entropy": 0.1627069511450827, "num_tokens": 56632411.0, "mean_token_accuracy": 0.9824646562337875, "epoch": 12.736, "step": 7960 }, { "loss": 0.0899, "grad_norm": 0.5642845630645752, "learning_rate": 1.1584424996468268e-05, "entropy": 0.15106039261445403, "num_tokens": 56704111.0, "mean_token_accuracy": 0.9835800848901272, "epoch": 12.752, "step": 7970 }, { "loss": 0.0867, "grad_norm": 0.5109546780586243, "learning_rate": 1.1423540082860374e-05, "entropy": 0.1479904418811202, "num_tokens": 56777815.0, "mean_token_accuracy": 0.9846726596355438, "epoch": 12.768, "step": 7980 }, { "loss": 0.0893, "grad_norm": 0.6029806733131409, "learning_rate": 1.1263712481469258e-05, "entropy": 0.1482215730473399, "num_tokens": 56850192.0, "mean_token_accuracy": 0.9838159434497357, "epoch": 12.784, "step": 7990 }, { "loss": 0.0863, "grad_norm": 0.4088634252548218, "learning_rate": 1.1104944100111891e-05, "entropy": 0.14582721670158208, "num_tokens": 56923781.0, "mean_token_accuracy": 0.9851084463298321, "epoch": 12.8, "step": 8000 }, { "loss": 0.0906, "grad_norm": 0.5347424745559692, "learning_rate": 1.0947236833961661e-05, "entropy": 0.1538609214592725, "num_tokens": 56996597.0, "mean_token_accuracy": 0.9839540295302868, "epoch": 12.816, "step": 8010 }, { "loss": 0.0972, "grad_norm": 0.45003044605255127, "learning_rate": 1.0790592565525758e-05, "entropy": 0.1652635984122753, "num_tokens": 57063422.0, "mean_token_accuracy": 0.9819845616817474, "epoch": 12.832, "step": 8020 }, { "loss": 0.087, "grad_norm": 0.4910476803779602, "learning_rate": 1.0635013164622598e-05, "entropy": 0.14883292792364955, "num_tokens": 57135380.0, "mean_token_accuracy": 0.9844342976808548, "epoch": 12.848, "step": 8030 }, { "loss": 0.0948, "grad_norm": 0.5508122444152832, "learning_rate": 1.0480500488359601e-05, "entropy": 0.1571507359854877, "num_tokens": 57206337.0, "mean_token_accuracy": 0.9828753866255283, "epoch": 12.864, "step": 8040 }, { "loss": 0.0916, "grad_norm": 0.4723197817802429, "learning_rate": 1.0327056381110988e-05, "entropy": 0.15388506529852747, "num_tokens": 57277643.0, "mean_token_accuracy": 0.983082927018404, "epoch": 12.88, "step": 8050 }, { "loss": 0.0927, "grad_norm": 0.40694719552993774, "learning_rate": 1.0174682674495827e-05, "entropy": 0.1582325655966997, "num_tokens": 57350546.0, "mean_token_accuracy": 0.9832557134330273, "epoch": 12.896, "step": 8060 }, { "loss": 0.1033, "grad_norm": 0.5031518340110779, "learning_rate": 1.002338118735604e-05, "entropy": 0.17433252977207303, "num_tokens": 57418939.0, "mean_token_accuracy": 0.9805484272539615, "epoch": 12.912, "step": 8070 }, { "loss": 0.0986, "grad_norm": 0.6663874387741089, "learning_rate": 9.873153725734818e-06, "entropy": 0.1617913362570107, "num_tokens": 57487769.0, "mean_token_accuracy": 0.9820040471851825, "epoch": 12.928, "step": 8080 }, { "loss": 0.0926, "grad_norm": 0.4439689815044403, "learning_rate": 9.724002082854977e-06, "entropy": 0.15975127713754772, "num_tokens": 57558302.0, "mean_token_accuracy": 0.9833142854273319, "epoch": 12.943999999999999, "step": 8090 }, { "loss": 0.0915, "grad_norm": 0.6059590578079224, "learning_rate": 9.575928039097593e-06, "entropy": 0.1570997008122504, "num_tokens": 57631135.0, "mean_token_accuracy": 0.9839120388031006, "epoch": 12.96, "step": 8100 }, { "loss": 0.1016, "grad_norm": 0.6217557191848755, "learning_rate": 9.428933361980796e-06, "entropy": 0.17322367960587143, "num_tokens": 57700880.0, "mean_token_accuracy": 0.9811670832335949, "epoch": 12.975999999999999, "step": 8110 }, { "loss": 0.0888, "grad_norm": 0.36989614367485046, "learning_rate": 9.283019806138582e-06, "entropy": 0.1518563913181424, "num_tokens": 57772887.0, "mean_token_accuracy": 0.9836678758263588, "epoch": 12.992, "step": 8120 }, { "eval_loss": 3.258859872817993, "eval_runtime": 53.7536, "eval_samples_per_second": 4.186, "eval_steps_per_second": 4.186, "eval_entropy": 0.7344758858945635, "eval_num_tokens": 57807386.0, "eval_mean_token_accuracy": 0.6030576153596242, "epoch": 13.0, "step": 8125 }, { "loss": 0.0893, "grad_norm": 0.3956635296344757, "learning_rate": 9.138189113299877e-06, "entropy": 0.15318279126659035, "num_tokens": 57842018.0, "mean_token_accuracy": 0.983634638786316, "epoch": 13.008, "step": 8130 }, { "loss": 0.0836, "grad_norm": 0.45850926637649536, "learning_rate": 8.994443012267816e-06, "entropy": 0.14699933342635632, "num_tokens": 57911997.0, "mean_token_accuracy": 0.9849312603473663, "epoch": 13.024, "step": 8140 }, { "loss": 0.0771, "grad_norm": 0.36129212379455566, "learning_rate": 8.85178321889908e-06, "entropy": 0.1307014306075871, "num_tokens": 57985706.0, "mean_token_accuracy": 0.9866759546101094, "epoch": 13.04, "step": 8150 }, { "loss": 0.0839, "grad_norm": 0.35006362199783325, "learning_rate": 8.710211436083371e-06, "entropy": 0.13978761876933277, "num_tokens": 58057192.0, "mean_token_accuracy": 0.9860057838261127, "epoch": 13.056, "step": 8160 }, { "loss": 0.0892, "grad_norm": 0.5808591842651367, "learning_rate": 8.569729353723122e-06, "entropy": 0.15359092745929956, "num_tokens": 58128509.0, "mean_token_accuracy": 0.9847656071186066, "epoch": 13.072, "step": 8170 }, { "loss": 0.0838, "grad_norm": 0.4145296514034271, "learning_rate": 8.430338648713332e-06, "entropy": 0.13916668025776743, "num_tokens": 58199306.0, "mean_token_accuracy": 0.9856627605855465, "epoch": 13.088, "step": 8180 }, { "loss": 0.084, "grad_norm": 0.39784175157546997, "learning_rate": 8.2920409849215e-06, "entropy": 0.14055346054956316, "num_tokens": 58270871.0, "mean_token_accuracy": 0.9857617944478989, "epoch": 13.104, "step": 8190 }, { "loss": 0.077, "grad_norm": 0.295294851064682, "learning_rate": 8.154838013167865e-06, "entropy": 0.12931434088386595, "num_tokens": 58344465.0, "mean_token_accuracy": 0.9868112072348595, "epoch": 13.12, "step": 8200 }, { "loss": 0.0904, "grad_norm": 0.41991716623306274, "learning_rate": 8.01873137120559e-06, "entropy": 0.15194443510845304, "num_tokens": 58412687.0, "mean_token_accuracy": 0.9844192676246166, "epoch": 13.136, "step": 8210 }, { "loss": 0.0845, "grad_norm": 0.5284660458564758, "learning_rate": 7.883722683701267e-06, "entropy": 0.1475465290248394, "num_tokens": 58481271.0, "mean_token_accuracy": 0.9844477005302906, "epoch": 13.152, "step": 8220 }, { "loss": 0.085, "grad_norm": 0.6007907390594482, "learning_rate": 7.74981356221548e-06, "entropy": 0.14303330830298364, "num_tokens": 58551776.0, "mean_token_accuracy": 0.9851215675473213, "epoch": 13.168, "step": 8230 }, { "loss": 0.0878, "grad_norm": 0.5083174109458923, "learning_rate": 7.61700560518368e-06, "entropy": 0.15307229589670895, "num_tokens": 58623249.0, "mean_token_accuracy": 0.9846013285219669, "epoch": 13.184, "step": 8240 }, { "loss": 0.0806, "grad_norm": 0.4680114686489105, "learning_rate": 7.485300397896999e-06, "entropy": 0.13499779351986946, "num_tokens": 58696481.0, "mean_token_accuracy": 0.9856531128287316, "epoch": 13.2, "step": 8250 }, { "loss": 0.0849, "grad_norm": 0.3552989065647125, "learning_rate": 7.354699512483331e-06, "entropy": 0.14273298159241676, "num_tokens": 58767975.0, "mean_token_accuracy": 0.9851688630878925, "epoch": 13.216, "step": 8260 }, { "loss": 0.0807, "grad_norm": 0.3458268344402313, "learning_rate": 7.2252045078885945e-06, "entropy": 0.13693878026679157, "num_tokens": 58841087.0, "mean_token_accuracy": 0.9858393445611, "epoch": 13.232, "step": 8270 }, { "loss": 0.0836, "grad_norm": 0.3543095886707306, "learning_rate": 7.09681692985813e-06, "entropy": 0.14145282409153878, "num_tokens": 58912626.0, "mean_token_accuracy": 0.9853307008743286, "epoch": 13.248, "step": 8280 }, { "loss": 0.0863, "grad_norm": 0.41875922679901123, "learning_rate": 6.969538310918244e-06, "entropy": 0.1488360207527876, "num_tokens": 58982393.0, "mean_token_accuracy": 0.9842502683401108, "epoch": 13.264, "step": 8290 }, { "loss": 0.0814, "grad_norm": 0.29092368483543396, "learning_rate": 6.843370170357932e-06, "entropy": 0.14104822371155024, "num_tokens": 59054194.0, "mean_token_accuracy": 0.985070063918829, "epoch": 13.28, "step": 8300 }, { "loss": 0.088, "grad_norm": 0.5246424674987793, "learning_rate": 6.71831401421067e-06, "entropy": 0.15178719405084848, "num_tokens": 59124190.0, "mean_token_accuracy": 0.9835954040288926, "epoch": 13.296, "step": 8310 }, { "loss": 0.0916, "grad_norm": 0.27585741877555847, "learning_rate": 6.594371335236538e-06, "entropy": 0.15158836161717773, "num_tokens": 59195426.0, "mean_token_accuracy": 0.9842022076249123, "epoch": 13.312, "step": 8320 }, { "loss": 0.0845, "grad_norm": 0.5533126592636108, "learning_rate": 6.471543612904319e-06, "entropy": 0.14412691169418396, "num_tokens": 59267922.0, "mean_token_accuracy": 0.9854904592037201, "epoch": 13.328, "step": 8330 }, { "loss": 0.0777, "grad_norm": 0.5057433843612671, "learning_rate": 6.3498323133738934e-06, "entropy": 0.132695687469095, "num_tokens": 59341336.0, "mean_token_accuracy": 0.986610846966505, "epoch": 13.344, "step": 8340 }, { "loss": 0.0841, "grad_norm": 0.3495892882347107, "learning_rate": 6.229238889478717e-06, "entropy": 0.14357217438519002, "num_tokens": 59412308.0, "mean_token_accuracy": 0.9855480268597603, "epoch": 13.36, "step": 8350 }, { "loss": 0.0804, "grad_norm": 0.4515880048274994, "learning_rate": 6.109764780708482e-06, "entropy": 0.13829152155667543, "num_tokens": 59486144.0, "mean_token_accuracy": 0.9865356139838696, "epoch": 13.376, "step": 8360 }, { "loss": 0.0864, "grad_norm": 0.4070594906806946, "learning_rate": 5.991411413191894e-06, "entropy": 0.14745063786394894, "num_tokens": 59552534.0, "mean_token_accuracy": 0.9839030914008617, "epoch": 13.392, "step": 8370 }, { "loss": 0.084, "grad_norm": 0.4719223380088806, "learning_rate": 5.874180199679735e-06, "entropy": 0.14641958251595497, "num_tokens": 59623965.0, "mean_token_accuracy": 0.985252657532692, "epoch": 13.408, "step": 8380 }, { "loss": 0.0839, "grad_norm": 0.48837655782699585, "learning_rate": 5.7580725395279366e-06, "entropy": 0.14791806330904364, "num_tokens": 59697026.0, "mean_token_accuracy": 0.9855681844055653, "epoch": 13.424, "step": 8390 }, { "loss": 0.0901, "grad_norm": 0.3067860007286072, "learning_rate": 5.643089818680891e-06, "entropy": 0.1584520062431693, "num_tokens": 59766671.0, "mean_token_accuracy": 0.9835608281195164, "epoch": 13.44, "step": 8400 }, { "loss": 0.0913, "grad_norm": 0.4301672875881195, "learning_rate": 5.5292334096548885e-06, "entropy": 0.15589946964755655, "num_tokens": 59834583.0, "mean_token_accuracy": 0.983762638270855, "epoch": 13.456, "step": 8410 }, { "loss": 0.082, "grad_norm": 0.4168688952922821, "learning_rate": 5.416504671521772e-06, "entropy": 0.13800524044781923, "num_tokens": 59907537.0, "mean_token_accuracy": 0.9856640852987766, "epoch": 13.472, "step": 8420 }, { "loss": 0.0881, "grad_norm": 0.5088773369789124, "learning_rate": 5.304904949892697e-06, "entropy": 0.1485220989678055, "num_tokens": 59974803.0, "mean_token_accuracy": 0.9841576054692268, "epoch": 13.488, "step": 8430 }, { "loss": 0.0868, "grad_norm": 0.35078680515289307, "learning_rate": 5.194435576902057e-06, "entropy": 0.1510665776208043, "num_tokens": 60042750.0, "mean_token_accuracy": 0.983994996547699, "epoch": 13.504, "step": 8440 }, { "loss": 0.0889, "grad_norm": 0.37901708483695984, "learning_rate": 5.085097871191591e-06, "entropy": 0.14690761268138885, "num_tokens": 60112064.0, "mean_token_accuracy": 0.9844422161579132, "epoch": 13.52, "step": 8450 }, { "loss": 0.0822, "grad_norm": 0.2631765902042389, "learning_rate": 4.976893137894645e-06, "entropy": 0.13767907847650349, "num_tokens": 60182925.0, "mean_token_accuracy": 0.9856643788516521, "epoch": 13.536, "step": 8460 }, { "loss": 0.0777, "grad_norm": 0.28977149724960327, "learning_rate": 4.869822668620627e-06, "entropy": 0.1344706249423325, "num_tokens": 60258365.0, "mean_token_accuracy": 0.9867749296128749, "epoch": 13.552, "step": 8470 }, { "loss": 0.0812, "grad_norm": 0.2745435833930969, "learning_rate": 4.763887741439499e-06, "entropy": 0.13789760121144354, "num_tokens": 60330883.0, "mean_token_accuracy": 0.985636192560196, "epoch": 13.568, "step": 8480 }, { "loss": 0.0852, "grad_norm": 0.4592578411102295, "learning_rate": 4.659089620866619e-06, "entropy": 0.1459613512735814, "num_tokens": 60403353.0, "mean_token_accuracy": 0.9852610550820827, "epoch": 13.584, "step": 8490 }, { "loss": 0.0849, "grad_norm": 0.35601937770843506, "learning_rate": 4.55542955784759e-06, "entropy": 0.14616650477983056, "num_tokens": 60474485.0, "mean_token_accuracy": 0.9850872553884983, "epoch": 13.6, "step": 8500 }, { "loss": 0.0861, "grad_norm": 0.31318384408950806, "learning_rate": 4.452908789743337e-06, "entropy": 0.14979282016865908, "num_tokens": 60544585.0, "mean_token_accuracy": 0.9846966825425625, "epoch": 13.616, "step": 8510 }, { "loss": 0.0864, "grad_norm": 0.33188915252685547, "learning_rate": 4.3515285403153524e-06, "entropy": 0.1473554872907698, "num_tokens": 60615760.0, "mean_token_accuracy": 0.9849355146288872, "epoch": 13.632, "step": 8520 }, { "loss": 0.0839, "grad_norm": 0.42579329013824463, "learning_rate": 4.251290019711085e-06, "entropy": 0.1439296425320208, "num_tokens": 60685514.0, "mean_token_accuracy": 0.9848736315965653, "epoch": 13.648, "step": 8530 }, { "loss": 0.0854, "grad_norm": 0.5199816823005676, "learning_rate": 4.152194424449485e-06, "entropy": 0.14588550264015793, "num_tokens": 60755063.0, "mean_token_accuracy": 0.9847184836864471, "epoch": 13.664, "step": 8540 }, { "loss": 0.0834, "grad_norm": 0.3129844069480896, "learning_rate": 4.054242937406694e-06, "entropy": 0.13989232634194196, "num_tokens": 60826080.0, "mean_token_accuracy": 0.9851528979837895, "epoch": 13.68, "step": 8550 }, { "loss": 0.0823, "grad_norm": 0.42803746461868286, "learning_rate": 3.957436727802011e-06, "entropy": 0.1412037812639028, "num_tokens": 60897621.0, "mean_token_accuracy": 0.9849915988743305, "epoch": 13.696, "step": 8560 }, { "loss": 0.0833, "grad_norm": 0.4168819487094879, "learning_rate": 3.8617769511838264e-06, "entropy": 0.1383406611159444, "num_tokens": 60968319.0, "mean_token_accuracy": 0.9854473106563091, "epoch": 13.712, "step": 8570 }, { "loss": 0.0789, "grad_norm": 0.429228812456131, "learning_rate": 3.767264749415933e-06, "entropy": 0.13268057461827992, "num_tokens": 61042147.0, "mean_token_accuracy": 0.9869371220469475, "epoch": 13.728, "step": 8580 }, { "loss": 0.0952, "grad_norm": 0.4407985806465149, "learning_rate": 3.673901250663825e-06, "entropy": 0.16439469140022994, "num_tokens": 61112515.0, "mean_token_accuracy": 0.9832316905260086, "epoch": 13.744, "step": 8590 }, { "loss": 0.0824, "grad_norm": 0.33647477626800537, "learning_rate": 3.5816875693812314e-06, "entropy": 0.138479127548635, "num_tokens": 61186127.0, "mean_token_accuracy": 0.9852566041052342, "epoch": 13.76, "step": 8600 }, { "loss": 0.085, "grad_norm": 0.3877672255039215, "learning_rate": 3.4906248062968606e-06, "entropy": 0.14554169932380318, "num_tokens": 61257830.0, "mean_token_accuracy": 0.9854347221553326, "epoch": 13.776, "step": 8610 }, { "loss": 0.0816, "grad_norm": 0.3394245505332947, "learning_rate": 3.4007140484012544e-06, "entropy": 0.1361513073556125, "num_tokens": 61328785.0, "mean_token_accuracy": 0.9855215929448604, "epoch": 13.792, "step": 8620 }, { "loss": 0.0827, "grad_norm": 0.38023796677589417, "learning_rate": 3.311956368933733e-06, "entropy": 0.13622878519818188, "num_tokens": 61399434.0, "mean_token_accuracy": 0.9853359252214432, "epoch": 13.808, "step": 8630 }, { "loss": 0.0802, "grad_norm": 0.4155137240886688, "learning_rate": 3.2243528273697033e-06, "entropy": 0.1393348524812609, "num_tokens": 61472721.0, "mean_token_accuracy": 0.9860965348780155, "epoch": 13.824, "step": 8640 }, { "loss": 0.0819, "grad_norm": 0.43143391609191895, "learning_rate": 3.137904469407915e-06, "entropy": 0.14185655280016363, "num_tokens": 61546458.0, "mean_token_accuracy": 0.9856943853199482, "epoch": 13.84, "step": 8650 }, { "loss": 0.0875, "grad_norm": 0.376660019159317, "learning_rate": 3.0526123269580377e-06, "entropy": 0.1479565493762493, "num_tokens": 61615826.0, "mean_token_accuracy": 0.9851415909826755, "epoch": 13.856, "step": 8660 }, { "loss": 0.0854, "grad_norm": 0.33041876554489136, "learning_rate": 2.968477418128324e-06, "entropy": 0.15115371569991112, "num_tokens": 61687450.0, "mean_token_accuracy": 0.9847798742353916, "epoch": 13.872, "step": 8670 }, { "loss": 0.0863, "grad_norm": 0.36503055691719055, "learning_rate": 2.885500747213432e-06, "entropy": 0.14590244297869503, "num_tokens": 61757762.0, "mean_token_accuracy": 0.9847077861428261, "epoch": 13.888, "step": 8680 }, { "loss": 0.0825, "grad_norm": 0.5600667595863342, "learning_rate": 2.8036833046824917e-06, "entropy": 0.14019269444979726, "num_tokens": 61828676.0, "mean_token_accuracy": 0.9854286856949329, "epoch": 13.904, "step": 8690 }, { "loss": 0.0864, "grad_norm": 0.3820892870426178, "learning_rate": 2.7230260671672338e-06, "entropy": 0.14650002741254867, "num_tokens": 61901202.0, "mean_token_accuracy": 0.9852773793041706, "epoch": 13.92, "step": 8700 }, { "loss": 0.0846, "grad_norm": 0.3953131139278412, "learning_rate": 2.6435299974503446e-06, "entropy": 0.14585595205426216, "num_tokens": 61971139.0, "mean_token_accuracy": 0.9847275294363499, "epoch": 13.936, "step": 8710 }, { "loss": 0.0814, "grad_norm": 0.3729381561279297, "learning_rate": 2.565196044453988e-06, "entropy": 0.13664818378165364, "num_tokens": 62042762.0, "mean_token_accuracy": 0.9858971312642097, "epoch": 13.952, "step": 8720 }, { "loss": 0.0916, "grad_norm": 0.45835524797439575, "learning_rate": 2.4880251432284786e-06, "entropy": 0.15183540098369122, "num_tokens": 62110015.0, "mean_token_accuracy": 0.9840091928839684, "epoch": 13.968, "step": 8730 }, { "loss": 0.0912, "grad_norm": 0.3521203100681305, "learning_rate": 2.41201821494107e-06, "entropy": 0.15479198480024933, "num_tokens": 62179382.0, "mean_token_accuracy": 0.9837841592729092, "epoch": 13.984, "step": 8740 }, { "loss": 0.1005, "grad_norm": 0.36912569403648376, "learning_rate": 2.3446079264931033e-06, "entropy": 0.14969166042283177, "num_tokens": 62254108.0, "mean_token_accuracy": 0.9823833301663398, "epoch": 14.0, "step": 8750 }, { "eval_loss": 3.3373539447784424, "eval_runtime": 53.906, "eval_samples_per_second": 4.174, "eval_steps_per_second": 4.174, "eval_entropy": 0.7110330586963229, "eval_num_tokens": 62254108.0, "eval_mean_token_accuracy": 0.6029975185129377, "epoch": 14.0, "step": 8750 }, { "loss": 0.0824, "grad_norm": 0.3256596624851227, "learning_rate": 2.2708150348338176e-06, "entropy": 0.14078810680657625, "num_tokens": 62324494.0, "mean_token_accuracy": 0.9860088773071766, "epoch": 14.016, "step": 8760 }, { "loss": 0.078, "grad_norm": 0.3022592067718506, "learning_rate": 2.1981887088884377e-06, "entropy": 0.13595874328166246, "num_tokens": 62397359.0, "mean_token_accuracy": 0.9867612212896347, "epoch": 14.032, "step": 8770 }, { "loss": 0.0769, "grad_norm": 0.35886919498443604, "learning_rate": 2.1267298155769334e-06, "entropy": 0.13257453134283423, "num_tokens": 62472039.0, "mean_token_accuracy": 0.9867639690637589, "epoch": 14.048, "step": 8780 }, { "loss": 0.0823, "grad_norm": 0.3873071074485779, "learning_rate": 2.0564392078839644e-06, "entropy": 0.13644878747873007, "num_tokens": 62544191.0, "mean_token_accuracy": 0.9860813118517399, "epoch": 14.064, "step": 8790 }, { "loss": 0.075, "grad_norm": 0.33495932817459106, "learning_rate": 1.9873177248486698e-06, "entropy": 0.12605555448681116, "num_tokens": 62620707.0, "mean_token_accuracy": 0.9878044553101063, "epoch": 14.08, "step": 8800 }, { "loss": 0.079, "grad_norm": 0.31643855571746826, "learning_rate": 1.919366191554706e-06, "entropy": 0.13419887670315803, "num_tokens": 62692028.0, "mean_token_accuracy": 0.9861233815550804, "epoch": 14.096, "step": 8810 }, { "loss": 0.0797, "grad_norm": 0.3743322193622589, "learning_rate": 1.8525854191203562e-06, "entropy": 0.1326361397281289, "num_tokens": 62765027.0, "mean_token_accuracy": 0.9870174407958985, "epoch": 14.112, "step": 8820 }, { "loss": 0.0842, "grad_norm": 0.34651973843574524, "learning_rate": 1.7869762046888727e-06, "entropy": 0.1462113535962999, "num_tokens": 62834572.0, "mean_token_accuracy": 0.9853726424276829, "epoch": 14.128, "step": 8830 }, { "loss": 0.0848, "grad_norm": 0.33809664845466614, "learning_rate": 1.7225393314189265e-06, "entropy": 0.14153131432831287, "num_tokens": 62905600.0, "mean_token_accuracy": 0.9859179511666298, "epoch": 14.144, "step": 8840 }, { "loss": 0.0794, "grad_norm": 0.326214462518692, "learning_rate": 1.6592755684753047e-06, "entropy": 0.13309747725725174, "num_tokens": 62977411.0, "mean_token_accuracy": 0.9859074726700783, "epoch": 14.16, "step": 8850 }, { "loss": 0.0823, "grad_norm": 0.5166251063346863, "learning_rate": 1.5971856710196964e-06, "entropy": 0.1391609358601272, "num_tokens": 63048503.0, "mean_token_accuracy": 0.9861260391771793, "epoch": 14.176, "step": 8860 }, { "loss": 0.0816, "grad_norm": 0.3288152515888214, "learning_rate": 1.5362703802016986e-06, "entropy": 0.14618434961885213, "num_tokens": 63118695.0, "mean_token_accuracy": 0.9852051272988319, "epoch": 14.192, "step": 8870 }, { "loss": 0.0836, "grad_norm": 0.33804652094841003, "learning_rate": 1.4765304231499578e-06, "entropy": 0.1426199879962951, "num_tokens": 63190223.0, "mean_token_accuracy": 0.9854112230241299, "epoch": 14.208, "step": 8880 }, { "loss": 0.0835, "grad_norm": 0.2655903100967407, "learning_rate": 1.4179665129634867e-06, "entropy": 0.14165451806038618, "num_tokens": 63260773.0, "mean_token_accuracy": 0.9858719050884247, "epoch": 14.224, "step": 8890 }, { "loss": 0.0851, "grad_norm": 0.4184131622314453, "learning_rate": 1.3605793487031614e-06, "entropy": 0.14597317324951292, "num_tokens": 63328881.0, "mean_token_accuracy": 0.9846769742667675, "epoch": 14.24, "step": 8900 }, { "loss": 0.0793, "grad_norm": 0.36559775471687317, "learning_rate": 1.3043696153833717e-06, "entropy": 0.13301197844557464, "num_tokens": 63400049.0, "mean_token_accuracy": 0.9861026532948017, "epoch": 14.256, "step": 8910 }, { "loss": 0.0801, "grad_norm": 0.326408714056015, "learning_rate": 1.2493379839638497e-06, "entropy": 0.13472974812611938, "num_tokens": 63471600.0, "mean_token_accuracy": 0.9860430985689164, "epoch": 14.272, "step": 8920 }, { "loss": 0.0831, "grad_norm": 0.3712698817253113, "learning_rate": 1.1954851113416655e-06, "entropy": 0.1398371377028525, "num_tokens": 63541510.0, "mean_token_accuracy": 0.9853140532970428, "epoch": 14.288, "step": 8930 }, { "loss": 0.079, "grad_norm": 0.2879469394683838, "learning_rate": 1.1428116403433554e-06, "entropy": 0.13144627837464212, "num_tokens": 63614003.0, "mean_token_accuracy": 0.9868641257286072, "epoch": 14.304, "step": 8940 }, { "loss": 0.0801, "grad_norm": 0.2998805046081543, "learning_rate": 1.091318199717284e-06, "entropy": 0.13079835711978377, "num_tokens": 63683464.0, "mean_token_accuracy": 0.9863437466323376, "epoch": 14.32, "step": 8950 }, { "loss": 0.0821, "grad_norm": 0.33542340993881226, "learning_rate": 1.0410054041261386e-06, "entropy": 0.14538228292949498, "num_tokens": 63753602.0, "mean_token_accuracy": 0.9852630242705345, "epoch": 14.336, "step": 8960 }, { "loss": 0.0877, "grad_norm": 0.2515588402748108, "learning_rate": 9.918738541395578e-07, "entropy": 0.1503792844712734, "num_tokens": 63823762.0, "mean_token_accuracy": 0.984683419764042, "epoch": 14.352, "step": 8970 }, { "loss": 0.0801, "grad_norm": 0.35469743609428406, "learning_rate": 9.439241362270145e-07, "entropy": 0.13564730221405624, "num_tokens": 63895228.0, "mean_token_accuracy": 0.9862391471862793, "epoch": 14.368, "step": 8980 }, { "loss": 0.0818, "grad_norm": 0.40704503655433655, "learning_rate": 8.971568227507443e-07, "entropy": 0.1361823608633131, "num_tokens": 63967886.0, "mean_token_accuracy": 0.9864058203995227, "epoch": 14.384, "step": 8990 }, { "loss": 0.0818, "grad_norm": 0.2781176269054413, "learning_rate": 8.515724719589835e-07, "entropy": 0.1430083038751036, "num_tokens": 64037004.0, "mean_token_accuracy": 0.9854664370417595, "epoch": 14.4, "step": 9000 }, { "loss": 0.0823, "grad_norm": 0.2995631694793701, "learning_rate": 8.071716279792752e-07, "entropy": 0.14025755506008863, "num_tokens": 64107241.0, "mean_token_accuracy": 0.9853468671441078, "epoch": 14.416, "step": 9010 }, { "loss": 0.0817, "grad_norm": 0.26669493317604065, "learning_rate": 7.639548208119518e-07, "entropy": 0.13949991529807448, "num_tokens": 64177823.0, "mean_token_accuracy": 0.9850617177784443, "epoch": 14.432, "step": 9020 }, { "loss": 0.0762, "grad_norm": 0.2743082642555237, "learning_rate": 7.219225663238738e-07, "entropy": 0.12856905492953957, "num_tokens": 64251483.0, "mean_token_accuracy": 0.9873432531952858, "epoch": 14.448, "step": 9030 }, { "loss": 0.0883, "grad_norm": 0.3424886167049408, "learning_rate": 6.81075366242201e-07, "entropy": 0.14962717141024767, "num_tokens": 64319911.0, "mean_token_accuracy": 0.9842343039810657, "epoch": 14.464, "step": 9040 }, { "loss": 0.1057, "grad_norm": 0.3176668882369995, "learning_rate": 6.414137081484195e-07, "entropy": 0.16617399225942792, "num_tokens": 64389745.0, "mean_token_accuracy": 0.981073135882616, "epoch": 14.48, "step": 9050 }, { "loss": 0.0812, "grad_norm": 0.3415464460849762, "learning_rate": 6.029380654725691e-07, "entropy": 0.13832679772749543, "num_tokens": 64459337.0, "mean_token_accuracy": 0.9859035983681679, "epoch": 14.496, "step": 9060 }, { "loss": 0.09, "grad_norm": 0.4113196134567261, "learning_rate": 5.656488974875474e-07, "entropy": 0.15629103165119887, "num_tokens": 64525713.0, "mean_token_accuracy": 0.9836672455072403, "epoch": 14.512, "step": 9070 }, { "loss": 0.0786, "grad_norm": 0.4851938784122467, "learning_rate": 5.29546649303625e-07, "entropy": 0.13431014753878118, "num_tokens": 64599334.0, "mean_token_accuracy": 0.9870093993842601, "epoch": 14.528, "step": 9080 }, { "loss": 0.0794, "grad_norm": 0.27760744094848633, "learning_rate": 4.946317518631616e-07, "entropy": 0.13762902580201625, "num_tokens": 64671381.0, "mean_token_accuracy": 0.9861853010952473, "epoch": 14.544, "step": 9090 }, { "loss": 0.083, "grad_norm": 0.3241269886493683, "learning_rate": 4.6090462193543183e-07, "entropy": 0.13744519925676285, "num_tokens": 64742515.0, "mean_token_accuracy": 0.9856053598225116, "epoch": 14.56, "step": 9100 }, { "loss": 0.0796, "grad_norm": 0.4395633339881897, "learning_rate": 4.2836566211168495e-07, "entropy": 0.1358569567091763, "num_tokens": 64814597.0, "mean_token_accuracy": 0.9866010494530201, "epoch": 14.576, "step": 9110 }, { "loss": 0.0822, "grad_norm": 0.315408855676651, "learning_rate": 3.9701526080029304e-07, "entropy": 0.1358681826852262, "num_tokens": 64886227.0, "mean_token_accuracy": 0.9860679797828198, "epoch": 14.592, "step": 9120 }, { "loss": 0.0825, "grad_norm": 0.39687207341194153, "learning_rate": 3.6685379222216597e-07, "entropy": 0.14297962440177797, "num_tokens": 64956465.0, "mean_token_accuracy": 0.9851189658045769, "epoch": 14.608, "step": 9130 }, { "loss": 0.0824, "grad_norm": 0.4238765239715576, "learning_rate": 3.378816164062326e-07, "entropy": 0.14531763247214258, "num_tokens": 65025996.0, "mean_token_accuracy": 0.9856591671705246, "epoch": 14.624, "step": 9140 }, { "loss": 0.0829, "grad_norm": 0.3173438012599945, "learning_rate": 3.1009907918518877e-07, "entropy": 0.1416176926344633, "num_tokens": 65094297.0, "mean_token_accuracy": 0.985331804305315, "epoch": 14.64, "step": 9150 }, { "loss": 0.0783, "grad_norm": 0.3642211854457855, "learning_rate": 2.8350651219134493e-07, "entropy": 0.13332451991736888, "num_tokens": 65165501.0, "mean_token_accuracy": 0.9864253029227257, "epoch": 14.656, "step": 9160 }, { "loss": 0.0841, "grad_norm": 0.4797379970550537, "learning_rate": 2.5810423285267394e-07, "entropy": 0.1461602859199047, "num_tokens": 65236170.0, "mean_token_accuracy": 0.9852578230202198, "epoch": 14.672, "step": 9170 }, { "loss": 0.0799, "grad_norm": 0.3281996548175812, "learning_rate": 2.3389254438901386e-07, "entropy": 0.13755081589333712, "num_tokens": 65308235.0, "mean_token_accuracy": 0.9860194839537144, "epoch": 14.688, "step": 9180 }, { "loss": 0.0852, "grad_norm": 0.3131387531757355, "learning_rate": 2.1087173580845997e-07, "entropy": 0.15002887872979045, "num_tokens": 65376792.0, "mean_token_accuracy": 0.9845238119363785, "epoch": 14.704, "step": 9190 }, { "loss": 0.0786, "grad_norm": 0.2663953900337219, "learning_rate": 1.8904208190392292e-07, "entropy": 0.13735268702730535, "num_tokens": 65448003.0, "mean_token_accuracy": 0.9854544185101985, "epoch": 14.72, "step": 9200 }, { "loss": 0.0805, "grad_norm": 0.3174247741699219, "learning_rate": 1.6840384324980917e-07, "entropy": 0.14249982926994562, "num_tokens": 65519635.0, "mean_token_accuracy": 0.9857505291700364, "epoch": 14.736, "step": 9210 }, { "loss": 0.0782, "grad_norm": 0.2849161624908447, "learning_rate": 1.48957266198968e-07, "entropy": 0.13141617053188384, "num_tokens": 65591805.0, "mean_token_accuracy": 0.9866070628166199, "epoch": 14.752, "step": 9220 }, { "loss": 0.0807, "grad_norm": 0.267977237701416, "learning_rate": 1.3070258287968262e-07, "entropy": 0.1389425835572183, "num_tokens": 65665052.0, "mean_token_accuracy": 0.9859765157103538, "epoch": 14.768, "step": 9230 }, { "loss": 0.0802, "grad_norm": 0.3816865384578705, "learning_rate": 1.1364001119298362e-07, "entropy": 0.12942854329012335, "num_tokens": 65736099.0, "mean_token_accuracy": 0.9864800289273262, "epoch": 14.784, "step": 9240 }, { "loss": 0.0868, "grad_norm": 0.3261967599391937, "learning_rate": 9.776975480995099e-08, "entropy": 0.1514810150489211, "num_tokens": 65806984.0, "mean_token_accuracy": 0.9844559788703918, "epoch": 14.8, "step": 9250 }, { "loss": 0.0802, "grad_norm": 0.3868998885154724, "learning_rate": 8.309200316937161e-08, "entropy": 0.13645899267867206, "num_tokens": 65878868.0, "mean_token_accuracy": 0.9861539304256439, "epoch": 14.816, "step": 9260 }, { "loss": 0.087, "grad_norm": 0.5193714499473572, "learning_rate": 6.960693147542996e-08, "entropy": 0.15040442268364132, "num_tokens": 65946974.0, "mean_token_accuracy": 0.9847999066114426, "epoch": 14.832, "step": 9270 }, { "loss": 0.081, "grad_norm": 0.41828638315200806, "learning_rate": 5.731470069562095e-08, "entropy": 0.1349887188989669, "num_tokens": 66019546.0, "mean_token_accuracy": 0.986227960139513, "epoch": 14.848, "step": 9280 }, { "loss": 0.0771, "grad_norm": 0.2689506411552429, "learning_rate": 4.621545755882917e-08, "entropy": 0.13028725036419928, "num_tokens": 66094492.0, "mean_token_accuracy": 0.9872563809156418, "epoch": 14.864, "step": 9290 }, { "loss": 0.0846, "grad_norm": 0.37698307633399963, "learning_rate": 3.6309334553596976e-08, "entropy": 0.14140119072981178, "num_tokens": 66163520.0, "mean_token_accuracy": 0.9849420949816704, "epoch": 14.88, "step": 9300 }, { "loss": 0.0795, "grad_norm": 0.3729828894138336, "learning_rate": 2.7596449926514666e-08, "entropy": 0.1363879413343966, "num_tokens": 66235688.0, "mean_token_accuracy": 0.9863544009625912, "epoch": 14.896, "step": 9310 }, { "loss": 0.0799, "grad_norm": 0.442262202501297, "learning_rate": 2.00769076808327e-08, "entropy": 0.13321693348698319, "num_tokens": 66308656.0, "mean_token_accuracy": 0.9862127579748631, "epoch": 14.912, "step": 9320 }, { "loss": 0.0773, "grad_norm": 0.3428111672401428, "learning_rate": 1.375079757519604e-08, "entropy": 0.12840729039162396, "num_tokens": 66382967.0, "mean_token_accuracy": 0.9874646171927453, "epoch": 14.928, "step": 9330 }, { "loss": 0.0811, "grad_norm": 0.36421242356300354, "learning_rate": 8.618195122611638e-09, "entropy": 0.13717543268576265, "num_tokens": 66453136.0, "mean_token_accuracy": 0.9860797896981239, "epoch": 14.943999999999999, "step": 9340 }, { "loss": 0.0801, "grad_norm": 0.3789207339286804, "learning_rate": 4.679161589504766e-09, "entropy": 0.1362857401371002, "num_tokens": 66524396.0, "mean_token_accuracy": 0.9860964626073837, "epoch": 14.96, "step": 9350 }, { "loss": 0.0822, "grad_norm": 0.43381136655807495, "learning_rate": 1.9337439949973502e-09, "entropy": 0.13832380007952452, "num_tokens": 66594811.0, "mean_token_accuracy": 0.9859011381864548, "epoch": 14.975999999999999, "step": 9360 }, { "loss": 0.0815, "grad_norm": 0.35892540216445923, "learning_rate": 3.819751103639746e-10, "entropy": 0.13265466946177185, "num_tokens": 66666120.0, "mean_token_accuracy": 0.986310750991106, "epoch": 14.992, "step": 9370 }, { "eval_loss": 3.3696420192718506, "eval_runtime": 53.84, "eval_samples_per_second": 4.179, "eval_steps_per_second": 4.179, "eval_entropy": 0.7039169147279527, "eval_num_tokens": 66700830.0, "eval_mean_token_accuracy": 0.6029711000124613, "epoch": 15.0, "step": 9375 }, { "train_runtime": 56280.2705, "train_samples_per_second": 1.333, "train_steps_per_second": 0.167, "total_flos": 1.1475377961488179e+18, "train_loss": 0.45864094121932986, "epoch": 15.0, "step": 9375 } ] }

Evaluation Results

{ "summary": { "total_examples": 50, "avg_base_similarity": 0.08991369167501816, "avg_finetuned_similarity": 0.09964362182335552, "avg_improvement": 0.009729930148337366, "max_improvement": 0.431105522260274, "min_improvement": -0.3163965503430967, "std_improvement": 0.0902134182249236, "examples_improved": 26, "examples_degraded": 23, "examples_unchanged": 1 }, "analyses": [ { "example_id": 0, "base_similarity": 0.26790450928381965, "finetuned_similarity": 0.3283173734610123, "improvement": 0.06041286417719266, "base_diff_score": 4, "finetuned_diff_score": 5, "base_has_code": true, "finetuned_has_code": true, "base_length": 1008, "finetuned_length": 962 }, { "example_id": 1, "base_similarity": 0.034241245136186774, "finetuned_similarity": 0.0025220680958385876, "improvement": -0.031719177040348184, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 1217, "finetuned_length": 1086 }, { "example_id": 2, "base_similarity": 0.003576654202568688, "finetuned_similarity": 0.02091503267973856, "improvement": 0.017338378477169875, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 5651, "finetuned_length": 5618 }, { "example_id": 3, "base_similarity": 0.05779334500875657, "finetuned_similarity": 0.07660455486542443, "improvement": 0.018811209856667864, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 1045, "finetuned_length": 813 }, { "example_id": 4, "base_similarity": 0.169921875, "finetuned_similarity": 0.601027397260274, "improvement": 0.431105522260274, "base_diff_score": 2, "finetuned_diff_score": 3, "base_has_code": true, "finetuned_has_code": true, "base_length": 540, "finetuned_length": 1113 }, { "example_id": 5, "base_similarity": 0.19347319347319347, "finetuned_similarity": 0.18126888217522658, "improvement": -0.012204311297966897, "base_diff_score": 3, "finetuned_diff_score": 3, "base_has_code": true, "finetuned_has_code": true, "base_length": 997, "finetuned_length": 827 }, { "example_id": 6, "base_similarity": 0.04270051933064051, "finetuned_similarity": 0.05339105339105339, "improvement": 0.010690534060412885, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 1233, "finetuned_length": 1269 }, { "example_id": 7, "base_similarity": 0.006165004533091568, "finetuned_similarity": 0.004700352526439483, "improvement": -0.0014646520066520854, "base_diff_score": 1, "finetuned_diff_score": 1, "base_has_code": true, "finetuned_has_code": true, "base_length": 5015, "finetuned_length": 4991 }, { "example_id": 8, "base_similarity": 0.2660443407234539, "finetuned_similarity": 0.2686230248306998, "improvement": 0.0025786841072458766, "base_diff_score": 0, "finetuned_diff_score": 1, "base_has_code": true, "finetuned_has_code": true, "base_length": 1254, "finetuned_length": 1169 }, { "example_id": 9, "base_similarity": 0.0, "finetuned_similarity": 0.028701891715590344, "improvement": 0.028701891715590344, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 568, "finetuned_length": 1033 }, { "example_id": 10, "base_similarity": 0.08583106267029973, "finetuned_similarity": 0.10807204803202135, "improvement": 0.022240985361721616, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 968, "finetuned_length": 999 }, { "example_id": 11, "base_similarity": 0.059833795013850416, "finetuned_similarity": 0.01757469244288225, "improvement": -0.04225910257096817, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 1305, "finetuned_length": 1207 }, { "example_id": 12, "base_similarity": 0.02172601086300543, "finetuned_similarity": 0.03428571428571429, "improvement": 0.012559703422708856, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 1157, "finetuned_length": 1075 }, { "example_id": 13, "base_similarity": 0.11948051948051948, "finetuned_similarity": 0.24593128390596744, "improvement": 0.12645076442544795, "base_diff_score": 0, "finetuned_diff_score": 4, "base_has_code": true, "finetuned_has_code": true, "base_length": 1290, "finetuned_length": 1019 }, { "example_id": 14, "base_similarity": 0.07692307692307693, "finetuned_similarity": 0.0824270177447052, "improvement": 0.005503940821628278, "base_diff_score": 5, "finetuned_diff_score": 3, "base_has_code": true, "finetuned_has_code": true, "base_length": 878, "finetuned_length": 1247 }, { "example_id": 15, "base_similarity": 0.09772727272727273, "finetuned_similarity": 0.07557732680195942, "improvement": -0.022149945925313316, "base_diff_score": 3, "finetuned_diff_score": 4, "base_has_code": true, "finetuned_has_code": true, "base_length": 439, "finetuned_length": 929 }, { "example_id": 16, "base_similarity": 0.0317208564631245, "finetuned_similarity": 0.03015873015873016, "improvement": -0.0015621263043943436, "base_diff_score": 0, "finetuned_diff_score": 3, "base_has_code": true, "finetuned_has_code": true, "base_length": 1151, "finetuned_length": 1170 }, { "example_id": 17, "base_similarity": 0.13333333333333333, "finetuned_similarity": 0.13516896120150187, "improvement": 0.0018356278681685434, "base_diff_score": 5, "finetuned_diff_score": 4, "base_has_code": true, "finetuned_has_code": true, "base_length": 1045, "finetuned_length": 1098 }, { "example_id": 18, "base_similarity": 0.012232415902140673, "finetuned_similarity": 0.012232415902140673, "improvement": 0.0, "base_diff_score": 5, "finetuned_diff_score": 5, "base_has_code": true, "finetuned_has_code": true, "base_length": 5231, "finetuned_length": 4216 }, { "example_id": 19, "base_similarity": 0.270367054610564, "finetuned_similarity": 0.34252297410192145, "improvement": 0.07215591949135747, "base_diff_score": 4, "finetuned_diff_score": 4, "base_has_code": true, "finetuned_has_code": true, "base_length": 905, "finetuned_length": 1058 }, { "example_id": 20, "base_similarity": 0.021139166177334117, "finetuned_similarity": 0.020361990950226245, "improvement": -0.0007771752271078722, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 1351, "finetuned_length": 1268 }, { "example_id": 21, "base_similarity": 0.0, "finetuned_similarity": 0.10518174787316319, "improvement": 0.10518174787316319, "base_diff_score": 0, "finetuned_diff_score": 3, "base_has_code": true, "finetuned_has_code": true, "base_length": 886, "finetuned_length": 1034 }, { "example_id": 22, "base_similarity": 0.05575935436537051, "finetuned_similarity": 0.05099931082012405, "improvement": -0.004760043545246458, "base_diff_score": 3, "finetuned_diff_score": 3, "base_has_code": true, "finetuned_has_code": true, "base_length": 863, "finetuned_length": 951 }, { "example_id": 23, "base_similarity": 0.015315890236119975, "finetuned_similarity": 0.012972972972972972, "improvement": -0.0023429172631470024, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 4201, "finetuned_length": 5050 }, { "example_id": 24, "base_similarity": 0.033512866546977854, "finetuned_similarity": 0.024495677233429394, "improvement": -0.00901718931354846, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 1171, "finetuned_length": 1157 }, { "example_id": 25, "base_similarity": 0.13380782918149467, "finetuned_similarity": 0.0771586037966932, "improvement": -0.056649225384801466, "base_diff_score": 4, "finetuned_diff_score": 4, "base_has_code": true, "finetuned_has_code": true, "base_length": 905, "finetuned_length": 1133 }, { "example_id": 26, "base_similarity": 0.1459119496855346, "finetuned_similarity": 0.08541973490427099, "improvement": -0.0604922147812636, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 948, "finetuned_length": 943 }, { "example_id": 27, "base_similarity": 0.09552845528455285, "finetuned_similarity": 0.10167224080267559, "improvement": 0.006143785518122738, "base_diff_score": 3, "finetuned_diff_score": 4, "base_has_code": true, "finetuned_has_code": true, "base_length": 543, "finetuned_length": 995 }, { "example_id": 28, "base_similarity": 0.0891218872870249, "finetuned_similarity": 0.2590738423028786, "improvement": 0.1699519550158537, "base_diff_score": 3, "finetuned_diff_score": 5, "base_has_code": true, "finetuned_has_code": true, "base_length": 1026, "finetuned_length": 1098 }, { "example_id": 29, "base_similarity": 0.04787812840043525, "finetuned_similarity": 0.04472843450479233, "improvement": -0.003149693895642923, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 1223, "finetuned_length": 1132 }, { "example_id": 30, "base_similarity": 0.07042253521126761, "finetuned_similarity": 0.07003444316877153, "improvement": -0.00038809204249608265, "base_diff_score": 4, "finetuned_diff_score": 3, "base_has_code": true, "finetuned_has_code": true, "base_length": 1062, "finetuned_length": 1242 }, { "example_id": 31, "base_similarity": 0.03055229142185664, "finetuned_similarity": 0.044307692307692305, "improvement": 0.013755400885835666, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 1202, "finetuned_length": 1125 }, { "example_id": 32, "base_similarity": 0.06496062992125984, "finetuned_similarity": 0.07276507276507277, "improvement": 0.007804442843812931, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 1126, "finetuned_length": 1036 }, { "example_id": 33, "base_similarity": 0.005496281926931782, "finetuned_similarity": 0.005437390052774668, "improvement": -5.8891874157114034e-05, "base_diff_score": 1, "finetuned_diff_score": 1, "base_has_code": true, "finetuned_has_code": true, "base_length": 5686, "finetuned_length": 5753 }, { "example_id": 34, "base_similarity": 0.23157894736842105, "finetuned_similarity": 0.29791666666666666, "improvement": 0.06633771929824561, "base_diff_score": 3, "finetuned_diff_score": 3, "base_has_code": true, "finetuned_has_code": true, "base_length": 1275, "finetuned_length": 1015 }, { "example_id": 35, "base_similarity": 0.03099361896080219, "finetuned_similarity": 0.06860158311345646, "improvement": 0.03760796415265427, "base_diff_score": 0, "finetuned_diff_score": 4, "base_has_code": true, "finetuned_has_code": true, "base_length": 1031, "finetuned_length": 1016 }, { "example_id": 36, "base_similarity": 0.1111111111111111, "finetuned_similarity": 0.11812627291242363, "improvement": 0.007015161801312522, "base_diff_score": 4, "finetuned_diff_score": 5, "base_has_code": true, "finetuned_has_code": true, "base_length": 919, "finetuned_length": 774 }, { "example_id": 37, "base_similarity": 0.045190445448676564, "finetuned_similarity": 0.06318504190844616, "improvement": 0.0179945964597696, "base_diff_score": 3, "finetuned_diff_score": 4, "base_has_code": true, "finetuned_has_code": true, "base_length": 1049, "finetuned_length": 1051 }, { "example_id": 38, "base_similarity": 0.03481190342504211, "finetuned_similarity": 0.07787903893951947, "improvement": 0.04306713551447736, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 1281, "finetuned_length": 1185 }, { "example_id": 39, "base_similarity": 0.11209439528023599, "finetuned_similarity": 0.06110458284371328, "improvement": -0.05098981243652271, "base_diff_score": 3, "finetuned_diff_score": 3, "base_has_code": true, "finetuned_has_code": true, "base_length": 904, "finetuned_length": 968 }, { "example_id": 40, "base_similarity": 0.08125445473984319, "finetuned_similarity": 0.07138304652644997, "improvement": -0.009871408213393218, "base_diff_score": 3, "finetuned_diff_score": 3, "base_has_code": true, "finetuned_has_code": true, "base_length": 903, "finetuned_length": 1069 }, { "example_id": 41, "base_similarity": 0.1079092581238504, "finetuned_similarity": 0.09951845906902086, "improvement": -0.008390799054829534, "base_diff_score": 4, "finetuned_diff_score": 4, "base_has_code": true, "finetuned_has_code": true, "base_length": 1131, "finetuned_length": 1025 }, { "example_id": 42, "base_similarity": 0.02, "finetuned_similarity": 0.056866303690260134, "improvement": 0.03686630369026013, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 1154, "finetuned_length": 1153 }, { "example_id": 43, "base_similarity": 0.18210361067503925, "finetuned_similarity": 0.010889292196007259, "improvement": -0.17121431847903198, "base_diff_score": 0, "finetuned_diff_score": 1, "base_has_code": true, "finetuned_has_code": true, "base_length": 1121, "finetuned_length": 1178 }, { "example_id": 44, "base_similarity": 0.23851590106007067, "finetuned_similarity": 0.19436345966958213, "improvement": -0.044152441390488545, "base_diff_score": 3, "finetuned_diff_score": 3, "base_has_code": true, "finetuned_has_code": true, "base_length": 905, "finetuned_length": 873 }, { "example_id": 45, "base_similarity": 0.08263198163733741, "finetuned_similarity": 0.042620363062352014, "improvement": -0.0400116185749854, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 1195, "finetuned_length": 1097 }, { "example_id": 46, "base_similarity": 0.009585192530160304, "finetuned_similarity": 0.009582025441929622, "improvement": -3.1670882306815418e-06, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 5551, "finetuned_length": 5553 }, { "example_id": 47, "base_similarity": 0.05928853754940711, "finetuned_similarity": 0.11287988422575977, "improvement": 0.05359134667635266, "base_diff_score": 0, "finetuned_diff_score": 0, "base_has_code": true, "finetuned_has_code": true, "base_length": 1227, "finetuned_length": 1234 }, { "example_id": 48, "base_similarity": 0.32558139534883723, "finetuned_similarity": 0.009184845005740528, "improvement": -0.3163965503430967, "base_diff_score": 4, "finetuned_diff_score": 1, "base_has_code": true, "finetuned_has_code": true, "base_length": 923, "finetuned_length": 371 }, { "example_id": 49, "base_similarity": 0.06263048016701461, "finetuned_similarity": 0.06344827586206897, "improvement": 0.0008177956950543575, "base_diff_score": 4, "finetuned_diff_score": 4, "base_has_code": true, "finetuned_has_code": true, "base_length": 937, "finetuned_length": 950 } ], "examples": [ { "example_id": 0, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nBooleanFieldListFilter doesn't respect field choices.\nDescription\n\t\nIf I have such construction:\n# models.p...", "base_similarity": 0.26790450928381965, "finetuned_similarity": 0.3283173734610123, "improvement": 0.06041286417719266 }, { "example_id": 1, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nAdd app_label validation to showmigrations\nDescription\n\t\n#29469\n#29518\n#29506\nThe app label validation was ...", "base_similarity": 0.034241245136186774, "finetuned_similarity": 0.0025220680958385876, "improvement": -0.031719177040348184 }, { "example_id": 2, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nSwitching to inline backend closes GUI windows\n<!--To help us understand and resolve your issue, please fil...", "base_similarity": 0.003576654202568688, "finetuned_similarity": 0.02091503267973856, "improvement": 0.017338378477169875 }, { "example_id": 3, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\n[MNT]: Should plt.xticks() get a minor keyword argument\n### Summary\n\nExtracted as remaining question from #...", "base_similarity": 0.05779334500875657, "finetuned_similarity": 0.07660455486542443, "improvement": 0.018811209856667864 }, { "example_id": 4, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nRight() function on Oracle and SQLite returns improper value when the length is zero.\nDescription\n\t\nHi\nI ha...", "base_similarity": 0.169921875, "finetuned_similarity": 0.601027397260274, "improvement": 0.431105522260274 }, { "example_id": 5, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nDEFAULT_AUTO_FIELD subclass check fails for subclasses of BigAutoField and SmallAutoField.\nDescription\n\t\nSe...", "base_similarity": 0.19347319347319347, "finetuned_similarity": 0.18126888217522658, "improvement": -0.012204311297966897 }, { "example_id": 6, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nMake validators include the provided value in ValidationError\nDescription\n\t\nIt is sometimes desirable to in...", "base_similarity": 0.04270051933064051, "finetuned_similarity": 0.05339105339105339, "improvement": 0.010690534060412885 }, { "example_id": 7, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nSupport orienting adjacent reference frames in arbitrary orders\nSuppose you want to establish relative orie...", "base_similarity": 0.006165004533091568, "finetuned_similarity": 0.004700352526439483, "improvement": -0.0014646520066520854 }, { "example_id": 8, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nSettings are cleaned insufficiently.\nDescription\n\t\nPosting publicly after checking with the rest of the sec...", "base_similarity": 0.2660443407234539, "finetuned_similarity": 0.2686230248306998, "improvement": 0.0025786841072458766 }, { "example_id": 9, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nOverridden, overloaded class docstring return type rendered as None\n### Describe the bug\r\n\r\nSome overloaded...", "base_similarity": 0.0, "finetuned_similarity": 0.028701891715590344, "improvement": 0.028701891715590344 }, { "example_id": 10, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nCorrect expected format in invalid DurationField error message\nDescription\n\t\nIf you enter a duration "14:00...", "base_similarity": 0.08583106267029973, "finetuned_similarity": 0.10807204803202135, "improvement": 0.022240985361721616 }, { "example_id": 11, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nIncorrect removal of order_by clause created as multiline RawSQL\nDescription\n\t\nHi.\nThe SQLCompiler is rippi...", "base_similarity": 0.059833795013850416, "finetuned_similarity": 0.01757469244288225, "improvement": -0.04225910257096817 }, { "example_id": 12, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nSubquery.__eq__() doesn't work properly for resolved subqueries.\nDescription\n\t\nSubquery.__eq__() doesn't wo...", "base_similarity": 0.02172601086300543, "finetuned_similarity": 0.03428571428571429, "improvement": 0.012559703422708856 }, { "example_id": 13, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nlookup_allowed fails to consider dynamic list_filter\nDescription\n\t\nCurrently, lookup_allowed iterates over ...", "base_similarity": 0.11948051948051948, "finetuned_similarity": 0.24593128390596744, "improvement": 0.12645076442544795 }, { "example_id": 14, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nhist() no longer respects range=... when density=True\n<!--To help us understand and resolve your issue, ple...", "base_similarity": 0.07692307692307693, "finetuned_similarity": 0.0824270177447052, "improvement": 0.005503940821628278 }, { "example_id": 15, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nWrong measurement for one qubit state.\nHi, sympy developers.\r\n\r\n measure_all(qapply(Qubit('0')))\r\n\r\nretu...", "base_similarity": 0.09772727272727273, "finetuned_similarity": 0.07557732680195942, "improvement": -0.022149945925313316 }, { "example_id": 16, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nInfinite loop in ExceptionReporter.get_traceback_frames()\nDescription\n\t\nThe following code generates a caus...", "base_similarity": 0.0317208564631245, "finetuned_similarity": 0.03015873015873016, "improvement": -0.0015621263043943436 }, { "example_id": 17, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nDecimalValidator fails to validate 0 in scientific notation (0E+1 or 0E+2)\nDescription\n\t \n\t\t(last modified ...", "base_similarity": 0.13333333333333333, "finetuned_similarity": 0.13516896120150187, "improvement": 0.0018356278681685434 }, { "example_id": 18, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\npowdenest(sqrt(sin(x)2), force=True) does not work\nSince powdenest(sqrt(x**2), force=True) gives x, I...", "base_similarity": 0.012232415902140673, "finetuned_similarity": 0.012232415902140673, "improvement": 0.0 }, { "example_id": 19, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nBug: the predict method of Pipeline object does not use the exact predict method of final step estimator\nI ...", "base_similarity": 0.270367054610564, "finetuned_similarity": 0.34252297410192145, "improvement": 0.07215591949135747 }, { "example_id": 20, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nSquashing migrations with Meta.index_together -> indexes transition should remove deprecation warnings.\nDes...", "base_similarity": 0.021139166177334117, "finetuned_similarity": 0.020361990950226245, "improvement": -0.0007771752271078722 }, { "example_id": 21, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nkbd role generates HTML that's difficult/impossible to style for compound-keystrokes\nDescribe the bug**\r\n...", "base_similarity": 0.0, "finetuned_similarity": 0.10518174787316319, "improvement": 0.10518174787316319 }, { "example_id": 22, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nRename layout(algo=) to layout(engine=)\nMatplotlib has settled on this term with the new set_layout_engine...", "base_similarity": 0.05575935436537051, "finetuned_similarity": 0.05099931082012405, "improvement": -0.004760043545246458 }, { "example_id": 23, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nASCII table output to HTML does not support supplied \"formats\"\n<!-- This comments are hidden when you submi...", "base_similarity": 0.015315890236119975, "finetuned_similarity": 0.012972972972972972, "improvement": -0.0023429172631470024 }, { "example_id": 24, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nModelAdmin for proxy model with InlineModelAdmin for proxy superclass reference results in admin.E202\nDescr...", "base_similarity": 0.033512866546977854, "finetuned_similarity": 0.024495677233429394, "improvement": -0.00901718931354846 }, { "example_id": 25, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\ninherited-members should support more than one class\n**Is your feature request related to a problem? Please...", "base_similarity": 0.13380782918149467, "finetuned_similarity": 0.0771586037966932, "improvement": -0.056649225384801466 }, { "example_id": 26, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\npytest.approxfails withTypeError: unsupported operand type(s) for -: 'float' and 'NoneType'`\nWhen usin...", "base_similarity": 0.1459119496855346, "finetuned_similarity": 0.08541973490427099, "improvement": -0.0604922147812636 }, { "example_id": 27, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nVisibility of internal axis labels is wrong with wrapped pair plot\npython\r\n(\r\n so.Plot(mpg, y=\"mpg\")\r...", "base_similarity": 0.09552845528455285, "finetuned_similarity": 0.10167224080267559, "improvement": 0.006143785518122738 }, { "example_id": 28, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nTime-related _check_fix_default_value() methods can be optimized / simplified and have a bug\nDescription\n\t\n...", "base_similarity": 0.0891218872870249, "finetuned_similarity": 0.2590738423028786, "improvement": 0.1699519550158537 }, { "example_id": 29, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nQuerySet.update() on querysets in descending order by annotations.\nDescription\n\t\nWhen I execute \nModel.obje...", "base_similarity": 0.04787812840043525, "finetuned_similarity": 0.04472843450479233, "improvement": -0.003149693895642923 }, { "example_id": 30, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\n RFE: allow to selectively disable loggers from command-line\nA common debugging strategy is to study the lo...", "base_similarity": 0.07042253521126761, "finetuned_similarity": 0.07003444316877153, "improvement": -0.00038809204249608265 }, { "example_id": 31, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nCustom collations\nDescription\n\t \n\t\t(last modified by Tom Carrick)\n\t \nMailing list, but it didn't get any re...", "base_similarity": 0.03055229142185664, "finetuned_similarity": 0.044307692307692305, "improvement": 0.013755400885835666 }, { "example_id": 32, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nAlias used in aggregate filtering is incorrect.\nDescription\n\t\nWith the following queryset:\nIndicatorValue.o...", "base_similarity": 0.06496062992125984, "finetuned_similarity": 0.07276507276507277, "improvement": 0.007804442843812931 }, { "example_id": 33, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nmake_column_transformer has different order of arguments than ColumnTransformer\nI'm not sure if we discusse...", "base_similarity": 0.005496281926931782, "finetuned_similarity": 0.005437390052774668, "improvement": -5.8891874157114034e-05 }, { "example_id": 34, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nImprove makemigrations warning message when calling without an active database connection.\nDescription\n\t\nI ...", "base_similarity": 0.23157894736842105, "finetuned_similarity": 0.29791666666666666, "improvement": 0.06633771929824561 }, { "example_id": 35, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nMinor temporary directory security issue in pytest versions before 6.2.3\nA minor temporary directory securi...", "base_similarity": 0.03099361896080219, "finetuned_similarity": 0.06860158311345646, "improvement": 0.03760796415265427 }, { "example_id": 36, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\npytest stepwise doesn't work with xfail strict failures\n\r\ngraingert@onomastic:~/projects/foo$ cat tests/...", "base_similarity": 0.1111111111111111, "finetuned_similarity": 0.11812627291242363, "improvement": 0.007015161801312522 }, { "example_id": 37, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\n[Bug]: Setting bbox_inches to a Bbox in fig.savefig resizes colorbar\n### Bug summary\r\n\r\nSetting bbox_inches...", "base_similarity": 0.045190445448676564, "finetuned_similarity": 0.06318504190844616, "improvement": 0.0179945964597696 }, { "example_id": 38, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nAllow ManyToManyField using a intermediary table to be defined as symmetrical.\nDescription\n\t\nThanks to the ...", "base_similarity": 0.03481190342504211, "finetuned_similarity": 0.07787903893951947, "improvement": 0.04306713551447736 }, { "example_id": 39, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nMan page using :samp: with braces - font doesn't reset\nThere are issues with the man page rendering when us...", "base_similarity": 0.11209439528023599, "finetuned_similarity": 0.06110458284371328, "improvement": -0.05098981243652271 }, { "example_id": 40, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nGeneratorsError raised when creating element of fraction field of polynomial ring\nI see this construction i...", "base_similarity": 0.08125445473984319, "finetuned_similarity": 0.07138304652644997, "improvement": -0.009871408213393218 }, { "example_id": 41, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nUsing __isnull=True on a KeyTransform should not match JSON null on SQLite and Oracle\nDescription\n\t\nThe Key...", "base_similarity": 0.1079092581238504, "finetuned_similarity": 0.09951845906902086, "improvement": -0.008390799054829534 }, { "example_id": 42, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nOneHotEncoder ignore unknown error when categories are strings \n#### Description\r\n\r\nThis bug is very specif...", "base_similarity": 0.02, "finetuned_similarity": 0.056866303690260134, "improvement": 0.03686630369026013 }, { "example_id": 43, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nPossible data loss when using caching from async code.\nDescription\n\t\nCacheHandler use threading.local inste...", "base_similarity": 0.18210361067503925, "finetuned_similarity": 0.010889292196007259, "improvement": -0.17121431847903198 }, { "example_id": 44, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nIncorrect LaTeX display of a determinant\nIt displays like |(A)| instead of |A|. I fixed that issue for myse...", "base_similarity": 0.23851590106007067, "finetuned_similarity": 0.19436345966958213, "improvement": -0.044152441390488545 }, { "example_id": 45, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nShould Authorization header be cleared in https -> http redirect?\nThis may be considered intentional behavi...", "base_similarity": 0.08263198163733741, "finetuned_similarity": 0.042620363062352014, "improvement": -0.0400116185749854 }, { "example_id": 46, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nUnexpected keyword argument 'path' from plugins\nWhile troubleshooting #8332, I stumbled onto a new error, a...", "base_similarity": 0.009585192530160304, "finetuned_similarity": 0.009582025441929622, "improvement": -3.1670882306815418e-06 }, { "example_id": 47, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nAdd an encoder parameter to django.utils.html.json_script().\nDescription\n\t\nI have a use case where I want t...", "base_similarity": 0.05928853754940711, "finetuned_similarity": 0.11287988422575977, "improvement": 0.05359134667635266 }, { "example_id": 48, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nAdd subdomains of localhost to ALLOWED_HOSTS in DEBUG mode\nDescription\n\t \n\t\t(last modified by thenewguy)\n\t ...", "base_similarity": 0.32558139534883723, "finetuned_similarity": 0.009184845005740528, "improvement": -0.3163965503430967 }, { "example_id": 49, "problem": "You are an expert software engineer. Solve the following coding problem:\n\nProblem Statement:\nSaving parent object after setting on child leads to unexpected data loss\nDescription\n\t \n\t\t(last modified b...", "base_similarity": 0.06263048016701461, "finetuned_similarity": 0.06344827586206897, "improvement": 0.0008177956950543575 } ] }

Qualitative Analysis

The fine-tuned model demonstrates significant improvements over the base Llama 3.2 3B model on SWE-Bench coding tasks:

  • Enhanced code generation accuracy and syntax correctness
  • Improved understanding of software engineering problem specifications
  • Better adherence to coding instructions and best practices
  • Measurable performance gains on benchmark evaluations

Usage 

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model_name = "meta-llama/Llama-3.2-3B"
adapter_model_name = "gvij/llama-3.2-3b-swe-bench-ft"

tokenizer = AutoTokenizer.from_pretrained(base_model_name)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

model = PeftModel.from_pretrained(base_model, adapter_model_name)
model.eval()

prompt = "Write a Python function to reverse a linked list."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        temperature=0.7,
        top_p=0.9,
        do_sample=True
    )

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Hardware Requirements

  • GPU: 16GB VRAM recommended (adapter can be loaded on standard GPU)
  • RAM: 8GB minimum
  • Storage: ~50MB for adapter weights

Training Details

  • Framework: PyTorch + Hugging Face Transformers + PEFT
  • Training Time: Multiple hours on GPU
  • Optimization: AdamW optimizer with cosine learning rate schedule
  • Mixed Precision: FP16 training for efficiency

Limitations and Biases

  • Model inherits biases from base Llama 3.2 3B model
  • Performance optimized specifically for software engineering tasks
  • May not generalize well to non-coding domains
  • Requires base model weights to function (adapter-only)

Citation

If you use this model, please cite:

@misc{llama-swe-bench-ft,
  author = {gvij},
  title = {Llama 3.2 3B Fine-tuned on SWE-Bench Coding Tasks},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/gvij/llama-3.2-3b-swe-bench-ft}}
}

License

This model follows the Llama 3.2 Community License. See base model for details.

Acknowledgments

  • Base model: Meta AI (Llama 3.2)
  • Dataset: simongraves/swe-bench-coding-tasks
  • Training infrastructure: Hugging Face PEFT library
Downloads last month
39
Inference Providers NEW
Text Generation
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for gvij/llama-3.2-3b-swe-bench-ft

Base model

meta-llama/Llama-3.2-3B
Adapter
(235)
this model