yungisimon/Qwen2.5-32B-Instruct-memory-nqa-full-ep1

Qwen/Qwen2.5-32B-Instruct fine-tuned as a memory model on NarrativeQA synthetic QA pairs (NQA_chunks_subset101_numsamplingepochs1_crossdoc_entity_combination_v3.json, 1,276,668 examples) โ€” checkpoint after epoch 1 of 3.

Weights are bf16 safetensors and load with plain transformers (AutoModelForCausalLM.from_pretrained). Use the Qwen chat template.

Training

method full fine-tuning (FSDP)
trainable params 32,763,876,352 (100.00%)
effective batch size 512 (64/device x 1 accum x 8 GPUs)
learning rate 2e-05 (constant with warmup, warmup ratio 0.05)
epochs 3 (this is epoch 1)
weight decay / max grad norm 0.01 / 1.0
max sequence length 8096
precision bf16, gradient checkpointing, loss on assistant tokens only
hardware 8x AMD Instinct MI325X
total training wall-clock (3 epochs) 8.61 h (68.9 GPU-h)
Downloads last month
195
Safetensors
Model size
33B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for yungisimon/Qwen2.5-32B-Instruct-memory-nqa-full-ep1

Base model

Qwen/Qwen2.5-32B
Finetuned
(1,236)
this model