TrialMatch LoRA adapters (Qwen2.5-1.5B-Instruct)

Educational screening prototype, not a clinical enrollment decision.

Three LoRA adapters trained for one task: given a numbered patient note and one trial criterion (inclusion or exclusion), output a JSON object with the supporting sentence ids and one of the dataset's labels.

{"evidence_sentence_ids":[0,2],"label":"included"}

Files

Folder Seed Validation macro-F1 (best epoch)
seed42/ 42 0.426
seed43/ 43 0.440
seed44/ 44 0.473

Each folder holds adapter_model.safetensors and adapter_config.json.

draft/ holds a separate adapter for the smaller Qwen/Qwen2.5-0.5B-Instruct (revision 7ae557604adf67be50417f59c2c2f167def9a775), trained with the same recipe (seed 42). It is the draft model for speculative decoding against the 1.5B target and is not meant to be used alone. Validation macro-F1 0.438, accuracy 0.738 (same caveats on the rare classes: it predicted not included 3 times, none correct, and never excluded). The base model of this folder is the 0.5B model, not the 1.5B one.

Training

  • Base model: Qwen/Qwen2.5-1.5B-Instruct at revision 989aa7980e4cf806f80c7fef2b1adb7bc71aa306.
  • Data: ncbi/TrialGPT-Criterion-Annotations at revision 1cfcafde94a1560a33b4addc1638664fe28fc059 (license listed on the dataset page as other). Patient-level split 37/8/8 patients with seed 42; training used the 37 training patients only (602 usable rows).
  • LoRA: r=16, alpha=32, dropout 0.05, targets q_proj,k_proj,v_proj,o_proj.
  • Optimiser: AdamW, lr 2e-4, cosine schedule with 5% warmup, batch 8, 3 epochs, fp16 base with float32 adapter and a GradScaler, gradient checkpointing.
  • Rare-label oversampling: rows labelled excluded or not included are repeated 4 times.
  • Loss covers the assistant tokens only. Prompt, split and training code are in the project repository.
  • Checkpoint per seed: the epoch with the highest strict validation macro-F1.

Results so far

Validation only (206 rows from 8 held-out patients). The test split has not been opened.

  • Mean validation macro-F1 over the three seeds: 0.446 (SD 0.024). Mean accuracy 0.731. For comparison, a per-type majority baseline scores 0.215 macro-F1 and 0.621 accuracy.
  • The rare classes are not learned. The validation split has 2 excluded and 3 not included rows, and across these three adapters only one row of each was ever predicted correctly. Do not read these adapters as able to detect those two labels.
  • A competing configuration (r=32, no oversampling) scores 0.434 (SD 0.010) and cannot be told apart from this one.

Limits

  • Trained on 37 patients from one dataset. Expect weak transfer to other note styles.
  • Labels follow the dataset's annotation scheme, not a clinical judgement.
  • Output must be validated: the cited sentence ids can be wrong, and a missing citation never means a label.
Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for AkifJK/trialmatch-qwen2.5-1.5b-lora

Adapter
(1,764)
this model