Qwen2.5-1.5B-Instruct LoRA for criterion-level clinical-trial eligibility screening

A LoRA adapter for Qwen/Qwen2.5-1.5B-Instruct. It takes a patient note (numbered sentences), a criterion type (inclusion or exclusion) and the criterion text. It returns the expert-style eligibility label and the IDs of the note sentences that support it, as JSON:

{"label": "not enough information", "evidence_sentence_ids": [2]}

Educational research prototype (university assignment). It is not a clinical enrollment decision tool.

Labels

Criterion type Labels
inclusion included, not included, not enough information, not applicable
exclusion excluded, not excluded, not enough information, not applicable

not enough information and not applicable are kept as separate classes and are never collapsed into yes/no.

Results

Evaluated on the patient-grouped test split (8 unseen patients, 206 criteria). The test split was evaluated once, with greedy decoding (repetition_penalty=1.0).

Model Accuracy Macro-F1 Evidence micro-F1 Evidence F1 (gold non-empty)
Qwen2.5-1.5B-Instruct (base) 0.325 0.256 0.123 0.111
+ this adapter 0.743 0.545 0.508 0.533

Validation (8 patients, 206 criteria): accuracy 0.733, macro-F1 0.431, evidence micro-F1 0.522.

Per class (test): included F1 0.86, not excluded 0.83, not enough information 0.63, not applicable 0.40, not included 0.00 (only 13 training examples).

Training

  • Data: ncbi/TrialGPT-Criterion-Annotations (expert labels and expert evidence sentences only; the GPT-4 columns are never used). Patient-grouped split: sorted patient IDs shuffled with numpy.random.default_rng(42), 37/8/8 patients (603/206/206 criteria). The manifest is in split_manifest.json.
  • Objective: causal-LM cross-entropy on the JSON answer tokens only (prompt tokens masked).
  • LoRA: r=16, alpha=32, dropout 0.05, on q/k/v/o/gate/up/down projections (18.5M trainable parameters, 1.18%).
  • Optimisation: AdamW, lr 2e-4, cosine schedule, 5% warm-up, 3 epochs, micro-batch 2 x gradient accumulation 8 (effective batch 16, 114 steps), fp16 with gradient checkpointing, seed 42.
  • Checkpoint: step 80, selected by validation loss (0.113).
  • Compute: 1x NVIDIA T4 (Kaggle), 15.8 minutes. No input was truncated (maximum 650 tokens; limit 1024).

How to use

The prompt format must match training. The easiest way is the repository's helper:

# pip install torch transformers peft ; git clone https://github.com/ibrahimmlk/trialsieve
import sys, torch
sys.path.insert(0, "trialsieve/src")
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
from trialsieve.data import messages, parse_output, number_note, note_sentences

base = "Qwen/Qwen2.5-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, dtype=torch.float32)
model = PeftModel.from_pretrained(model, "ibrahimmlk/trialsieve-qwen2.5-1.5b-lora").merge_and_unload().eval()

note = number_note("A 62-year-old man with type 2 diabetes. He had a myocardial infarction two years ago.")
rec = {"note": note, "criterion_type": "exclusion", "criterion_text": "Myocardial infarction within the past 6 months"}
prompt = tok.apply_chat_template(messages(rec), tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt", add_special_tokens=False).input_ids
out = model.generate(ids, do_sample=False, max_new_tokens=64, repetition_penalty=1.0)
text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
print(parse_output(text, "exclusion", len(note_sentences(note))))  # (label, evidence_ids, parsed_ok)

For a complete screening workflow (all criteria of a trial, evidence, rule-based outcome, report export), run the browser application in the repository (run_app.bat).

Limitations

  • Small training set (37 patients). The minority classes not included (13 examples) and excluded (8) are not learned, and the test split has no excluded case.
  • The most frequent error is reading missing information as a negative finding (not enough information predicted as not excluded, 28 of 206 test criteria). It also struggles with numeric range checks.
  • Low evidence recall on exclusion criteria (0.09): the model often answers not excluded without citing a sentence.
  • The patient notes are short synthetic case descriptions, not real EHR text.
  • Not validated for clinical use. Every output must be reviewed by qualified clinical research staff.

Citation

The dataset comes from: Jin et al., "Matching patients to clinical trials with large language models", Nature Communications 15, 9074 (2024). https://www.nature.com/articles/s41467-024-53081-z

Downloads last month
88
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for ibrahimmlk/trialsieve-qwen2.5-1.5b-lora

Adapter
(1,764)
this model

Dataset used to train ibrahimmlk/trialsieve-qwen2.5-1.5b-lora