ngqtrung/video-8b-grpo-sft770

Qwen3-VL-8B · Video-MC · GRPO from OMR-SFT-770 warmstart

RLVR post-training of Qwen/Qwen3-VL-8B-Instruct on multiple-choice video QA with fully-async GRPO, warm-started from an OpenMMReasoner (OMR) supervised checkpoint (checkpoint-770-hf). This is experiment_name=grpo_video_4node_full_v3_24f100k_8b_sft770_perf, global_step_180 (the keeper).

Video RL from the OMR-SFT-770 warm-start reaches offline full-set val mean_accuracy 0.4845 @ step 180 (full set = 5645 rows: VideoMME-v1 2700 + PerceptionComp 1108 + Video-Holmes 1837). Training reward climbed 0.72→0.89 but held-out val stayed flat (reward/val divergence). The cold-start base sibling catches and matches this by step 80 — SFT warm-start buys no durable video val advantage; both land in the ~0.485 dead-heat.

Results

Offline full-set eval (VideoMME-v1 2700 + PerceptionComp 1108 + Video-Holmes 1837 = 5645 rows), scored with the repo's vero compute_score. mean = macro-mean of the 3 bench accuracies.

Keeper = global_step_180 (peak):

metric	mean	videomme	holmes	perceptioncomp	format
SFT-770 RL @180 (keeper)	0.4845	0.6574	0.4513	0.3448	0.983
cold-base RL keeper @80 (sibling)	0.4918	0.6581	0.4741	0.3430	—
stock Qwen3-VL-8B ckpt-0 (zero-shot)	0.4444	0.6426	0.4143	0.2762	—

Trajectory (mean): flat plateau ~0.481 (±0.01) over steps 60–160, peak 0.4845 @180, then edged down (0.467 @200, 0.471 @220). Format compliance saturated ~0.97–0.98 throughout. The high reward did not convert to held-out accuracy — visible only because of the offline full-set eval.

Training

• Base model: Qwen/Qwen3-VL-8B-Instruct, warm-started from the OMR SFT checkpoint qwen3vl8b_ommr_sft_3node/checkpoint-770-hf. (SFT-770 = the stock 8B SFT'd on OpenMMReasoner reasoning data with lmms-engine — a math/visual-reasoning SFT, not a video model — used here to seed video RL.)
• Framework: fork of volcengine/verl — ngquangtrung57/verl@videorl-mods. Fully-async GRPO: FSDP2 trainer + vLLM rollouter.
• Reward: dapo-style score = 0.8·accuracy + 0.2·format (FORMAT_WEIGHT=0.2, FORMAT_MIN_THINK_CHARS=100). No KL penalty.
• Exploration: OFF.
• Topology: 4-node 2+2 — 2 trainer nodes (16-GPU FSDP2, dp=16) + 2 rollout nodes (16 GPU, vLLM TP=2 → 8 replicas). H100×8 per node.
• Batch: ppo_mini_batch_size=16 × require_batches=4 × rollout.n=8 = 512 trajectories/step.
• Data: GROUP_VIDEO_TRAIN_MC_24F100K (5 video-MC parquets, 24 frames / 100k pixels).
• Optim / seq: lr 1e-6, warmup 25 steps; total_epochs=2; clip_ratio 0.2 / 0.3 (clip_c=10.0); max_prompt_length=17408, max_response_length=16384; enforce_eager=true; gpu_memory_utilization=0.75; staleness 0.4.
• Validation: inline val OFF (test_freq=10000); video val is offline full-set eval on a dedicated 8×H100 node (vLLM TP1), every 20 fit-steps.
• Train metrics: ~216 s/step; reward 0.72→0.89 (peak 0.887); final response_length ~137 tok; 249 steps trained (stopped at fit-step ~245). Zero crashes in ~15 h.

W&B

Project verl_fully_async (entity quangtrung5705-nanyang-technological-university-singapore). Train metrics only — video val is offline, not on W&B: https://wandb.ai/quangtrung5705-nanyang-technological-university-singapore/verl_fully_async/runs/s59ethn1

Intended use / limitations

Research checkpoint — the SFT-warmstart arm of an 8B video study whose headline finding is a 7-way dead-heat at ~0.485 full-set val. This run shows SFT warm-start gives faster early convergence but no durable val edge over cold-start (which actually peaks slightly higher at 0.4918). Multiple-choice video QA, <think>…</think> then-answer format. No safety/RLHF alignment beyond the base.

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "ngqtrung/video-8b-grpo-sft770"
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

messages = [{
    "role": "user",
    "content": [
        {"type": "video", "video": "clip.mp4"},
        {"type": "text", "text": "Answer the multiple-choice question. Reason inside <think>...</think>, then give the final letter."},
    ],
}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device)
out = model.generate(**inputs, max_new_tokens=1024)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

Citation / lineage

• Base model: Qwen3-VL-8B-Instruct (Qwen team). Inherits the Qwen3-VL license — review the base model's terms; the Apache-2.0 tag refers to this repo's RLVR training artifacts.
• Warm start: OMR-SFT-770 (OpenMMReasoner SFT of Qwen3-VL-8B-Instruct, trained with lmms-engine).
• Framework: verl (volcengine/verl), fork ngquangtrung57/verl@videorl-mods; fully-async GRPO (FSDP2 + vLLM).
• Study: controlled OMR/Video exploration study on Qwen3-VL-8B; SFT-warmstart video arm (docs/experiments_summary_8b.md).