AlexWortega/capabilityvectors-qwen3-4b

Capability Vectors — Qwen3-4B LoRA adapters (weight-space geometry of reasoning losses)

Attention-only LoRA adapters (q,k,v,o_proj, rank 32, alpha 64) for Qwen3-4B-Instruct-2507, trained on identical DeepScaleR math rollouts (math-verify reward) for the study "Same Data, Different Losses, Same Circuits?" — companion to github.com/AlexWortega/capabilityvectors.

All 28 adapters are trained in one consistent setup, so their LoRA deltas (ΔW = (α/r)·B·A) are directly comparable in weight space.

Weight-space geometry across all losses

SFT/RFT/RIFT colinear (0.94–0.98); DFT ~0.55; Offline GRPO 0.71–0.80 to the cluster; DPO near-orthogonal (≤0.13). Online GRPO & DAPO are each near-orthogonal to every offline loss (cos ≈0.02) and to each other (−0.16) — orthogonal-fraction off SFT 0.998/0.995 vs 0.69 for offline GRPO. On-policy sampling, not the group-relative loss, drives the departure from SFT. (Small negative cosines ≈ orthogonal: the on-policy ΔW are ~10× smaller in norm.)

Seed & learning-rate sensitivity

Same loss at two seeds has low raw cosine, but the top-1 output direction agrees at 0.99 and the two seeds sit in the same basin (no linear-mode barrier, midpoint +0.004) — the low cosine is a LoRA input-init artifact, not a different solution. A 10× LR change rotates ΔW (cos ≈0.55), it is not a pure rescaling.

Adapters (adapters/<method>_lr<lr>_s<seed>/)

family	method	grid
offline (reward-weighted MLE)	sft, rft	lr {5e-7,5e-6,5e-5} × seed {42,123}
offline (other)	dft, rift, offgrpo (offline GRPO), dpo	seed 42, paper LR (dpo 5e-7)
online RL	grpo (online GRPO), dapo (online DAPO)	lr {5e-7,5e-6,5e-5} × seed {42,123}

offgrpo = offline GRPO; grpo = online GRPO (on-policy rollouts, group-relative advantage, TRL + vLLM); dapo = online DAPO (clip-higher, no-KL, token-level, dynamic sampling).

Accuracy (greedy pass@1)

method	GSM8K	AIME26
base instruct	94.0	16.7
SFT / Offline GRPO	87.6 / 87.3	6.7
DPO	94.2	13.3
Online GRPO	93.7	20.0
Online DAPO	93.3	16.7

Reward-orthogonal methods (DPO, online GRPO/DAPO) keep the base 93–94% on GSM8K across the full lr×seed grid (91–94%); the SFT direction drops it to ~87%.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = "Qwen/Qwen3-4B-Instruct-2507"
m = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
m = PeftModel.from_pretrained(m, "AlexWortega/capabilityvectors-qwen3-4b", subfolder="adapters/grpo_lr5e-6_s42")
tok = AutoTokenizer.from_pretrained(base)

See results/RESULTS.md for full tables and results/figures/ for all plots.