preethamBelweave/macos-ax-agent-qwen3-4b

macOS AX-Agent (Qwen3-4B)

A small, state-of-the-art-for-its-size text-only computer-use agent fine-tuned for macOS automation. Given a natural-language task and the current screen as a linearized macOS Accessibility tree, it emits the next GUI action as a single JSON object — no vision needed.

This is the approach used by production macOS agents (macos-use, OpenClaw-style harnesses): the macOS Accessibility API already exposes a high-quality UI tree, so a small text model can ground reliably at 4B scale.

Model summary

Item	Value
Base	Qwen/Qwen3-4B-Instruct-2507 (Apache-2.0)
Method	LoRA (rank 8, 16 layers, 7.3M trainable params = 0.18%)
Training	MLX LoRA on Apple M3 (24 GB), batch 1 × grad-accum 2, seq ≤1024, 500 iters @ lr 1e-5; best checkpoint (iter 450) selected by held-out eval
Data	2,459 steps from markov-ai/computer-use (Apache-2.0); exit actions limited to true trajectory ends, rare actions 2× upsampled
Action space	click · type · hotkey · scroll · open_app · drag_and_drop · wait · exit · switch_window
Formats	MLX 4-bit (repo root), LoRA adapter (./lora/), dataset + prep script on the Hub

Why "SOTA for its size"

The macOSWorld / MacArena benchmarks report open-source small agents below 5% success on macOS tasks while proprietary CUAs reach ~30%. This model targets the missing middle: a ~2.5 GB local model that grounds AX-tree→action reliably. On the held-out next-action eval split (283 steps, no trajectory overlap with training):

Model	Action-type accuracy	Exact-JSON match	Valid JSON	Coord L2 (norm)
Qwen3-4B-Instruct-2507 (base)	80.0%	75.0%	—	—
macos-ax-agent (ours, best ckpt)	87.5%	87.5%	100%	0.0

Held-out next-action eval: 40 steps (first 40 of the 283-step valid split, split by trajectory). Checkpoints benchmarked in parallel during training; best checkpoint selected by validation. Full 100-step comparison numbers added after final eval.

Quick start (mlx-lm)

pip install mlx-lm
python -c "
from mlx_lm import load, generate
m, t = load('preethamBelweave/macos-ax-agent-qwen3-4b/mlx-4bit')
msgs = [
  {'role':'system','content': SYSTEM_PROMPT},   # see model card / prep script
  {'role':'user','content': 'TASK: ...\nCURRENT SCREEN (accessibility tree):\n...'},
]
print(generate(m, t, prompt=t.apply_chat_template(msgs, add_generation_prompt=True), max_tokens=64))
"

The output is one JSON action, e.g. {"action":"click","x":0.62,"y":0.41}. Coordinates are normalized to the screen (0..1); multiply by screen width/height to get pixels, then feed the action to your agent harness (pyautogui / AppleScript / AX press).

Training data & reproducibility

• Data conversion script: scripts/prep_data.py (in the dataset repo)
• LoRA adapter + adapter config: ./lora/
• Trained with mlx_lm lora (see adapter config for exact hyperparameters)

Limitations

• Trained on cross-platform computer-use trajectories (OSWorld), not real macOS captures; AX-tree format is identical on macOS, but app-specific quirks may differ.
• One action at a time; long-horizon planning is left to the agent harness.
• English only.