---
license: apache-2.0
base_model: Qwen/Qwen3-VL-2B-Instruct
base_model_relation: merge
tags: [vision-language, grounding, android, ui-automation, chinese]
language: [zh, en]
pipeline_tag: visual-question-answering
---

# Pony-AndroidGrounding

A vision-language model fine-tuned for **visual grounding on Android UI screenshots**.
Base model: Qwen3-VL-2B-Instruct. Trained with LoRA (r=16 on q/k/v/o_proj), then
**merged into full bf16 weights** (`merge_and_unload`). Also carries a custom
identity/behavior corpus (Identity + Behavior SFT + DPO).

## Usage
```python
from transformers import AutoModelForImageTextToText, AutoProcessor
model = AutoModelForImageTextToText.from_pretrained("JivinDotL/pony-android-grounding", dtype="bfloat16")
proc = AutoProcessor.from_pretrained("JivinDotL/pony-android-grounding")
# instruction + portrait phone screenshot -> <bbox>x1,y1,x2,y2</bbox> (normalized 0-1000), or 未找到 ("not found")
```
Protocol: for a target-finding instruction the model returns exactly one bbox;
if the target is absent it must output `未找到` (refusal).

## Metrics (frozen test sets: val 225 / test 115 = held-out app domain, frozen before training, reproducible; v10.0b, 2026-11)
| split | iou50 | iou75 | click | correct refusal | valid JSON |
|---|---|---|---|---|---|
| val (225) | 87.3 | 85.0 | 93.5 | 100 | 100 |
| **test (115)** | **80.4** | **72.5** | **87.5** | **100** | **100** |
| dense enumeration holdout (52, multi-target) | 100.0 | 98.1 | — | — | 100 |

Latency: ~0.5 s per request (bf16, single datacenter GPU). The test split contains
a fully held-out app domain, so the numbers reflect cross-interface generalization.
Split manifests (`splits/v9.json`, `splits/dense_exam.json`) ship with this repo.
Tool-call protocol (`<tool_call>` JSON; `locate_ui_element` / `tap`) scores 6/6 under a
strict parser; the model **only outputs suggested coordinates and never executes actions**.

## Known issues (honest limits, measured 2026-11-04, v10.0b)
1. **Persona probes 5/6.** Direct and paraphrased "who are you / what can you do /
   when were you released" are answered truthfully (including "I cannot tap for you,
   I only output coordinates"). The single unstable probe Q6 (relationship to
   "Pony.ai") is **confabulation-prone**: depending on the numerical form of the same
   model (adapter vs merged weights were both measured), it either honestly denies any
   relationship or fabricates a fake "co-founded by" story. This axis is not fully
   fixed — treat its answers as unreliable chit-chat, not facts. No other probe shows
   this behavior.
2. Meta info such as the release date (2027-01-01) is a fictional persona value, not a real event.
3. Training domain is portrait 720x1600 phones; landscape/web UIs are out of distribution.
   Dense enumeration ("list all clickable elements") scores perfect within trained app families only.
4. Weak on small icons (<40 px) and near-identical icon groups.
5. The personal/lab names the model uses about itself are self-created fictional personas,
   unrelated to any real person or organization (author attribution in public texts follows
   the usual abbreviation; model behavior keeps the original names).

## Data & Training
- Training screenshots were captured on real devices and are **not redistributed** with the
  model (third-party app UI copyright). Annotations are released separately as JSON
  (L1 scope, annotations only) — see `DATASET_CARD.md`.
- Training: two-stage SFT (LoRA) + DPO on a single GPU; recipes and protocols are documented
  in the project repository.

## Citation / License
Model weights Apache-2.0 (aligned with the base model Qwen3-VL-2B-Instruct).