laya-desktop-v1 — desktop computer-use decision head (accessibility-text channel)

A second continuation of cklxx/laya-browser v17s, specialised for desktop operation on Ubuntu: the observation is the AT-SPI accessibility tree rendered as TSV text (tag / name / text / class / description / position / size), the actions are AT-SPI / xdotool-style clicks, typing, key presses and scrolls. Same architecture (mmBERT-base 322M encoder + 2-layer decision head, laya_fmt v3), same checkpoint layout — a drop-in for v17s/.

Laya does not generate text: it answers typed questions (choice / noul / score) about a state in one bidirectional encoder pass and returns calibrated probabilities. This is the decision organ of a desktop agent loop, not a chat model.

• Published state: epoch 1 complete + ~48 % of epoch 2 (run stopped gently at epoch-2 item 24,292 / 50,788).
• Files: model.safetensors (643 MB, bf16), encoder/, tokenizer/, rl_agent_config.json.

Benchmarks at a glance

Measured locally on one RX 7800 XT 16 GB with tools/desktop_eval.py over the full 12,076-item held-out set. "This model" is the published checkpoint (epoch 1 + ~48 % of epoch 2). Details in Results below.

benchmark	epoch 1 only	this model
operation accuracy (n = 7,069)	0.929	0.939
element target top-1 (n = 5,007)	0.217	0.231
CLICK accuracy (n = 6,704)	0.970	0.985
TYPE_TEXT accuracy (n = 189)	0.344	0.190
SCROLL_UP / DONE / PRESS_ENTER / SCROLL_DOWN	0.000	0.000 (n = 102 / 39 / 18 / 17)
fitted temperature T	0.983	1.0225

How much better, plainly: completing epoch 2 to its halfway point bought +1.0 point of operation accuracy (0.929 → 0.939) and +1.4 points of element top-1 (0.217 → 0.231) — a small but real gain, and it regressed TYPE_TEXT (0.344 → 0.190) on a 189-case class, which is why the next round needs more target and typed-text data rather than more epochs.

Server-shaped measurement (chunked selection, systemone_server.py, chunk 12, 200 held-out cases): operation 100 %, element top-1 12.5 %, zero overflow errors, ~959 ms per pass on the 7800 XT. Handing the model the whole accessibility tree instead (161 candidates on average, up to 344) gives ~2 % top-1 — this head is a re-ranker for a shortlist, never a selector. On a Raspberry Pi 5 (CPU, fp32) one full desktop step takes 5.1 s.

Comparison to the base model and to Jev on this channel is not published on purpose: neither convaiinnovations/laya nor the browser-only parent v17s was trained on desktop decisions, so a head-to-head here would measure the dataset rather than the models. For the decision-quality benchmarks where all three were measured on identical harnesses (typed-decisions, PhishNChips), see the sibling model card ikaanakkok/laya-browser-v1.

Results (held-out, 12,076 items — never used for training)

metric	value
operation accuracy	0.939 (n = 7,069)
target top-1	0.231 (n = 5,007, mean normalized rank 0.260)
calibration	T = 1.0225 (fitted on the same held-out split; nll 1.390)

Per gold operation: CLICK 0.985 (n = 6,704) · TYPE_TEXT 0.190 (n = 189) · SCROLL_UP 0.000 (n = 102) · DONE 0.000 (n = 39) · PRESS_ENTER 0.000 (n = 18) · SCROLL_DOWN 0.000 (n = 17). The zeros are sample starvation (17–102 cases per class), not a measured collapse. The previous export (epoch 1 + 21 % of epoch 2) scored 0.929 / 0.217 at T = 0.983 — the extra half-epoch bought +1.0 op and +1.4 target points.

The target head is a re-ranker, not a selector. Live 80-case probe on the training machine, given the full candidate list (161 candidates on average, max 344): top-1 2 % — chance. Restricted to chunked two-pass selection (chunk 12 + final) it reaches 13 % (gold chunk winner 34 %). The 0.231 above is therefore measured on restricted candidate sets. Do not hand it the whole accessibility tree.

Training

• Base: cklxx/laya-browser v17s, RLCD continuation, single GPU (RX 7800 XT 16 GB, bf16).
• Data: gui-wm/agentnet-success-v1 → 34,898 cases (34,686 steps + 212 DONE) → 50,788 items/epoch (operation 31,227 / target 19,561), eval 12,076. Applications: vs_code 9,274 · libreoffice_impress 7,845 · gimp 5,010 · chrome 4,926 · os 3,325 · lo_writer 2,624 · vlc 1,067 · lo_calc 827.
• Run: MICRO=4 ACCUM=8 LAYA_HEAD=768 EPOCHS=2 COMPILE=0 CKPT=1 (CKPT=0 OOMs at 16 GB), ~5.4 items/s, ~2.6 h/epoch, peak VRAM 6.7 GB. rl_agent_config.json carries {"epoch": 1, "step": 2346, "partial": true} (epoch is 0-indexed: 1 = the second epoch).

How to use

import laya
agent = laya.load("ikaanakkok/laya-desktop-v1")     # or a local checkpoint dir
answers = agent.predict(state, questions)["answers"]

HTTP (same server the numbers above were measured with):

python systemone_server.py 8792 /path/to/this/ckpt 12   # 12 = chunk width for wide choice questions
curl -s localhost:8792/health

With chunking the training machine measured op 100 % and ~959 ms/pass; a full desktop step on a Raspberry Pi 5 (CPU, fp32, 4 threads) takes ~5.1 s, the same step on the 7800 XT tens of milliseconds.

State format: {"page": {"url": "desktop://<app>", "title": ..., "text": <a11y TSV, ~1200 chars>}, "recent_actions": []}; options are 1-based accessibility row ids plus role; the operation set is CLICK / TYPE_TEXT / SCROLL_DOWN / SCROLL_UP / WAIT / PRESS_ENTER / DONE / BLOCKED. Screenshots are never used.

Limitations

• Text-only, no vision. Elements without an accessible name cannot be targeted; screenshots are audit material only.
• Eight Ubuntu applications were in the training mix (VS Code, LibreOffice Impress / Writer / Calc, GIMP, Chrome, VLC, OS shell) — anything else is untested.
• English UI text. Turkish text inside an application is out-of-distribution for this head; the browser head (laya-browser-v1) is the Turkish-weighted one.
• An opt-in encoder-embedding shortlist (laya.shortlist) was tried on this head's held-out set and did not work: recall@20 12.5 %, versus 34 % for a simple lexical baseline. Chunked selection stays the recommended wide-list path.
• Not to be used for authentication, payments, or irreversible actions without a human check.

Provenance & credits

• Base: convaiinnovations/laya (Apache-2.0) → cklxx/laya-browser v17s (Apache-2.0) → this checkpoint. The parent is Apache-2.0; this derivative is published under MIT with the upstream notices kept.
• Desktop data: gui-wm/agentnet-success-v1 (training only, under its own terms).
• Execution side (not shipped here): AT-SPI observer + xdotool/ydotool actions on Ubuntu.

Türkçe özet

cklxx/laya-browser v17s'in masaüstü (bilgisayar kullanımı) için sürdürülmüş ince ayarı: gözlem AT-SPI erişilebilirlik ağacının TSV metni, eylemler tıklama/yazma/tuş/kaydırma. Held-out 12.076 vakada operation %93,9 · target top-1 %23,1 (kısıtlı aday listesinde; tüm ağaç verilirse şans). Hedef başlığı kısa listeyi sıralayan bir yeniden sıralayıcıdır. Metin üretmez, görsel kullanmaz.