owaski/infinisst-thinker-phrase-zh

InfiniSST thinker — phrase-gated simultaneous en→zh

The translation model of a cascaded simultaneous speech-to-speech system. It reads English audio in fixed chunks and emits Chinese text deltas, one per chunk; a streaming TTS speaks those deltas. This checkpoint is the phrase-gated variant: it was trained on trajectories rewritten so that nothing shorter than a semantically complete phrase is ever released in one chunk.

Internal run id: omni_sft_train_s_zh_phrase_10278289 (export hf/).

Why the phrase gating matters

A word-aligned trajectory releases whatever text the aligner had committed by the end of each 0.96 s chunk, which is often a fragment of one word. The downstream TTS then has to speak that fragment, and it does so badly. Deferring each release to the last complete phrase boundary costs nothing in total text — the concatenation over a whole utterance is unchanged — but it changes when text arrives, and it is worth +2.9 BLEU end to end.

Measured results

ACL 60/60 dev, en→zh, 1.0× speed, chunk 1920 ms, paired with owaski/moss-tts-realtime-delta-zh-125k as the TTS (backend moss-delta, per-delta turn policy). Generation job 10279835.

metric	CU	CA
BLEU	40.64
XCOMET-XL	0.723
LongYAAL (ms) ↓	4207	6114
Ending offset (ms) ↓	4018	5942

CU is the text-only computation-unaware regime; CA is the computation-aware rendering, which is the one that includes the system's own compute. Latency alignment is SEGALE; skipped segments under/over = 0/0 in both regimes.

Comparison at the same condition and the same TTS: the word-aligned thinker scores 37.7 BLEU. Swapping in a phrase-gated TTS instead of this one loses 3.9 BLEU (36.75) and adds ~800 ms, because that corpus removed nearly all short turns and the TTS then cannot speak the short deltas the server still feeds it. The win here is on the thinker side only.

Training

• Base: Qwen/Qwen3-Omni-30B-A3B-Instruct
• Method: LoRA SFT of the thinker, ms-swift + Megatron-LM, bf16, 2 GPUs (expert parallelism = GPU count)
• Data: 12,481 rows of phrase-gated GigaSpeech en→zh trajectories (train_s_zh_phrase.jsonl, sha256 fd741eda…e1e45f8e)
• Megatron-LM: 73a28a1078a8da8e6062199f7f1079a52173ab77
• Full training arguments, image digest and data hashes are in olt_provenance.json in this repo.

Use

Served with vLLM as an OpenAI-compatible endpoint and driven by a SimulEval policy that sends one audio chunk at a time. It is not a general chat model: it expects the simultaneous interpreting system prompt and an alternating audio/text conversation, and it emits deltas meant to be concatenated, not standalone translations.

Caveats

• Trained and evaluated for en→zh only.
• The numbers above are one dev split at one chunk size; a different chunk size or speed is a different operating point, not a different measurement of the same one.
• olt_provenance.json records absolute paths from the cluster it was trained on.