Canary 1B v2, OpenVINO fp16 (CPU / GPU)
nvidia/canary-1b-v2 at commit
d455706339a6b32e1aa40f82c713a482a0c938e2, exported to OpenVINO IR as two models, with weights stored as fp16.
Made for vinoWhisper, which runs it on CPU and GPU for live captions and dictation, and pins this repo by commit and per-file sha256.
For the Intel NPU use arankays/canary-1b-v2-int8-ov, which is half the size and faster there.
Attribution
Canary-1B-v2 is by NVIDIA and licensed CC-BY-4.0. This repository is a modified version: the weights are unchanged apart from storing them as fp16, but the network is split into the two OpenVINO graphs described below and rewired for fixed shapes. It is not affiliated with or endorsed by NVIDIA.
What is in it
| File | |
|---|---|
encoder.xml / .bin |
FastConformer encoder for a fixed 12 s window. Inputs feats [1, 1201, 128] (log-mel) and fmask [1, 1201] (1 for real frames). Outputs the cross-attention mask, then K and V for each of the 8 decoder layers. |
decoder.xml / .bin |
One decoder token per call, static shapes, no internal state. Inputs ids, pos, mask ([1, 1, 1, 193] additive), cmask, past_k0..7 / past_v0..7 ([1, 8, 192, 128]) and cross_k0..7 / cross_v0..7. Outputs logits, then the new K and V for each layer, which the caller writes into its 192-slot cache. |
tokenizer.json |
The base model's tokenizer, unchanged. |
These are not a drop-in for transformers or OpenVINO GenAI; something has to drive the
loop. vinowhisper/canary.py
does, including a numpy port of the mel features. The prompt is the base model's:
<|startofcontext|><|startoftranscript|><|emo:undefined|><|src|><|tgt|><|pnc|><|noitn|><|notimestamp|><|nodiarize|>.
Feature extraction trap: the base model's extractor marks len(audio) // 160 frames as
real, one fewer than it computes (1200 of 1201 for 12 s). Pass that mask in fmask, or
the encoder's output drifts and decoding degrades.
How it was made
uv run scripts/export_canary.py
The script
traces both graphs from transformers' CanaryForConditionalGeneration, checks the
wrapper decodes the same tokens as generate() with logits within 1e-3 (measured 1e-4),
and saves fp16 IR; transformers 5.22.2, torch 2.15.0+cpu, OpenVINO 2026.4.0,
nncf 3.4.0.
Measured (2026-11-06, Intel Core 5 320, OpenVINO 2026.4.0)
Per 12 s window on the same 50-token check clip: CPU 1.65 s (4 threads), Intel Xe3 iGPU
0.71 s, NPU 0.75 s. Decoded tokens identical to transformers on all three.
One laptop, audiobook narration, n=1 per figure. Full write-up in docs/models.md.
Model tree for arankays/canary-1b-v2-fp16-ov
Base model
nvidia/canary-1b-v2