Canary 1B v2, OpenVINO fp16 (CPU / GPU)

nvidia/canary-1b-v2 at commit d455706339a6b32e1aa40f82c713a482a0c938e2, exported to OpenVINO IR as two models, with weights stored as fp16.

Made for vinoWhisper, which runs it on CPU and GPU for live captions and dictation, and pins this repo by commit and per-file sha256.

For the Intel NPU use arankays/canary-1b-v2-int8-ov, which is half the size and faster there.

Attribution

Canary-1B-v2 is by NVIDIA and licensed CC-BY-4.0. This repository is a modified version: the weights are unchanged apart from storing them as fp16, but the network is split into the two OpenVINO graphs described below and rewired for fixed shapes. It is not affiliated with or endorsed by NVIDIA.

What is in it

File
encoder.xml / .bin FastConformer encoder for a fixed 12 s window. Inputs feats [1, 1201, 128] (log-mel) and fmask [1, 1201] (1 for real frames). Outputs the cross-attention mask, then K and V for each of the 8 decoder layers.
decoder.xml / .bin One decoder token per call, static shapes, no internal state. Inputs ids, pos, mask ([1, 1, 1, 193] additive), cmask, past_k0..7 / past_v0..7 ([1, 8, 192, 128]) and cross_k0..7 / cross_v0..7. Outputs logits, then the new K and V for each layer, which the caller writes into its 192-slot cache.
tokenizer.json The base model's tokenizer, unchanged.

These are not a drop-in for transformers or OpenVINO GenAI; something has to drive the loop. vinowhisper/canary.py does, including a numpy port of the mel features. The prompt is the base model's: <|startofcontext|><|startoftranscript|><|emo:undefined|><|src|><|tgt|><|pnc|><|noitn|><|notimestamp|><|nodiarize|>.

Feature extraction trap: the base model's extractor marks len(audio) // 160 frames as real, one fewer than it computes (1200 of 1201 for 12 s). Pass that mask in fmask, or the encoder's output drifts and decoding degrades.

How it was made

uv run scripts/export_canary.py

The script traces both graphs from transformers' CanaryForConditionalGeneration, checks the wrapper decodes the same tokens as generate() with logits within 1e-3 (measured 1e-4), and saves fp16 IR; transformers 5.22.2, torch 2.15.0+cpu, OpenVINO 2026.4.0, nncf 3.4.0.

Measured (2026-11-06, Intel Core 5 320, OpenVINO 2026.4.0)

Per 12 s window on the same 50-token check clip: CPU 1.65 s (4 threads), Intel Xe3 iGPU 0.71 s, NPU 0.75 s. Decoded tokens identical to transformers on all three.

One laptop, audiobook narration, n=1 per figure. Full write-up in docs/models.md.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for arankays/canary-1b-v2-fp16-ov

Finetuned
(20)
this model