Canary-Qwen-2.5B — ONNX (speech-llm)

ONNX export of nvidia/canary-qwen-2.5b for onnx-asr with the speech-llm model type.

The source model is a NeMo SALM: a FastConformer encoder (32 layers, d_model 1024, 8x subsampling) with a linear projection into the embedding space of an unmodified Qwen3-1.7B decoder. It tops the Open ASR Leaderboard for English. Model and weights are by NVIDIA, released under CC-BY-4.0.

Requirements

This export needs the speech-llm model family, which is not in upstream onnx-asr yet:

branch feat/speech-llm-canary of the TigreGotico fork. The stock code of PR #3 is not enough: the NeMo encoder graph takes the feature length and returns the embedding length, which PR #3 does not pass.

pip install "onnx-asr @ git+https://github.com/TigreGotico/onnx-asr@feat/speech-llm-canary"

Usage

import onnx_asr

model = onnx_asr.load_model("speech-llm", "path/to/this/repo")  # or quantization="int8"
print(model.recognize("audio_16khz.wav"))

Graph layout

Graph	Inputs	Outputs
`encoder.onnx`	`input_features` `(1, 128, T)`, `input_features_lens` `(1,)`	`audio_embeds` `(1, L, 2048)`, `audio_embeds_lens` `(1,)`
`embed_tokens.onnx`	`input_ids` `(1, S)`	`inputs_embeds` `(1, S, 2048)`
`decoder.onnx`	`inputs_embeds`, `attn_bias`, `position_ids`, `past_key_values.{0..27}.{key,value}` `(1, 8, P, 128)`	`logits`, `present.{0..27}.{key,value}`

decoder.onnx is a merged prefill and decode graph; a zero-length past is valid, so there is no use_cache_branch. The chat-template prompt token ids (Transcribe the following: <audio>, Qwen chat template) are baked into config.json, so no tokenizer encoder is needed at runtime. The audio front end is the standard NeMo log-mel featurizer, so config.json selects the existing nemo128 preprocessor of onnx-asr; no new preprocessor code was added.

Files

File	Size
`encoder.onnx` + `encoder.onnx_data`	42.4 MB + 3.25 GB
`encoder_int8.onnx`	859.3 MB
`embed_tokens.onnx`	1.24 GB
`embed_tokens_int8.onnx`	311.2 MB
`decoder.onnx` + `decoder.onnx_data`	1.6 MB + 6.88 GB
`decoder_int8.onnx` + `decoder_int8.onnx_data`	1.7 GB
`vocab.json`	3.1 MB

Total fp32 about 11.4 GB, total int8 about 2.9 GB. int8 is dynamic weight quantization (QInt8, MatMulConstBOnly).

Accuracy

Four Google FLEURS validation clips, two en_us and two pt_br. The model is English-only; the Portuguese clips are robustness probes, not a quality claim.

ONNX fp32 matches native NeMo SALM.generate character for character on all four clips, including the two Portuguese ones where both produce nonsense.

int8 keeps the English clips correct, with two trivial differences on the longer one ("Archipelago" instead of "archipelago", "chilies" instead of "chillies"). On the Portuguese clips int8 collapses into repetition loops. Use int8 for English only.

Clip	native NeMo fp32	ONNX fp32	ONNX int8
en_1	When you call someone who is thousands of miles away you are using a satellite	identical	identical
en_2	Now widely available throughout the archipelago, Japanese cuisine features an array of simply seasoned dishes, ...	identical	"Archipelago", "chilies"
pt_1	(French-sounding nonsense)	identical	repetition loop
pt_2	(Spanish-sounding nonsense)	identical	repetition loop

Speed

AMD Ryzen 5 7600 (6 cores, 12 threads), onnxruntime 1.28.0 CPU provider, default thread count, greedy decoding, batch 1.

Clip	audio	native NeMo	ONNX fp32	ONNX int8
en_1	6.5 s	4.8 s (RTFx 1.4)	23.7 s (RTFx 0.28)	1.9 s (RTFx 3.5)
en_2	16.4 s	12.8 s (RTFx 1.3)	37.4 s (RTFx 0.44)	4.7 s (RTFx 3.5)
pt_1	11.8 s	10.5 s (RTFx 1.1)	14.2 s (RTFx 0.83)	4.0 s (RTFx 3.0)
pt_2	14.6 s	12.6 s (RTFx 1.2)	13.4 s (RTFx 1.1)	37.2 s (loop)

Model load (cold, weights on a 7200 rpm HDD): 787 s fp32, 119 s int8. On an SSD this is dominated by the file read, not by onnxruntime.

fp32 on CPU is slower than PyTorch because the 11 GB of weights do not fit in cache and each decode step is a separate graph run. int8 is the useful CPU configuration.

Limitations

English only. The encoder saw German, French and Spanish during pretraining, so other languages produce plausible-looking nonsense rather than an error.
Batch 1 graphs. recognize_batch loops over the waveforms.
Greedy decoding only. No beam search, no timestamps.
Trained on audio up to 40 s and 1024 total tokens. Longer audio may degrade.
Only the baked ASR prompt. The LLM mode of the source model (summarize, answer questions about a transcript) is not exported.

License and attribution

CC-BY-4.0, inherited from the source model. Model and weights: NVIDIA (nvidia/canary-qwen-2.5b), built on nvidia/canary-1b-flash and Qwen/Qwen3-1.7B. This repository only adds the ONNX export.