pyros-vault/Ornith-1.5-35B-A3B-fixed-mtp-NInfer

Ornith-1.5-35B-A3B fixed-MTP for NInfer-4090

This repository contains a single-file, mixed-precision .ninfer conversion of shisa-ai/Ornith-1.5-35B-A3B-MTP, built and tested with UDPSendToFailed/ninfer-4090 on a 24 GB RTX 4090 (sm_89).

The artifact contains the Ornith target model, Shisa AI's replacement MTP head, the Vision tower, NInfer's optimized proposal head, and embedded Qwen3.6 DFlash draft weights. It is not a Safetensors or GGUF checkpoint and is not intended for Transformers, llama.cpp, or unrelated NInfer forks.

Why this upstream matters: this is the Ornith variant whose MTP component was deliberately repaired. Shisa AI replaced the native head that had underperformed with a Qwen3.6-initialized, KL-distilled head intended for working speculative decoding. This conversion retains it, and the local MTP4 test below confirmed 66.94% accepted draft tokens with zero fallback steps.

Quick facts

Item	Value
Artifact	Ornith-1.5-35B-A3B-MTP.ninfer
File size	22,783,246,080 bytes / 21.22 GiB
SHA-256	bc58fa4900d99560904bb94987704e712091a8e72a1a91d07242313631a919a3
NInfer identity	qwen3.6-35b-a3b / groupwise-int
Conversion recipe	qwen3_6_35b_a3b-v2
Direct weight source	shisa-ai/Ornith-1.5-35B-A3B-MTP@779a91e
Embedded DFlash source	z-lab/Qwen3.6-35B-A3B-DFlash@f181eec
Target runtime	UDPSendToFailed/ninfer-4090, feat/rtx-4090-sm89-native
Converter revision	f27421ab512f
Converted and tested on	Windows 11 Pro build 26200, RTX 4090 24 GB, NVIDIA 581.42, Python 3.11.13, PyTorch 2.13.0+cu130

Model provenance

The direct source keeps the target transformer, embeddings, LM head, tokenizer, and multimodal components from ornith-ai/Ornith-1.5-35B-A3B. Its native MTP head was replaced with a Qwen3.6-initialized head that Shisa AI reports was trained through full-vocabulary KL distillation after the native head underperformed. The replacement is published separately as model-mtp.safetensors and is code-heavy by design. See the source model card for training details and upstream attribution.

This NInfer conversion preserves that MTP component and additionally embeds the Apache-2.0 DFlash checkpoint from Z-Lab. DFlash targets the shared Qwen3.6 architecture rather than this exact Ornith tune. NInfer verifies every proposal with the target model; the pairing worked strongly in the bounded test below, but acceptance remains prompt-dependent.

Frontend compatibility note

NInfer validates tokenizer, processor, and chat-template resources against its registered Qwen3.6 frontend. Five source frontend resources already matched exactly. The source chat_template.jinja had SHA-256 182e77dd83bd8e9ca818b240b82e28f243762cd5dda32e6eef327df7b1cd107e, while NInfer requires e84f32a23fdda27689f868aa4a1a5621f41133e51a48d7f3efcbea2839574259.

The original template was preserved before conversion; the artifact embeds the exact NInfer-compatible Qwen3.6 template. Model weights, tokenizer data, and processor resources were otherwise retained. The Qwen3.6 product name coexisting with Qwen3_5MoeForConditionalGeneration / qwen3_5_moe identifiers is expected.

What is inside the bundle

The converter validated 1,045 BF16 target tensors across 17 source files (16 main shards plus the replacement MTP shard) and 69 BF16 DFlash tensors before writing 934 runtime tensors plus six embedded frontend resources.

Component	Stored tensor bytes
Main text model	21,038,461,952
MTP	897,934,336
DFlash	410,053,632
Vision	280,664,992
Optimized draft head	143,130,624
Total tensors	22,770,245,536

Q4G64_F16S, for example, means 4-bit weights quantized in groups of 64 with FP16 scales; Q5/Q6 use the same group size, while W8G32_F16S uses 8-bit weights in groups of 32.

BF16	FP32	I32	Q4G64	Q5G64	Q6G64	W8G32
487	60	1	95	91	5	195

NInfer loads only the components selected at startup:

Mode	Resident weights	Planned device total
Baseline text	19.59 GiB	19.80 GiB
MTP4	20.56 GiB	20.77 GiB
DFlash7	20.11 GiB	20.51 GiB
Vision + MTP4	20.82 GiB	21.04 GiB

KV state, workspaces, CUDA Graphs, and concurrency add further runtime memory. The published conversion report is path-sanitized and contains the full tensor-format and environment summary.

Download

Install the Hugging Face CLI, then download the complete repository:

hf download pyros-vault/Ornith-1.5-35B-A3B-fixed-mtp-NInfer `
  --local-dir .\Ornith-1.5-35B-A3B-fixed-mtp-NInfer

Verify the model artifact:

Get-FileHash `
  '.\Ornith-1.5-35B-A3B-fixed-mtp-NInfer\Ornith-1.5-35B-A3B-MTP.ninfer' `
  -Algorithm SHA256

Serving with the intended NInfer runtime

Build a current native sm_89 version of UDPSendToFailed/ninfer-4090. Older release binaries or other forks that predate the qwen3_6_35b_a3b-v2 target may not load this artifact.

The examples use the conservative 4,096-token profile tested on one 24 GB RTX 4090.

Text with MTP4

.\ninfer-serve.exe '.\Ornith-1.5-35B-A3B-MTP.ninfer' `
  --kv-dtype rk4v4-e8 `
  --spec mtp --draft-tokens 4 --lm-head-draft `
  --max-context 4096 --kv-capacity 4096 --prefill-chunk 512 `
  --preserve-thinking --no-cuda-graph

Text with DFlash7

.\ninfer-serve.exe '.\Ornith-1.5-35B-A3B-MTP.ninfer' `
  --kv-dtype rk4v4-e8 `
  --spec dflash --draft-tokens 7 --lm-head-draft `
  --max-context 4096 --kv-capacity 4096 --prefill-chunk 512 `
  --preserve-thinking --no-cuda-graph

Vision with MTP4

.\ninfer-serve.exe '.\Ornith-1.5-35B-A3B-MTP.ninfer' `
  --vision --vision-max-tokens 1024 `
  --kv-dtype rk4v4-e8 `
  --spec mtp --draft-tokens 4 --lm-head-draft `
  --max-context 4096 --kv-capacity 4096 --prefill-chunk 512 `
  --preserve-thinking --no-cuda-graph

The server exposes OpenAI-compatible /v1/chat/completions and /v1/responses plus Anthropic-compatible /v1/messages. Compatible-prefix reuse is enabled by default, and current builds report it as usage.prompt_tokens_details.cached_tokens.

Local verification results

These are bounded single-run smoke/integration measurements, not a benchmark suite. Text rows used the same code-generation prompt with greedy decoding, thinking disabled, rk4v4-e8 KV, a 4,096-token KV capacity, 512-token prefill chunks, and CUDA Graphs disabled.

Path	Result	Generated	Decode	Speculation
Baseline text	Correct Fibonacci function, docstring, and 3 assertions	111	191.86 tok/s	Off
MTP4 text	Correct near-identical implementation	113	383.71 tok/s	66.94% accepted; 3.68 tok/round; 0 fallback
DFlash7 text	Correct near-identical implementation	113	648.73 tok/s	87.50% accepted; 7.12 tok/round; 0 fallback
Vision + MTP4	Exact NIFER VISION 731；3；左侧	14	106.28 tok/s	40.00% accepted; 2.60 tok/round; 0 fallback

The Vision row uses a different, very short multimodal fixture and is not directly comparable to the text rows. Its 428-token expanded prompt used --vision-max-tokens 1024; preprocessing plus Vision execution took 0.066 seconds.

A separate ninfer-serve OpenAI test returned exact SERVE_OK responses. The cold request reported 0 cached tokens; repeated non-streaming and streaming requests each reported 30 of 34 prompt tokens cached (88.2%).

Single-run rates can vary with prompt, output, driver state, sampling, context, and runtime revision. They establish that all four execution paths work; they are not a universal speedup guarantee.

Important limitations

• Exact runtime matters: .ninfer is a registered, model-bound deployment format, not a generic container.
• MTP specialization: the replacement MTP head is upstream-described as code-heavy; acceptance may differ on prose, multilingual, tool, or Vision workloads.
• DFlash is text-only in NInfer: --spec dflash cannot be combined with --vision. Use MTP for multimodal speculative decoding.
• Startup capabilities are fixed: requests cannot enable MTP, DFlash, or Vision weights that the server did not load at startup.
• Only the documented 4k profile was tested here: larger contexts, higher concurrency, and CUDA Graphs need separate VRAM and stability validation.
• Vision defaults are large: NInfer defaults to 8,192 Vision scratch tokens. Start with --vision-max-tokens 1024 on a 24 GB card and increase only when required.
• No hosted inference: Hugging Face's hosted API cannot execute .ninfer files.

Conversion command

The artifact was produced with NInfer's dedicated converter:

.\.venv\Scripts\python.exe -m tools.convert.qwen3_6_35b_a3b.convert `
  --model '.\Ornith-1.5-35B-A3B-MTP' `
  --dflash-model '.\Qwen3.6-35B-A3B-DFlash' `
  --out '.\Ornith-1.5-35B-A3B-MTP.ninfer' `
  --device cuda

Conversion completed in 148.61 seconds after strict config, tensor-inventory, dtype, shape, and frontend-resource validation.

License and credits

The direct Shisa AI combined checkpoint is distributed under Apache-2.0 because its replacement MTP head derives from Qwen3.6. The retained Ornith target components remain MIT. The embedded Z-Lab DFlash component is Apache-2.0. See NOTICE and LICENSES.md for component-level attribution and pinned sources.

NInfer runtime code is not redistributed here; obtain it from its separately Apache-2.0-licensed upstream repository.

Converted, tested, and packaged by pyros-vault.