Instructions to use coolthor/Qwen3.8-27B-Uncensored-NVFP4-FastLLM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use coolthor/Qwen3.8-27B-Uncensored-NVFP4-FastLLM with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="coolthor/Qwen3.8-27B-Uncensored-NVFP4-FastLLM")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("coolthor/Qwen3.8-27B-Uncensored-NVFP4-FastLLM")
model = AutoModelForMultimodalLM.from_pretrained("coolthor/Qwen3.8-27B-Uncensored-NVFP4-FastLLM", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use coolthor/Qwen3.8-27B-Uncensored-NVFP4-FastLLM with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "coolthor/Qwen3.8-27B-Uncensored-NVFP4-FastLLM"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "coolthor/Qwen3.8-27B-Uncensored-NVFP4-FastLLM",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/coolthor/Qwen3.8-27B-Uncensored-NVFP4-FastLLM

SGLang

How to use coolthor/Qwen3.8-27B-Uncensored-NVFP4-FastLLM with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "coolthor/Qwen3.8-27B-Uncensored-NVFP4-FastLLM" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "coolthor/Qwen3.8-27B-Uncensored-NVFP4-FastLLM",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "coolthor/Qwen3.8-27B-Uncensored-NVFP4-FastLLM" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "coolthor/Qwen3.8-27B-Uncensored-NVFP4-FastLLM",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use coolthor/Qwen3.8-27B-Uncensored-NVFP4-FastLLM with Docker Model Runner:
```
docker model run hf.co/coolthor/Qwen3.8-27B-Uncensored-NVFP4-FastLLM
```

Qwen3.8-27B-Uncensored-NVFP4-FastLLM

An NVFP4 build of orcarouter/Qwen3.8-27B-Uncensored with every linear layer in NVFP4. It is 20.6 GB, down from 55.6 GB in BF16 and 29 GB in FP8.

I made it to serve the model on two RTX 2080 Ti 22 GB cards with FastLLM. Turing has no FP4 hardware. FastLLM stores the NVFP4 weights as-is and unpacks them to FP16 inside the kernel. Decoding is limited by weight reads, so the smaller file is faster on these cards even with the extra unpacking.

Write-up with the full numbers, the launch command and the debugging notes: NVFP4 Without FP4 Hardware: Qwen3.8-27B on Two RTX 2080 Tis at 151.7 tok/s (中文: 2080 Ti 沒有 FP4 也能跑 NVFP4).

⚠️ Inherited disclaimer

The base model has had its safety alignment largely removed by abliteration. Quantizing it does not change that. Everything in the base model's disclaimer applies here. The model will comply with harmful requests. It is meant for research and controlled use, you are responsible for how you use it, and you should add your own moderation before exposing it to anyone else.

What's in it

Weights	Format
All linear layers: attention q/k/v/o, GDN projections (`in_proj_qkv/z/a/b`, `out_proj`), MLP	NVFP4 (E2M1 values, one FP8-E4M3 scale per 16 values, one FP32 global scale per tensor)
`lm_head`, `embed_tokens`, vision tower, `conv1d`, MTP head	BF16, unchanged

The layout matches lyf/Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL exactly: tensor names, dtypes, shapes, shard map, quantization_config and hf_quant_config.json (2,687 tensors, total_size 20,558,935,392). Only the base model differs.
As in ModelOpt's recipe, q/k/v share one global scale per layer, the four GDN input projections share one, and MLP gate/up share one.
No calibration: input_global_scale is set to 1.0. FastLLM on sm_75 computes with FP16 activations and does not read it. If you run this on an engine that does W4A4 with real activation scales, expect worse results than a calibrated checkpoint.

How it was made

I converted it with a numpy-only script that runs on the CPU, no GPU needed: qwen38-nvfp4-convert.py.

hf download orcarouter/Qwen3.8-27B-Uncensored --local-dir ./bf16
hf download lyf/Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL --include "*.json" "recipe.yaml" --local-dir ./nvfp4-ref
python3 qwen38-nvfp4-convert.py ./bf16 ./out --reference ./nvfp4-ref

To check the converter, I dequantized one attention, one GDN and one MLP layer of lyf's release to BF16 and quantized them again. The packed weights, block scales and global scales came out byte-identical to lyf's.

Results on two RTX 2080 Ti 22 GB (TP=2, FastLLM + DFlash2)

Same machine, same FastLLM build, temperature 0, 512 tokens, median of 3 runs, tok/s:

	code	math	prose	no speculation (code)
orcarouter FP8	116.1	135.5	74.2	32.9
this NVFP4	151.7	178.8	101.9	45.1

With fp8 KV cache and --max_batch 4: 2 concurrent streams run at 107.8 tok/s each, 4 streams at 74.3 each (about 297 combined).

140-question eval (GSM8K 50, MATH-500 30, MuSR 20, HumanEval+ 40):

	no thinking	thinking (effort low)
orcarouter FP8	132	127
this NVFP4	131	128

Trade-off: prefilling a 54K-token prompt takes 50.6 s instead of 40.5 s. Prefill is compute-bound, so the unpacking is pure overhead there.

I have only tested this with FastLLM on sm_75. I have not tested vLLM, SGLang or Blackwell GPUs.

Serving with FastLLM

Tested on FastLLM at upstream a2bf07fd with d4b04876 reverted and PRs #749 and #756 merged (build steps in the previous post). The draft model is z-lab/Qwen3.8-27B-DFlash2.

export CUDA_VISIBLE_DEVICES=0,1
export FASTLLM_CUDA_GRAPH=0
export FASTLLM_DRAFT_QUANT=nvfp4
export FASTLLM_COOPERATIVE_LONG_PREFILL=1   # from PR #756; leave out on stock FastLLM

ftllm server -p ./Qwen3.8-27B-Uncensored-NVFP4-FastLLM \
  --tp 2 --max_batch 4 --chunked_prefill_size 4096 \
  --gpu_mem_ratio 0.95 --kv_cache_dtype fp8_e4m3 \
  --speculative_algorithm dflash \
  --speculative_draft_model_path ./Qwen3.8-27B-DFlash2 \
  --speculative_num_draft_tokens 8 \
  --prefix_cache true --port 8080

Don't pass --tokens: with it set, FastLLM clamps --max_batch back to 1 for this model.

Credits

Base model: orcarouter/Qwen3.8-27B-Uncensored, derived from Qwen/Qwen3.8-27B. Apache 2.0.
Layout reference: lyf/Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL.
Inference engine: FastLLM.

Downloads last month: 50

Safetensors

Model size

28B params

Tensor type

BF16

Model tree for coolthor/Qwen3.8-27B-Uncensored-NVFP4-FastLLM

Base model

Qwen/Qwen3.8-27B

Finetuned

orcarouter/Qwen3.8-27B-Uncensored

Quantized

(61)

this model