tcclaviger/Minimax-M2-Thrift-GPTQ-W4A16-AMD

Minimax-M2-THRIFT (W4A16, GPTQ) — CPU-quantized with llm-compressor and tested on 4× Radeon AI Pro 9700s

An independant community created quantizer version which is fully-credited fork of VibeStud.io's THRIFT README with image references removed, quantization/provenance details added, VLLM service configuration, and notes on OpenWebUI thinking tag preservation.

I have been using this as a claude code provider via liteLLM without a single tool call failure. Thus far all functionlity presists, tool calls are correct, resume works, plan mode builds plans, and the exploration and code editing has been very strong. Still evaluating what works and doesnt in CC with this behind it, but highly satisfied thus far.

EDIT: Using as a CC LLM it works well, you will need to write your own web_search_20250305 serverside routine to:

• Identify the strangely shaped web_search_20250305 server action request

• Get a list of sites

• Crawl those results you deem appropriate

• inject that content into the conversation

• Return the informed response as the resposne to the web_search_20250305 request.

  All other CC MCP tools functions work correctly and consistently, bash, edit, fetch, pyhton code execution, etc.

Below is content from the original model.

TL;DR

We, over-caffinated researchers at VibeStud.io wanted to create a 50% pruned version of the SOTA MiniMax M2 that is best suited for local/air-gapped coding. This version we achieved ~25%. A 50% pruned version is under development while a not so sucky team of ours is working on a 50% pruned version of Kimi K2 Thinking.We're writing the paper and expanding the evaluation set to substantiate the results. Check back later, cheers!

Why it's useful

• Lower latency: Snappier responses for interactive apps and chatbots.
• Smaller memory footprint: Runs on cheaper GPUs or with fewer resources per replica.
• Higher throughput: Serve more concurrent users at the same cost.
• Deployment-friendly: Drop-in replacement for the base model in most inference stacks.
• Adaptable: Supports light fine-tuning to match your domain and style guidelines.

Intended use

• General chat and coding assistance
• Enterprise assistants with strict latency/VRAM budgets
• Batch or realtime serving in cloud and on-prem environments
• Edge or cost-sensitive deployments where efficiency matters

When to use it

• You're constrained by GPU memory or need shorter response times
• You want to increase QPS without scaling infrastructure
• You need a model that is "good enough" for most tasks at a better cost profile

---

Model Comparison Report

Models Under Evaluation

Model	Type
ModelCloud/MiniMax-M2-BF16	Base Model
VibeStudio/MiniMax-M2-THRIFT	Compressed/Optimized

Evaluation Dates: November 7–9, 2025

📊 Results Comparison

1) Multiple Choice Q&A (lm-eval)

Overall MMLU Performance

Model	MMLU Overall	Humanities	STEM	Social Sciences	Other
MiniMax-M2-BF16	83.16%	77.45%	80.91%	90.02%	87.29%
MiniMax-M2-THRIFT	77.72%	70.14%	77.61%	86.84%	80.27%
Δ (Difference)	-5.44%	-7.31%	-3.30%	-3.18%	-7.02%

Individual Task Performance

Task	BF16 (Base)	THRIFT-BF16	Difference
arc__challenge (acc__norm)	73.21%	61.01%	-12.20% ⬇️
arc__easy	88.30%	83.08%	-5.22% ⬇️
boolq	87.95%	84.95%	-3.00% ⬇️
hellaswag (acc__norm)	83.00%	77.09%	-5.91% ⬇️
mmlu	83.16%	77.72%	-5.44% ⬇️
openbookqa (acc__norm)	48.60%	43.00%	-5.60% ⬇️
rte	75.45%	80.14%	_+4.69% ⬆️
winogrande	76.48%	74.90%	-1.58% ⬇️

Average Accuracy Drop: -4.28%

2) Code Generation (EvalPlus)

MBPP Results (Python, 378 problems)

Model	MBPP (base)	MBPP+ (extended)	Average
MiniMax-M2-BF16	73.8%	64.0%	68.9%
MiniMax-M2-THRIFT	70.1%	60.1%	65.1%
Δ (Difference)	-3.7%	-3.9%	-3.8%

HumanEval Results (164 problems)

Model	HumanEval (base)	HumanEval+ (extended)	Average
MiniMax-M2-BF16	72.6%	71.3%	72.0%
MiniMax-M2-THRIFT	65.2%	63.4%	64.3%
Δ (Difference)	-7.4%	-7.9%	-7.7%

3) Math Benchmarks

GSM8K Results

Model	Accuracy	Problems	Status
MiniMax-M2-BF16	92.72%	1,319	✅ Complete
MiniMax-M2-THRIFT	93.25%	1,319	✅ Complete
Δ (Difference)	+0.53% ⬆️	-	THRIFT Better! ✨

Benchmark	MiniMax-M2-BF16	MiniMax-M2-THRIFT	Change
GSM8K	92.72%	93.25%	+0.53% ⬆️
MATH-500 (Levels 1-4)	91.25%	90.75%	-0.5% (near-parity)

4) LiveCodeBench (Live Coding Problems)

Model	pass@1	Problems	Status
MiniMax-M2-BF16	35.71%	182	✅ Complete
MiniMax-M2-THRIFT	36.81%	182	✅ Complete
Δ (Difference)	+1.10% ⬆️	-	THRIFT Better! ✨

---

📈 Analysis (Updated)

Highlights

• THRIFT wins on GSM8K (+0.53%) and LiveCodeBench (+1.10%), and on RTE (+4.69%).
• BF16 leads on broad MMLU, HumanEval, MBPP, and tasks like arc_challenge.

Compression Trade-off Average knowledge-task drop for THRIFT is ~4–5%, with math preserved or slightly improved.

Subject Breakdown (MMLU)

Category	BF16 (Base)	THRIFT-BF16	Difference	Status
High School Government	97.93%	94.82%	-3.11%	✅ Still Excellent
High School Psychology	95.41%	93.58%	-1.83%	✅ Well Preserved
Marketing	95.73%	91.88%	-3.85%	✅ Good
Professional Medicine	92.28%	79.78%	-12.50%	⚠️ Notable Drop
Clinical Knowledge	92.83%	85.66%	-7.17%	⚠️ Moderate Drop

---

sglang Deployment with Python

It is recommended to use a virtual environment (such as venv, conda, or uv) to avoid dependency conflicts.

We recommend installing SGLang in a fresh Python environment:

git clone -b v0.5.4.post1 https://github.com/sgl-project/sglang.git
cd sglang

# Install the python packages
pip install --upgrade pip
pip install -e "python"

Run the following command to start the SGLang server. SGLang will automatically download and cache the MiniMax-M2 model from Hugging Face.

4-GPU deployment command:

python -m sglang.launch_server \
--model-path MiniMaxAI/MiniMax-M2 \
--tp-size 4 \
--tool-call-parser minimax-m2 \
--reasoning-parser minimax-append-think \
--host 0.0.0.0 \
--trust-remote-code \
--port 8000 \
--mem-fraction-static 0.85

8-GPU deployment command:

python -m sglang.launch_server \
--model-path MiniMaxAI/MiniMax-M2 \
--tp-size 8 \
--ep-size 8 \
--tool-call-parser minimax-m2 \
--trust-remote-code \
--host 0.0.0.0 \
--reasoning-parser minimax-append-think \
--port 8000 \
--mem-fraction-static 0.85

VLLM Deployment (Docker SystemD Unit) Verified with 170k context across 4× Radeon AI Pro 9700s using the following VLLM configuration. Use as-is or adapt paths for your environment.

+[Unit]

Description=VLLM Minimax-M2 Inference Service

After=docker.service

Wants=docker.service

[Service]

Type=simple

Restart=always

RestartSec=10

ExecStartPre=/usr/bin/docker rm -f vllm-MMM2

ExecStart=/usr/bin/docker run --rm --name vllm-MMM2 \
    --network=host \
    --group-add=video \
    --ipc=host \
    --shm-size=16gb \
    --cap-add=SYS_PTRACE \
    --security-opt seccomp=unconfined \
    --privileged \
    --device /dev/kfd \
    --device /dev/dri \
    -e OMP_NUM_THREADS=16 \
    -e PYTORCH_TUNABLEOP_ENABLED=1 \
    -e PYTORCH_TUNABLEOP_TUNING=0 \
    -e PYTORCH_TUNABLEOP_FILENAME=/tunableop/tunableop_results%%d.csv \
    -e VLLM_SLEEP_WHEN_IDLE=1 \
    -v /<yourpath>/MiniMax-M2-THRIFT-W4A16-G128:/app/models \
    -v /<yourpath>/vllm-tunableop/2025-11-08:/tunableop \
    <yourdockerimage> \
    /app/models \
    --tensor-parallel-size 4 \
    --quantization compressed-tensors \
    --trust-remote-code \
    --tool-call-parser minimax_m2 \
    --reasoning-parser minimax_m2_append_think \
    --enable-auto-tool-choice \
    --max-num-seqs 4 \
    --max-model-len 170000 \
    --max-num-batched-tokens 2048 \
    --enable-chunked-prefill \
    --gpu-memory-utilization 0.95 \
    --kv-cache-dtype fp8 \
    --host 0.0.0.0 \
    --port 8078 \
    --served-model-name Minimax-M2 \
    --override-generation-config '{"temperature": 1.0, "top_p": 0.95, "top_k": 40}'

ExecStop=/usr/bin/docker stop vllm-MMM2

[Install]

WantedBy=multi-user.target

On the 4x9700 setup I currently am contrained by the kernel software, not bandwidth in most cases, typical decode is at ~35tps, without tunablop setup you will see ~15% lower, prefill phase is much faster without kv-cahce-dtype, so if you can satisfy your cache size needs without quantizing the kv cache, I recommend doing so.

Note: For OpenAI-compatible API consumption with tool/think tags, consider using the included robaithinkingproxy to preserve sections upstream of VLLM when using frontends that strip them (e.g., OpenWebUI).

Testing Deployment After startup, you can test the VLLM OpenAI-compatible API with the following command:

curl http://localhost:8078/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "Minimax-M2",
  "messages": [
    {"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]},
    {"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]}
  ]
}'

Benchmarks See the tables above for the latest MMLU, MBPP, HumanEval, GSM8K, MATH-500, and LiveCodeBench results (updated November 9, 2025).

Research paper Coming soon.

License This model is derived from MiniMax-M2 and distributed under the MIT License http://github.com/MiniMax-AI/MiniMax-M2/blob/main/LICENSE

Credits Base Model: MiniMaxAI/MiniMax-M2 THRIFT optimization and evaluation: VibeStud.io Model conversion and HF Transformers code: @Qubitum at ModelCloud Quantization: Rob P Smith — quantized with llm-compressor to GPTQ W4A16 on CPU (Intel 9950X3D, 256 GB RAM) and tested with a 170k-token context on 4× Radeon AI Pro 9700s using the VLLM configuration above Thinking tag proxy for OpenWebUI: included under robaithinkingproxy/ in this repo Deployment notes GPU memory tuning: set --gpu-memory-utilization and --max-num-batched-tokens to fit your workload. Context length: --max-model-len 150000 is used in the example; adjust carefully to avoid OOM. Tool and reasoning parsers: --tool-call-parser minimax_m2 and --reasoning-parser minimax_m2_append_think enable MiniMax tool/think semantics. If using OpenWebUI: frontends may strip ; deploy robaithinkingproxy in front of VLLM to restore tags on every request. References (BibTeX)

@article{cai2025thinking,
title        = {Thinking with DistilQwen: A Tale of Four Distilled Reasoning and Reward Model Series},
author       = {Cai, Wenrui and Wang, Chengyu and Yan, Junbing and Huang, Jun and Fang, Xiangzhong},
journal      = {arXiv preprint arXiv:2511.01354},
year        = {2025},
eprinttype   = {arXiv},
eprint       = {2511.01354},
primaryclass = {cs.CL},
institution  = {Shanghai Jiao Tong University and Alibaba Cloud Computing},
note         = {License: arXiv.org perpetual non-exclusive license}
}

@misc{lasby-reap,
title       = {{REAP the Experts: Why Pruning Prevails for One-Shot MoE compression}},
author      = {Lasby, Mike and Lazarevich, Ivan and Sinnadurai, Nish and Lie, Sean and Ioannou, Yani and Thangarasa, Vithursan},
year        = {2025},
publisher   = {arXiv},
note        = {arXiv:2510.13999v1 [cs]},
url         = {https://arxiv.org/abs/2510.13999v1},
}

@article{yang2025wanda++,
title        = {Wanda++: Pruning Large Language Models via Regional Gradients},
author       = {Yang, Yifan and Zhen, Kai and Ganesh, Bhavana and Galstyan, Aram and Huybrechts, Goeric and Müller, Markus and Kübler, Jonas M. and Swaminathan, Rupak Vignesh and Mouchtaris, Athanasios and Bodapati, Sravan Babu and Susanj, Nathan and Zhang, Zheng and FitzGerald, Jack and Kumar, Abhishek},
journal      = {arXiv preprint arXiv:2503.04992},
year        = {2025},
eprinttype   = {arXiv},
eprint       = {2503.04992},
primaryclass = {cs.CL}
}

@article{li2025tyr,
title        = {Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization},
author       = {Li, G. and Xu, Yixing and Li, Zeping and Liu, Ji and Yin, Xuanwu and Li, Dong and Barsoum, Emad},
journal      = {arXiv preprint arXiv:2503.09657},
year        = {2025},
eprinttype   = {arXiv},
eprint       = {2503.09657},
primaryclass = {cs.CL}
}

@article{xia2023sheared,
title        = {Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning},
author       = {Xia, Mengzhou and Gao, Tianyu and Zeng, Zhiyuan and Chen, Danqi},
journal      = {arXiv preprint arXiv:2310.06694},
year        = {2023},
eprinttype   = {arXiv},
eprint       = {2310.06694},
primaryclass = {cs.CL}
}

@article{ma2023llmpruner,
title        = {LLM-Pruner: On the Structural Pruning of Large Language Models},
author       = {Ma, Xinyin and Fang, Gongfan and Wang, Xinchao},
journal      = {arXiv preprint arXiv:2305.11627},
year        = {2023},
eprinttype   = {arXiv},
eprint       = {2305.11627},
primaryclass = {cs.CL}
}

@article{yang2023wanda,
title        = {Wanda: Pruning by Weights and Activation-based Discriminant Analysis},
author       = {Yang, Yifan and Ganesh, Bhavana and Galstyan, Aram and Huybrechts, Goeric and Müller, Markus and Kübler, Jonas M. and Swaminathan, Rupak Vignesh and Mouchtaris, Athanasios and Bodapati, Sravan Babu and Zhang, Zheng and FitzGerald, Jack and Kumar, Abhishek},
journal      = {arXiv preprint arXiv:2306.11695},
year        = {2023},
eprinttype   = {arXiv},
eprint       = {2306.11695},
primaryclass = {cs.CL}
}

@article{frantar2023sparsegpt,
title        = {SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot},
author       = {Frantar, Elias and Alistarh, Dan},
journal      = {arXiv preprint arXiv:2301.00774},
year        = {2023},
eprinttype   = {arXiv},
eprint       = {2301.00774},
primaryclass = {cs.CL}
}

@article{dettmers2023qlora,
title        = {QLoRA: Efficient Finetuning of Quantized LLMs},
author       = {Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke},
journal      = {arXiv preprint arXiv:2307.02973},
year        = {2023},
eprinttype   = {arXiv},
eprint       = {2307.02973},
primaryclass = {cs.CL}
}