Quantized model available?

#1
by dieselheart01 - opened

Is there any quantized model?

for the most part I leave that to others. Quantizing introduced risk of security threat and massive undermining of the model. So I leave up to the user to do. this model would be easier than most becusse of the structure. Check out unsloth or just pure nvidia tooling depending on your gpu. If you use vllm and want to cut the size in almost 1/2 on the fly just add -q fp8

Sign up or log in to comment