why is the nvfp4 model so large?
it's the same size as int8. am i missing something or being stupid?
It is a mix of fp8, bf16 and nvfp4 , maybe I could be more clear in the naming. The convrot int8 is int8, bf16, nvfp4. The fp8 and int8 will be the same size.
I did that one first so now I would describe the mixing more clearly. It should be fairly identical to the nvfp4_fp8 version i put up.
Others have quantized nvfp4 versions exist that will quantize more or all the tensors into nvfp4 for the fastest, lowest memory options. For example under https://huggingface.co/lilcheaty/MiniMax-H3-NVFP4 has a nvfp4 12GB quantization.
My attempt was to get a quant that has lower error while utilizing nvfp4 on the less sensitive components. Further improvements to this could be possible as I went a little more conservative.