Thanks for this
I've been wanting to mess around the quanting for this one. Thanks for making it available. Have you tried the mlxoptiq package or the oQ4 that oMLX does at all? If so, your thoughts?
No, to be honest I have so limited space in my lab, that I am playing musical chairs with what to delete today.
There is simply no room to do other stuff than the standard tests I do :)
This model is the stablest form of high-arc gemma I could cook, given my previous experience with this platform. David's E4B models are great by themselves, but in a NuSLERP they create the perfect environment for gemma to play Alice in Wonderland. And gemma loves Alice, and would much rather be at the Queen's party than play inference with you.
With the extra models in the configuration I created a bit of a Möbius loop in the inference, that bounces between models just enough to prevent that.
Space is always an issue with these hungry beasts. I'll report back anything interesting I find in my quanting. Appreciate yours and David's work. Keep cooking :)
I updated the jinja with the template I used in testing, and enable think mode.