Model quantization based on the principle and math used ISTA-DASLab for low quants of LLM

Pure proof of concept

TL:DR

The custom int8 x bf16 is very close to base model for only 2.4GB

The custom GGUF is the best I've manage to get before quality take a hit and lose to closeness to base model.

Based on ISTA-DAS labs Non-uniform GGUF quantizations via GSQ + RCO: per-tensor mixed precision in standard GGUF form.

There is also a int8 convrot created using the same pattern, where a handfull of tensor have been "buffed" to BF16.

Both models went throught some overnight gpu compute to be done and manual curating by me.

I post here the 2 models I think are the closest to base model at a fraction of the weight.

It's mostly experimental, can be usefull for 2 class of people

• User with "old" 4gb or 6gb gpu, This give some headroom for lora stacks and higher res and batchs

• User with massive workflow who want everything to in VRAM.

• People like me who just want to try ^