Anima INT4 quantization, size is 1.437 GB, about 63% smaller than BF16.

Simple convrot_w4a4 seems to cause a massive quality loss so this is an attempt to restore some of it while retaining the storage advantage of INT4.

The improvement is primarily from using MSE (Mean Squared Error) method when selecting scale rather than the default maximum, it allows to find a scale that minimizes quantization error as much as possible. Tests show about 43% lower weight reconstruction error related to just convrot_w4a4.

Comparison images are at the end so you can see how the model fares compared to int8_convrot and convrot_w4a4.