Please re-download int8 version. I have uploaded much faster, proper version.
Versions
int8: recommended. Fast, accurate, compatible with almost any GPU.
mxfp8: added for comparison. In theory (and according to nVidia PR) should be more accurate than int8, but in practice I was not able to spot any definitive advantages. A bit slower than int8, but still faster than original bf16. Compatible only with RTX 50xx series (Blackwell).
Performance on my setup
original bf16 (baseline): 2.20 it/s +0%
int8: 3.23 it/s +46%
int8 + torch compile (comfy core): 3.59 it/s +63%
mxfp8: 2.58 it/s +17%
This is high quality int8 quantized version of WAI-ANIMA model. It retains ~90% of original model quality, but uses about 50% less VRAM and also runs faster on almost any nVidia GPU (AMD not tested). Nice trade-off, especially for low-end GPUs.
Can be used as a drop-in replacement for original model in latest ComfyUI, no custom nodes required. If you have troubles running the model make sure that you updated both ComfyUI itself and its dependencies (e.g.pip install -U -r requirements.txt on manual linux install).
Converted to int8 / mxfp8 using convert_to_quant script.
Description
int8, ConvRot group size 256, rowwise, learned rounding SVD
Proper, fast version.
FAQ
Comments (6)
thanks! it's much faster than the experimental version. however, for some reason, the compressed model is always 1% slower than the native bf/fp16 model quantized to int8 convrot on-the-fly...
@jancok In this version some layers are not quantized to preserve the quality. If the list of layers is different it may explain difference in performance. But idk, maybe there are some other diffs in quant settings.
@somedoby yep..many confirmed that pre-compressed model have slower inference speed than on-the-fly quant
somehow int8 run slower than mxfp8 (and original) on my end, up to +50% of time needed per image. On the other hand mxfp8 perform on par with original, slightly faster too.
@madLad31 nVidia or AMD? Also make sure you have the latest model version downloaded. Previous (now removed) "experimental int8" version was slow.
me too, I just downloaded it and it's running slower than the normal version; I have an RTX 3070.