Experimental INT4 GGUF of Krea 2 Turbo offering decent visual output quality, 42% reduction in model size (7GB) and 35-59% performance improvement over INT8 safetensors.
IMPORTANT!
Please update the ComfyUI GGUF loader nodes (this fork: https://registry.comfy.org/publishers/molbal/nodes/comfyui-gguf-reboot) to load this properly. You will need version v28.09.04 or newer and use the Unet loader (GGUF, Dynamic VRAM) node to load it.
INT4 GGUF speed on my Windows and Linux machines, compared to INT8 ConvRot safetensors:
(Numbers, hardware, and testing setup in the article)
More information in the information page here: https://molbal.github.io/gguf/ecosystem/quant-formats.html#q4-cr-int4-convrot
Description
FAQ
Comments (12)
Hello,
does this require specific hardware? Using the Unet loader I get the error "ValueError: Unexpected architecture type in GGUF file: 'krea2'"
Im on an older 1080TI, so that might be an issue?
Hi, I have not tried it with a 1080Ti, but make sure you use the latest comfyui-gguf nodes (not city96, but this one: https://registry.comfy.org/publishers/molbal/nodes/comfyui-gguf-reboot )
Should we be using "Unet Loader (GGUF/Advanced)" or "Unet Loader (GGUF, Dynamic VRAM)"?
With the first node I see a slight improvement in speed over int8 (1.28it/s over 1.02it/s) but there is a long delay each time as the model loads (even though it says "loaded completely").
With the second node the model stays loaded, but the speed is the same as int8.
Use the dynamic VRAM one. That loading should be one time only and only if you use LoRAs. So if you use the same set of LoRAs and tweak the prompt then it should not do the loading.
That improvement of ~25-30% is the benefit that should be it.
May I ask how many LoRAs you use and what is your GPU?
I hope you'll make a Moody or Redcraft version.
Here you go, Moody V7: https://civitai.red/api/download/models/3223042?fileId=3187566
But you don't have to wait for me, if you have the loader nodes, you can make a conversion following this guide: https://molbal.github.io/gguf/ecosystem/quantizing-models.html :)
@molbal thanks you so much
@FFrince You're welcome!
Are your graphs saying that image generation becomes faster the more LoRAs you use?
No, both on Windows and Linux, both for INT8 safetensors and INT4 GGUF, the more LoRAs you use, the slower the generation. This is because INT4 GGUFs apply LoRAs in a differently (part of why I call it experimental).
However, there are 2 differences how the INT4 GGUF handles LoRAs compared to everything else:
- Adding a LoRA to an INT4 GGUF converted model slows down image generation less, than adding a LoRA to another model (be it any safetensors or any other format GGUF) <- this is the plus side
- Adding a LoRA to any model with the load lora node uses more memory (VRAM or unified memory on Mac). If you use an INT4 GGUF, then the additional memory need per LoRA is slightly higher, than the additional memory need per LoRA in other model formats. <- this is the tradeoff
To mitigate the tradeoff, you can take a safetensors model, and merge the loras in the model while converting it to INT4 GGUF, that will mean your target GGUF fill will already contain all LoRAs and you will not have any VRAM or performance overhead. However, that way you will have a multiple-gigabyte model for each combination. You can still apply further LoRAs to a model, which had other LoRAs fused into it.
More info on converting with LoRAs: https://molbal.github.io/gguf/ecosystem/quantizing-models.html#merging-loras-during-conversion
I hope I could clear things up rather than further confuse you :)
I was talking about your graphs, both of which clearly indicate that the speed INCREASES. not decreases.
@civitai3463 ah, I see what you mean now. I should have labeled the vertical axis better, it is indeed wrong, you are right.
Here are the exact it/s numbers of the tests I've based the charts on: https://substack-post-media.s3.amazonaws.com/public/images/7e384680-54d6-4c45-97a5-536db30744bd_1717x702.png
The Windows and Linux results are not cross-platform comparisons. Each section compares Q4_CR against INT8 ConvRot safetensors on the same platform. The workflows are slightly different on windows and linux, because my linux machine is a desktop with as 3090 and my laptop has a 3080 and runs windows
