27/05 : Updated with SVD INT4 Quant for Flux https://github.com/mit-han-lab/ComfyUI-nunchaku
Special thanks @theunlikely for the work done on quantification (took 6 hours on a H100 GPU), and thanks to JIB (J1B Creator Profile | Civitai) for his WF and clarifications on using this type of model:
For using this model you need to install the nunchaku project following the instructions there. and the nunchaku ComfyUI custom nodes to get it to work. Download and unzip the archive from Civitai to: \comfyui.git\app\models\diffusion_models\svdq-int4-flux-dev-de-distill - A Nunchaku workflow you can use : https://civarchive.com/models/617562
This is a repost from hugging faces, I did not play any role at all in the creation of this epic, groundbreaking model from nyanko7. I just had to post it asap because it is F A N T A S T I C
Flux.dev as we have known until now was a distilled model, meaning it was trained by flux.pro as its teacher. These new models change everything ! This is the first experimental, effectively de-distilled version of Flux.dev, meaning it is much closer to what flux.pro is capable of. And it's just the start !
(NB : This is not stating it was trained with flux.pro - I don't know the exact method)
/!\ READ ALL IMPORTANT STUFF BELOW (AFTER THE EXAMPLES) IN THIS DESCRIPTION OR IT WILL BE A FAIL FOR YOU /!\
EXAMPLES MADE MYSELF
Distilled CFG 8 VS Real CFG 8 - Fixed seed, no prompt change except for the text
Each picture is the first shot on each model : no cherrypicking / cheating
Ear gauge LoRa : "Cute blonde raver young girl smiling, facing viewer, with emo makeup and puffy emo hairstyle, green shiny colored hairstyle. 3argauge, both ears have Ear gauge plug with very large circular hole in her lobe. She is wearing a black hoodie with text "DEDISTILLED MAKES MY LORAS WORK" in golden letters
Cum on face Lora (with dedistilled it actually works anywhere) : "COF, Young woman with white sticky cum on face,white sticky semen on face,white sticky sperm on face,face covered with white sticky cum, face covered with white sticky semen, face covered with white sticky sperm. She is wearing a black hoodie with text letters "DISTILLED" in silver font on it"
No LoRa : "Dutch shot view of a black car speeding away from a massive explosion in the streets of a futuristic city with large buildings, towards the viewer, escaping the blast of the explosion. The motion lines around the car give the impression of speed. The front plate of the car has text "DISTILLED" on it."
No Lora : "A small kitten playing with a ball of yarn is seen through an old wooden window of a rustic house. The scene is cozy, with weathered wooden furniture inside the house and soft afternoon light streaming through the window, casting gentle shadows. Outside, in the distance, a photographer is approaching, camera in hand, ready to capture the playful moment of the kitten. The photographer is wearing a brown jacket and is framed by the soft glow of the golden hour light, adding a sense of warmth and tranquility to the scene. The overall atmosphere is peaceful, with a touch of nostalgia from the vintage setting."
SUMMARY OF THE IMPORTANT STUFF (aka current knowledge about it)
Disclaimer : These models are very new. So, just gathering here what is known about them for now. Please share your experiences in the comment section so we can update this together.
PARAMETERS
You can now forget about Distilled CFG and use real CFG (I've tried up to like 14)
NEVER, ever use it with CFG = 1 - It will automatically be a complete disaster, and most of the time the reason why you don't get results
You SHOULD use at least 40 - 60 steps, depending on the CFG you use.
It will be much longer but SO worth it
Unfortunately the current hyperdev 8-steps Lora doesn't seem to work with it to reduce steps
Dedistilled allows NEGATIVE PROMPTS
BENEFITS
Prompt adherence will be EXCEPTIONAL, even with Loras.
Faces Loras will work better, details will be better, text will be WAY better...
Everything from the prompt will be better basically
/!\ IF YOU DON'T SEE ANY IMPROVEMENT FROM DISTILLED MODELS : VERIFY YOU ARE NOT ACTUALLY USING REAL CFG = 1 WITHOUT KNOWING. NOT FLUX GUIDANCE. IT IS TRICKY /!\
AS ALL WORKFLOWS WERE TAILORED FOR DISTILLED, CONSIDER TRYING WITH FORGE IF IT'S NOT WORKING FOR YOU IN COMFY ?
GUIDELINES FOR USE IN FORGE UI
Works in Forge without any change. Will be loaded as if it was Schnell model, disabling Distilled CFG (cool).
EDIT : uploaded all the new Quants, you should find at least one working for you
If you are new to Forge, make sure you use similar settings :Flux workflow
DeDistilled as the checkpoint
In VAE / Text encoder files, provide the vae (ae.sft / ae.safetensors) + clip_l (or a modified clip) + t5xxl (whatever the quant you are using, fp16, fp8, etc).
Otherwise it won't work as they are NOT bundled in the model files herePlease also set Diffusion in Low Bits = Automatic (FP16 Lora), otherwise you might be in trouble with LoRas. This applies to any checkpoint in Forge.
I then recommend these settings for Dedistilled :
GUIDELINES FOR USE IN COMFY UI
Works in ComfyUI using a pretty standard workflow, the one cited below uses the GGUF Loader, Dual CLIP Loader for t5xxl and clip_l prompts, and KSampler Efficient
Recommended settings for Comfy :
Dual CLIP Loader guidance: 3.5 KSampler cfg: 2 to 10 Steps: 50 to 60 Negative Prompt: Can be left blank or can be provided if needed, will affect the image if provided
Sampler: DDIM or euler Scheduler: beta or exponential
WORFKLOW HERE : https://gist.github.com/dasilva333/87bdd5b5b8ebba5515a9919ede0e3c05
Found this one also on reddit (drag & drop it into Comfy) : https://files.catbox.moe/y99yl7.png
TRAINING AND LORAS
I have just trained myself my first LoRa using De-distilled and guidance = 6, after failing hard with Distilled and guidance = 1. Results are awesome, it basically saved my LoRa. Works great with both De-distilled and distilled (but better with De-distilled).
I will be using it to train from now on.
The first checkpoint fined tune with dedistilled has been posted on civitai here : https://civarchive.com/models/690991/sapianf-nude-men-and-women-for-flux-now-de-distilled
Awaiting answers from the author to update here
Sources
Dedistilled model FP16 : https://huggingface.co/nyanko7/flux-dev-de-distill
Dedistilled model FP8 : https://huggingface.co/MinusZoneAI/flux-dev-de-distill-fp8/tree/main
Dedistilled FP8 GGUG & Q4_ K_M GGUG : https://huggingface.co/TheYuriLover/flux-dev-de-distill-GGUF/tree/main
Description
FAQ
Comments (20)
please share a simplest nunchaku workflow,my workflow get nothing :(
use this workflow https://civitai.com/models/686814?modelVersionId=1595633
it's too complex to follow...
@vinque it works!thank you!
the nunchaku ,I tried all the workflow mentioned here, NOTING BUT A BLACK PICTURES.
Just learn how to use it PROPERLY...
Hah I was too optimistic.
This specific checkpoint is broken indeed.
But, it does work, if you select fp16 mode in model loader, images look like weird shit though.
All other existing checkpoints work perfectly fine in my workflow.
Also I updated Nunchaku package up to newest version 30.
This SVDQuant conversion is 100% broken.
But more interesting - it randomly works.
Normal SVD checkpoints work with CFG 1.0 + Flux CFG 3.5, but this one works with normal CFG 6.0 (same as SDXL) and Flux CFG 1.0.
Author is obviously messed up something, because in my workflow:
- Original Flux dev svd - Fine
- Jib mix - fine
- CreArt - fine
- Project0 - fine
- This one - with issues, sometimes black image, sometimes, colored checkmate pattern, sometimes normal image. maybe we all idiots,, idk.
PS - No setting env variable will not fix black images. It useless.
NUNCHAKU_LOAD_METHOD=READNOPIN
maybe it will fix other cases but not this specific one.
It's dedistilled so I'm using it with CFG > 1
I don't remember having changed anything to the workflow, although the first times I tried it didn't work
Make sure you are using a very recent version of Nunchaku, the library must be >= 3.0b
I've just generated a lot of stuff that I would never have gotten with distilled. I'm using the default nunchaku loader & nunchaku lora loader. I recently switched to Ksampler instead of the myriad of nodes from the example nunchaku workflow. So it's a vanilla flux workflow actually. I use two Clip text Encode nodes for the prompts, the positive is connected to a FluxGuidance with 3.5 (it's probably useless but I kept it from the workflows using distilled models), the negative is always empty (I recommend never using negatives even if dedistilled is theorically supposed to handle them, it makes the model suck). CFG I use 4 by default and sometimes up to 8. From 40 to 60 steps. I guarantee you something is wrong on your setup, this model fully works, I'm so used to dedistilled I would know right away if something was wrong
Can somebody be kind to tell me what this is for?
Will there be an FP4 version for Blackwell GPUs ?
a minor nitpick: The word is quantization, not quantification. Quantification is something different.
In your guidelines for use in ComfyUI, you say to use Dual CLIP Loader guidance: 3.5. Which CLIP loader also has a guidance setting?
I think they are two separate things. You need Dual clip loader to load the text encoders (usually clip l and t5xxl. The guidance 3.5 must refer to the Flux Guidance Conditioning node
I think they are two separate things. You need Dual clip loader to load the text encoders (usually clip l and t5xxl. The guidance 3.5 must refer to the Flux Guidance Conditioning node
There are two files in the folder: unquantized_layers and transformer_blocks.
How can I use the model now? I get the following error with both:
NunchakuFluxDiTLoader
'NoneType' object is not subscriptable
it is a pity, but i have not enough lifetime to bring it to work
Details
Available On (1 platform)
Same model published on other platforms. May have additional downloads or version variants.