This workflow is based on the incredible work by Kijai (https://github.com/kijai/ComfyUI-FluxTrainer) who created the training nodes for ComfyUI based on Kohya_ss (https://github.com/kohya-ss/sd-scripts) work. All credits go to them. Thanks also to u/tom83_be on Reddit who posted his installation and basic settings tips.
I just reworked the workflow and wrote a user-guide. I will keep updating the workflow too here.
With this workflow you can train LoRA's for FLUX on ComfyUI.
Around 12Gb Vram is all you need on your graphic card, so you don't need a RTX 3090 or 4090 Gpu, but it may need 32Gb Ram (set "split_mode" on "true").
If you are used to ComfyUI, the training process will be very easy, as the workflow is divided in nodes' group that allow you to set all the data needed for the LoRA training in a clear and easy way.
This workflow is also being tested on Runpod, using a A40 with 48Gb Vram.
You will need the following Flux Dev model files:
flux1-dev-fp8.safetensors
from https://huggingface.co/Kijai/flux-fp8/tree/main save it in ".../ComfyUI_training/ComfyUI/models/unet/t5xxl_fp8_e4m3fn.safetensors
from https://huggingface.co/comfyanonymous/flux_text_encoders/tree/main save it in ".../ComfyUI_training/ComfyUI/models/clip/"clip_l.safetensors
from https://huggingface.co/comfyanonymous/flux_text_encoders/tree/main save it in ".../ComfyUI_training/ComfyUI/models/clip/"ae.safetensors
from https://huggingface.co/black-forest-labs/FLUX.1-dev/tree/main save it in ".../ComfyUI_training/ComfyUI/models/vae/"
The images I published were all created with a LoRA I trained with this workflow for a fiction instagram influencer called Kiara Aigen.
My suggestions: start with 10-20 image, 512x512 resolution, 2000 steps (so you will have 500 steps in each of the 4 Training group) and use Adafactor optimizer.
Remeber to set the right folder for input_dir (images set) and output_dir (where the workflow saves the LoRA's) and to change (add) a prompt in the "Init Flux Lora Training" node for Validation.
Check yellow notes in the workflow for more instructions.
Guide to this workflow is here: https://civarchive.com/articles/7131/flux-lora-trainer-on-comfyui-v10
Enjoy!
Description
Changed settings to make the training a lot faster.
Removed Adafactor optimizer node, with Adamw8bit the training is great.
With these settings I fully trained a character LoRA in a little more than one hour (4070 Super 16Gb Vram, 64Gb Ram).
FAQ
Comments (16)
I would use this unfortunately got 14s/it on a 1024 training with adam8, got 3s/it with the same settings on FluxGym, about 5 times faster I wonder why
Just saw this on your profile. When you mean MAY need 32GB ram. Is it at all possible to get a way with 16gb and sacrifice another 3 hours of training? Possibly?...(I should read into it more)
4060ti 16GB with 64GB RAM is getting OOM for the 1024 mode with split_mode enabled. (19 images of various large sizes). Perhaps I'm missing something - any suggestions?
Could you share your comfyui args and/or optimizations? And your pytorch/cuda versions?
The 4060 Ti 16 GB, without split, worked well until the first batch of validation images was generated. I encountered an out-of-memory (OOM) error during this process, but I resolved it by changing the validation image settings from 1024 to 512.
Holy crap, I made a LORA of myself and its both awesome and scary at the same time! Works on AMD 7900XTX using the Zluda back end! Had to use normal AdamW optimization instead of AdamW 8 bit.
This is working great, other than I notice that I have the set the model strength to 0, and therefore it isn't working well for onsite gen. Any suggestions to alleviate this? (using default setting with 1.1 other than altering directory names etc.)
@pgcrt does your lora trainer workflow support single/specific layer lora training. As in this reddit post
https://www.reddit.com/r/StableDiffusion/s/vMIh4M7kVe
Checked this discussion
https://github.com/bmaltais/kohya_ss/issues/2701
and found out that this functionality was added to Kohya_ss training script last week.
Thanks for this! I'm focusing only on Schnell and locally seems to be the only way. So this was definitely appreciated!
Update 2: 1024x1024 images seem to work fine at 10 images, it's the 512 specifically that seem to need more.
Update on experience as a flux schnell user.
The 10 to 20 seems to be a Dev specific thing lol. For schnell you want to do 20 to 30 it seems as a minimum. I'm making Loras on a RTX 4090 laptop. My first Lora I went in blind and just winged it, and the lora came out great. Next I followed instructions and it came out horrible lol. 20-30 seems to be the minimum for schnell, but the rest applies fine it seems.
why I always got:
Can't load tokenizer for 'openai/clip-vit-large-patch14'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'openai/clip-vit-large-patch14' is the correct path to a directory containing all relevant files for a CLIPTokenizer tokenizer. :(
Thanks for this! I could create a lora for a consistent character with about 23 input pictures. It took a little over 4 hours on my 3060 12GB VRAM 🫶
Testing it now and it works, however I think I need to add more diversity of poses and expressions to the input material.
Note Plus (mtb) is missing and comfyui can't seem to find it nor can i find it online. Any help pls?
How do you set clip_skip? I need clip_skip=2, but the sanity checker says it is clip_skip=None
'FluxUpperLowerWrapper' object has no attribute 'prepare_block_swap_before_forward'
An error began to appear when generating images of the era. What could be the reason?
Getting "max arg" error? What does that mean?
Looks like we don't have an active mirror for this file right now.
CivArchive is a community-maintained index — we catalog mirrors that volunteers upload to HuggingFace, torrents, and other public hosts. Looks like no one has uploaded a copy of this file yet.
Some files do get recovered over time through contributions. If you're looking for this one, feel free to ask in Discord, or help preserve it if you have a copy.