Was just about to mention that. Only 2602-NSFW-BF16 is tagged as Illu instead of ZiT, the other versions seem fine.

Yes, you can. You can start by loading the fp8 version of the model as the “brain” you already have, then change its weights to a regular‑precision (fp16) format so they can be updated safely. Next, you'd attach small LoRA adjustment layers to that fp16 model and train those layers using normal‑precision arithmetic; the rest of the model stays frozen. After the training is finished, you'd merge the LoRA updates back into the fp16 weights, convert everything back to fp8, and use the new fp8 checkpoint for fast, low‑memory inference. This way you can keep the efficiency of the fp8 model while still being able to fine‑tune it with LoRA.

If it's a task you want to tackle I can tell you how to do it. This is something that I am still learning myself but I don't mind a programming challenge.

@freestuffpl0x42069 Damn that's alot of steps lol. But yes, I absolutely would love to see a rough demo by you if you have the time. Or any tutorials you can point to that would be awesome.

@Tezozomoctli Hey. Sorry for the delay, I am learning with you. There are a few routes to take. This workflow is one: https://civitai.com/models/808669/flux-dev2pro-fp8-special-use-for-training-flux-lora

If you want to be handheld through it here is a great tutorial: https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_z_image.md

what it does is it trains it as it is quantized so that you can run it with the quantized version.

Your second option is to have your training software upcast to fp16 automatically. Most tools like Kohya SS will automatically convert the fp8 model to fp16 in memory during training. You train the LoRA normally at fp16, and the resulting LoRA works on both fp8 and fp16 versions of the base model. This is the easiest route if you have enough VRAM (around 16GB+).

Another option is to train with native fp8 support. Newer versions of training tools (Kohya SS, SimpleTuner, ai-toolkit) support training directly in fp8. You’d enable fp8 precision in your training config. This uses less VRAM and trains faster while keeping everything in fp8. Quality is nearly identical to fp16 training based on recent research.

The option for a mixed precision approach keeps the base model frozen at fp8 but train the LoRA adapter layers at fp16/bf16. Some frameworks like Unsloth handle this automatically, giving you the memory savings of fp8 with the training stability of fp16.

The good news is that fp16 LoRAs generally work fine when applied to fp8 models during inference and the model just upcasts temporarily. So even if you train at fp16, it’ll still run on the fp8 version (with a slight VRAM increase during use ofc).

Personally, I will probably train at FP16 with the fp8 model upcasted during training since it gives the best accuracy-to-practicality ratio. The slight VRAM increase during training is worth it for the quality gain. I am running on a RTX 5070 Ti so it’s at the sweet spot for training LoRAs, if you’re running on <12GB of VRAM I would say training at fp8 is your best option.

I am going to be researching in the meantime and will touch base if I create anything. I would like to do the same I’ve just been slammed with learning how to merge checkpoints and bake in LoRAs. My goal is to make more checkpoints that are optimal for new users that run locally. I started with ComfyUI and ended with Forge Neo so I really want to bridge the gap for ease of use. I see a lot of people who should use Forge Neo on Pikokio but begin with ComfyUI as I initially did. Even with programming experience it has been a learning curve for me so I can assure you that you’re not dumb for asking questions.

@Tezozomoctli 

When training with AdamW (doesn't apply to prodigy) you can pass --use_8bit_adam to reduce the memory requirements of training. Make sure to install bitsandbytes if you want to do so.

AdamW has better quality with precision imo but it also takes longer, if you are wanting time efficiency, I suggest using Prodigy

To perform DreamBooth with LoRA on Z-Image, run:

To better track our training experiments, we're using the following flags in the command above:

report_to="wandb" will ensure the training runs are tracked on Weights and Biases. To use it, be sure to install wandb with pip install wandb. Don't forget to call wandb login <your_api_key> before training if you haven't done it before.

validation_prompt and validation_epochs to allow the script to do a few validation inference runs. This allows us to qualitatively check if the training is progressing as expected.

Note

If you want to train using long prompts, you can use --max_sequence_length to set the token limit. The default is 512. Note that this will use more resources and may slow down the training in some cases.

Training with FP8 Quantization

For reduced memory usage with FP8 training:

FSDP on the transformer

By setting the accelerate configuration with FSDP, the transformer block will be wrapped automatically. E.g. set the configuration to:

Prodigy Optimizer

Prodigy is an adaptive optimizer that dynamically adjusts the learning rate learned parameters based on past gradients, allowing for more efficient convergence. By using prodigy we can "eliminate" the need for manual learning rate tuning. Read more here.

To use prodigy, first make sure to install the prodigyopt library: pip install prodigyopt, and then specify:

Tip

When using prodigy it's generally good practice to set --learning_rate=1.0

LoRA Rank and Alpha

Two key LoRA hyperparameters are LoRA rank and LoRA alpha:

--rank: Defines the dimension of the trainable LoRA matrices. A higher rank means more expressiveness and capacity to learn (and more parameters).

--lora_alpha: A scaling factor for the LoRA's output. The LoRA update is scaled by lora_alpha / lora_rank.

lora_alpha vs. rank:

This ratio dictates the LoRA's effective strength:

lora_alpha == rank: Scaling factor is 1. The LoRA is applied with its learned strength. (e.g., alpha=16, rank=16)

lora_alpha < rank: Scaling factor < 1. Reduces the LoRA's impact. Useful for subtle changes or to prevent overpowering the base model. (e.g., alpha=8, rank=16)

lora_alpha > rank: Scaling factor > 1. Amplifies the LoRA's impact. Allows a lower rank LoRA to have a stronger effect. (e.g., alpha=32, rank=16)

Tip

A common starting point is to set lora_alpha equal to rank. Some also set lora_alpha to be twice the rank (e.g., lora_alpha=32 for lora_rank=16) to give the LoRA updates more influence without increasing parameter count. If you find your LoRA is "overcooking" or learning too aggressively, consider setting lora_alpha to half of rank (e.g., lora_alpha=8 for rank=16). Experimentation is often key to finding the optimal balance for your use case.

Target Modules

When LoRA was first adapted from language models to diffusion models, it was applied to the cross-attention layers in the UNet that relate the image representations with the prompts that describe them. More recently, SOTA text-to-image diffusion models replaced the UNet with a diffusion Transformer (DiT). With this change, we may also want to explore applying LoRA training onto different types of layers and blocks.

To allow more flexibility and control over the targeted modules we added --lora_layers, in which you can specify in a comma separated string the exact modules for LoRA training. Here are some examples of target modules you can provide:

For attention only layers: --lora_layers="to_k,to_q,to_v,to_out.0"

For attention and feed-forward layers: --lora_layers="to_k,to_q,to_v,to_out.0,ff.net.0.proj,ff.net.2"

Note

--lora_layers can also be used to specify which blocks to apply LoRA training to. To do so, simply add a block prefix to each layer in the comma separated string.

Note

Keep in mind that while training more layers can improve quality and expressiveness, it also increases the size of the output LoRA weights.

Aspect Ratio Bucketing

We've added aspect ratio bucketing support which allows training on images with different aspect ratios without cropping them to a single square resolution. This technique helps preserve the original composition of training images and can improve training efficiency.

To enable aspect ratio bucketing, pass --aspect_ratio_buckets argument with a semicolon-separated list of height,width pairs, such as:

Bilingual Prompts

Z-Image has strong support for both Chinese and English prompts. When training with Chinese prompts, ensure your dataset captions are properly encoded in UTF-8:

Tip

Z-Image excels at text rendering in generated images, especially for Chinese characters. If your use case involves generating images with text, consider including text-related examples in your training data.

Inference

Once you have trained a LoRA, you can load it for inference:

Since Z-Image finetuning is still in an experimental phase, we encourage you to explore different settings and share your insights! 🤗

Source: https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_z_image.md

I have the same question.

not for me, it breaks completely with a character lora

My character lora, trained on ZI base, works great with this checkpoint

@curiousasian86164 How do you adjust the LORA strenght in this workflow?

@Stable_Yogi Thanks, Someone suggested something similar, but the tip of manage each individually sounds cool!, thanks man, Im starting to love this checkpoint, do the ones in your PATRON are "better" in regards to functionality?

@Conejoquehace I use a basic workflow, and leave the lora strength at 1. I trained my character lora with AI Toolkit on Runpod with the following setting:

linear_alpha: 64
timestep_type: "sigmoid"
lr: 0.0003
differential_guidance_scale: 4

The generated images were near perfect likeness to the subject

@curiousasian86164 will it work if my lora is trained on de distilled model?

@desplumao22 same here.., did you find any solution?

absolutely not, destroys the character

Try the Z-Image LoRA Merger node from the ComfyUI-ZImage-LoRA-Merger package. It sometimes works wonders when using normalization and a general target_strength of 0.8.

Unet Folder

Checkpoint. Just use Neo forge on pikokio

oddly, recently it is requiring double the space for the initial download. I'm needing around 22gb-25gb to download.

@Stable_Yogi Hello. Could you tell me if the trigger works on all models, including the 2602 Q8? Also, is the list of LORs included in the Q8 and BF16 models extensive?

@Stable_Yogi Thank you

And one more question - I cannot generate st...on as well, is it possible to add it in the next update?

Yes, Don't! :P