Note: This is a highly experimental project.

This model was created with the aim of integrating the Z-Image Turbo and Z-Image Base models. As you know, Z-Image uses completely different models for Turbo and Base. When two such models are released, usually the Base version and a distilled version of it are released, but Z-Image uses models that are fundamentally different. Compatibility is poor. Regarding LoRA, I understand that some degree of interoperability is possible, but model merging is almost impossible.

One difficulty is that the base models are trained in completely different ways, and another is the existence of the refiner. Unlike many other models, the meaning of the DiT layer in Z-Image differs from model to model. In normal models, the DiT is created by a common text encoder, but in Z-Image, this is done via the refiner, and the refiner part changes from model to model. The meaning of the numerical values ​​stored in the DiT changes from model to model. Therefore, the compatibility of the DiT part is low. Incidentally, in the Unet era, I also trained text encoders as models, so it's a similar situation, but in that case, it was only used for cross-attention and didn't have as much impact.

What I'm doing

Now, let me explain how Alkahest is trying to solve this. First, I create a merged Turbo and Base model. I create it 1:1 without thinking too much. Naturally, it only produces noise. Next, using that as the base model, I perform Full Test (FT) with the refiner layer fixed. FT itself isn't impossible. You might feel like giving up, thinking that no images will ever appear, but if you keep going, the noise will gradually be reduced.

TB-0

This is a model that finally produces images. I used about 22,000 images from an anime-style dataset (0.75 series) and then about 15,000 images from a realistic-style dataset (part of 0.79). I've found that the more FTs I run, the less noise there is. While the current stage isn't quite sufficient, I'm testing whether we can handle both Turbo and Base models using this intermediate model.

Verification

I used the ComfyUI merge node set. Prepare a Z-Image Turbo model and a Z-Image Base model. Connect the model you want to merge to model1 and Alkahest to model2 in ModelMergeSubstruct. Next, prepare a ModelMergeAdd node and connect Alkahest to model1 and the difference output from the previous node to model2. Assume the resulting model is similar to the Base model. Use Distill LoRA, partly to stabilize the image. Re-Turbo LoRA for the Turbo model is also acceptable, but Fun Distill for the Base model is recommended.

I used several models for testing, but the ones listed here are rayZimageBaseSFW_artshoot and beretMiXZIT_v40.

Alkahest has a rather distinctive style, but perhaps because its ability to generate images from a single model is quite low, the merged result is strongly influenced by the model it merges with, and it incorporates the target model almost as is. The Alkahest component is extremely small. Note that there is also the option of using each model's base model as the source for difference calculations, but the base model didn't work properly. To stabilize the image style, please use ModelSamplingAuraFlow at around 10-12.

With this, I was able to capture the difference between the Turbo model and the Base model. Ideally, I would be able to merge them, but mixing them results in noise. According to ChatGPT, this is impossible because one was rotated in the Turbo space and the other in the Base space. I am exploring ways to solve this issue.

Note that while merging using ModelMergeSimple works well with the Turbo model, it doesn't work well with the Base model. The reason is unknown. When trying this with the Turbo model, you will need to the De-turbo element, which many models likely used as their base model. Find the Ostris training adapter LoRA and apply it with -1. Failure to do so will drastically increase noise.

Conclusion

This experiment was successful in that it allowed for the creation of a bridge model between two models, but it left challenges for further development.

One point of reflection is that creating the base model by merging De-turbo and Base models, rather than Turbo and Base models, would have made operation easier and likely smoother the FT process. A drawback is the need for processing when dealing with the Turbo base model itself, or models that simply add LoRA to the initial base model.

Performing the FT itself on an existing SDXL dataset was also questionable. Using the images output by Turbo and Base models would likely improve accuracy. Since the dataset consisted almost entirely of people, Alkahest showed very little background information. However, it did display some content not present in the dataset, indicating that it was beginning to recall the original model's training data. But there seemed to be too few elements to trigger further learning.

In reality, the Turbo and Base models are not yet unified. I plan to verify whether they can truly be unified in the next experiment.