Experimental conversion of SDXL architecture to 16 channel latent space

This is an experimental pretrain on top of Rouwei-0.8 that works with 16 channel latent space and uses Flux ae.

Goals:

• Achieve better details while maintaining low compute requirements and all existing knowledge and performance

• Possibility of joint sampling with Flux/Chroma/Lumina and other models with same latent space

Current state:

Early alpha version, it is pretty raw. Images may contain extra noise and have artefacts in small details, level varies from neglectable to significant. Upscale, samplers/schedulers, styles, even prompt affect it.

Use of GAN upscale models in pixels space instead of latent upscale gives much smoother results, bumping base resolution higher helps too.

Epsilon prediction now, can be converted to vpred or anything in future.

Usage:

Comfyui

Workflow example (Or just pick any image from showcase)

1. Download the checkpoint (FP32 and Unet-only can be found in HF REPO

2. Download these nodes (or just use install missing nodes using Comfy Manager)

3. Use SDXL 16ch loader node to load it, then work just like you used to with sdxl

4. DO NOT REMOVE Latent multyply NODES, latents should be scaled before and after processing just same as in regular SDXL inference. This step just isn't hidden yet.

If you're getting error mat1 and mat2 shapes cannot be multiplied (_x16 and 4x3) - disable the preview option for Ksampler. It happens because preview uses taesd vae designed for 4channel.

Other UI

Since the main difference is just shapes of tensors, used vae and latents scaling factor - it should be easy to implement support to any other UI.

Lora adapters, controlnet, ip-adapters, other things untested.

Joint sampling:

Since the model operates in 16channel latent space similat to Flux, Chroma, Limina-image and some other, you can implement complex workflows (if you have enough memory). This allows to utilize all knowlege of characters, styles, concepts from RouWei along with the performance of bigger models.

Here is an example workflow. Using just few (1..4) steps from Flux you create some rough basic composition. Then the latents come to 16channel sdxl model where denoised (skipping initial high noise timesteps).

It is the most simple approach, since you don't need to reconvert latents though series of vae or some adapters, you can change models on every denoising step without having any performance impact.

Just don't forget to apply Latents multiply nodes between transitions

How it's made

Basically, no changes to default architecture. Just re-initializing if input and output layers to new size, then training with gradual unfreezing of blocks towards the middle.

Default SDXL latent scale factor of 0.13025 doesn't work well here, 0.6 is used for this release.

This is not the most optimal approach. Some changes to the outer layers of the model instead of direct use 'as is' should give improvement in future. If you have any thoughts or ideas about it - please share them.

Training:

To train it (in current version) all you need is to change the number of in/out channels in UNET config and set scale factor to 0.6 instead of 0.13025. And probably check vae part to work properly.

(Code examples later)

I'm willing to help/cooperate:

Join Discord server where you can share your thoughts, proposals, requests, etc. Write to me directly here or dm in discord.

Thanks:

Part of training was performed using google TPU and sponsored by OpenRoot-Compute

Personal: NeuroSenko

And many thanks to all fellow brothers who supported me before.

Donations:

BTC bc1qwv83ggq8rvv07uk6dv4njs0j3yygj3aax4wg6c

ETH/USDT(e) 0x04C8a749F49aE8a56CB84cF0C99CD9E92eDB17db

XMR 47F7JAyKP8tMBtzwxpoZsUVB8wzg2VrbtDKBice9FAS1FikbHEXXPof4PAb42CQ5ch8p8Hs4RvJuzPHDtaVSdQzD6ZbA5TZ

License:

Same viral as for Illustrious base.