<h2 id="experimental-conversion-of-sdxl-architecture-to-16-channel-latent-space">Experimental conversion of SDXL architecture to 16 channel latent space</h2><p></p><p>This is an experimental pretrain on top of Rouwei-0.8 that works with 16 channel latent space and uses Flux ae.</p><p></p><h2 id="goals:">Goals:</h2><ul><li><p>Achieve better details while maintaining low compute requirements and all existing knowledge and performance</p></li><li><p>Possibility of joint sampling with Flux/Chroma/Lumina and other models with same latent space</p></li></ul><h2 id="current-state:">Current state:</h2><p><strong><u>Early alpha version, it is pretty raw.</u></strong> Images may contain extra noise and have artefacts in small details, level varies from neglectable to significant. Upscale, samplers/schedulers, styles, even prompt affect it.</p><p>Use of GAN upscale models in pixels space instead of latent upscale gives much smoother results, bumping base resolution higher helps too.</p><p>Epsilon prediction now, can be converted to vpred or anything in future.</p><h2 id="usage:">Usage:</h2><h3 id="comfyui">Comfyui</h3><p></p><p><a target="_blank" rel="ugc" href="https://huggingface.co/Minthy/RouWei-0.8-16ch-v0.1alpha/resolve/main/workflow.png">Workflow example</a> (Or just pick any image from showcase)</p><p></p><ol><li><p>Download the checkpoint (FP32 and Unet-only can be found in <a target="_blank" rel="ugc" href="https://huggingface.co/Minthy/RouWei-0.8-16ch-v0.1alpha">HF REPO</a></p></li><li><p>Download <a target="_blank" rel="ugc" href="https://github.com/NeuroSenko/ComfyUI-SDXL-16ch-loader">these nodes </a>(or just use <code>install missing nodes</code> using Comfy Manager)</p></li><li><p>Use <code>SDXL 16ch loader</code> node to load it, then work just like you used to with sdxl</p></li><li><p><strong>DO NOT REMOVE </strong><code>Latent multyply</code><strong> NODES</strong>, latents should be scaled before and after processing just same as in regular SDXL inference. This step just isn't hidden yet.</p></li></ol><p></p><h3 id="if-you're-getting-error-mat1-and-mat2-shapes-cannot-be-multiplied-(_x16-and-4x3)-disable-the-preview-option-for-ksampler.-it-happens-because-preview-uses-taesd-vae-designed-for-4channel.">If you're getting error <code>mat1 and mat2 shapes cannot be multiplied (_x16 and 4x3)</code> - disable the preview option for Ksampler. It happens because <span>preview uses taesd vae designed for 4channel.</span></h3><p></p><h3 id="other-ui">Other UI</h3><p>Since the main difference is just shapes of tensors, used vae and latents scaling factor - it should be easy to implement support to any other UI.</p><p>Lora adapters, controlnet, ip-adapters, other things untested.</p><p></p><h2 id="joint-sampling:">Joint sampling:</h2><p>Since the model operates in 16channel latent space similat to Flux, Chroma, Limina-image and some other, you can implement complex workflows (if you have enough memory). This allows to utilize all knowlege of characters, styles, concepts from RouWei along with the performance of bigger models.</p><p>Here is an <a target="_blank" rel="ugc" href="https://huggingface.co/Minthy/RouWei-0.8-16ch-v0.1alpha/resolve/main/examples/joint_with_flux/j_1.png">example workflow</a>. Using just few (1..4) steps from Flux you create some rough basic composition. Then the latents come to 16channel sdxl model where denoised (skipping initial high noise timesteps).</p><p>It is the most simple approach, since you don't need to reconvert latents though series of vae or some adapters, you can change models on every denoising step without having any performance impact.</p><p></p><p><strong>Just don't forget to apply </strong><code>Latents multiply</code><strong> nodes between transitions</strong></p><p></p><h2 id="how-it's-made">How it's made</h2><p>Basically, no changes to default architecture. Just re-initializing if input and output layers to new size, then training with gradual unfreezing of blocks towards the middle.</p><p></p><p><strong><u>Default SDXL latent scale factor of </u></strong><code>0.13025</code><strong><u> doesn't work well here, </u></strong><code>0.6</code><strong><u> is used for this release.</u></strong></p><p></p><p>This is not the most optimal approach. Some changes to the outer layers of the model instead of direct use 'as is' should give improvement in future. If you have any thoughts or ideas about it - please share them.</p><p></p><h2 id="training:">Training:</h2><p></p><p>To train it (in current version) all you need is to change the number of in/out channels in UNET config and set scale factor to 0.6 instead of 0.13025. And probably check vae part to work properly.</p><p>(Code examples later)</p><p></p><h2 id="i'm-willing-to-helpcooperate:">I'm willing to help/cooperate:</h2><p>Join <a target="_blank" rel="ugc" href="https://discord.gg/ZXHENAhqE9">Discord server</a> where you can share your thoughts, proposals, requests, etc. Write to me directly here or dm in discord.</p><p></p><h2 id="thanks:">Thanks:</h2><p>Part of training was performed using google TPU and sponsored by <strong>OpenRoot-Compute</strong></p><p>Personal: NeuroSenko</p><p>And many thanks to all fellow brothers who supported me before.</p><p></p><h2 id="donations:">Donations:</h2><p>BTC bc1qwv83ggq8rvv07uk6dv4njs0j3yygj3aax4wg6c</p><p>ETH/USDT(e) 0x04C8a749F49aE8a56CB84cF0C99CD9E92eDB17db</p><p>XMR 47F7JAyKP8tMBtzwxpoZsUVB8wzg2VrbtDKBice9FAS1FikbHEXXPof4PAb42CQ5ch8p8Hs4RvJuzPHDtaVSdQzD6ZbA5TZ</p><p></p><h2 id="license:">License:</h2><p>Same viral as for Illustrious base.</p><p></p>

Files with SHA256: 78a5fc69d6489d31a2eaf830513f4f2266c2fda299509b018a4e0833408b6b45