VAE for perfectly square pixels
This is a trained VAE to enforce that pixels are always 4x4 in size and solid in color. It replicates an existing pixelation model while being built into the model itself.
Compability
Most architectures uses an unmodified VAE and does not retrain it, so it can be used accross different generations of models.
SDXL: Can be used for all SDXL based models, such as Pony, Illustrious, and NoobXL. They all use the original VAE shipped with SDXL.
Anima: Can be used for Anima and Qwen Image (untested).
How VAEs works
VAE's are used to reduce the working resolution of the model so they don't use as much VRAM. SDXL for example runs in a 8 times smaller resolution, then uses the VAE decoder to upscale it to full resolution. This model exploits that to enforce that this upscale process always results in square pixels.
Further work
The VAE has been trained to produce pixelated images from non-pixelated images. This means it isn't as reliable if the original image produced is close to pixelated, but contains errors. Training so it keeps pixelated images exactly the same, or potentially also fixes images with have inconsistent pixel sizes could improve the output.
For high contrast areas, some ringing can happen. Training with syntetic data will likely help.
Only the VAE decoder has been trained. It is likely possible to train the encoder to remove the pixelation as well, which could potentially make it easier to train Loras.
Description
Improved training set and tweaked loss function to improve quality and reduce color changes.
FAQ
Comments (2)
Looking forward to playing around with the new version!
Something I noticed in the original x4 version is that it tends to produce ringing artifacts, sort of like what you'd see in an over-sharpened image. The x8 does not have that issue.
Any chance this is fixed in 1.1?
Potentially. The V1.0 replicated some of the artifacts the original pixelation model I used for the training set had, including very ugly ringing at high contrast areas. Most images would look good, but certain images would give really ugly results. Those artifacts have been greatly reduced, but since it is trained on top of V1.0 I'm not confident it is fully solved. (Training from scratch would likely be better, but V1.0 alone was 12+ hours and that could still benefit from more training.)
To be honest I haven't played too much with V1.1, so let me know if you still experience issues. If you can regenerate the image with the same seed, but with the standard VAE instead, I can use it for testing next time I feel like working on this.