I was not quite happy with existing solutions for Illustrious based checkpoints with had inconsistent pixel sizes, so I decided to finally try to make a VAE based solution that provides perfect x8 pixel scaling. (This VAE should work for all SDXL checkpoints.) The SDXL latent internally uses a x8 resolution reduction and this VAE tries use this for its advantage for pixel art. It is trained using the pixelation model you probably had to use anyway, but now inbuilt into the model which should hopefully give better results.
My idea was to train a LORA on top of this VAE for the actual style, but for now I found that using pixel art style with Illustrious actually works quite nicely, especially if you do a iteration (or multiple) with img2img.
Description
FAQ
Comments (12)
Really neat that someone finally went and did this concept. It works great for txt2img outputs. However, it has a few problems I am running into:
1. LOTS of saturation and brightness drift. It's basically impossible to inpaint with this VAE since the masked regions lose so much color on each pass. I'm not sure if the fix for that is to train the decode on pixelised images that have saturation added back in, or to retrain the encode so that it preserves the new decode for the pixelized image round trips. Potentially a bit of both?
2. Edge artifacts. For primarily solid-color backgrounds, the last couple of pixels at the edge are brighter than the rest of the background. Not super noticeable for normal generations, but an problem for outpainting. This is also issue with normal VAEs when they handle solid colors, but it might be addressable with some extra penalty at edges.
3. I get NaNs when running at fp16. Not a big issue since it runs fine at fp32, but folks on UIs that don't make it easily changeable or don't know what they're doing tend to get scared off by that.
Thank you for the feedback.
1) I do see a small color shift in the stable-diffusion-webui-pixelization which has been replicated by this VAE. I did consider trying to fix this and make a new version, and perhaps see how well a 4x version would work and maybe even an unpixelate model for upscaling, but after 3 weeks and only a single image posted here I have dropped those plans. I don't really do that much pixel art myself.
2) I haven't looked into this at all for SDXL VAE, but from my experience with other ML models they are usually caused by poor edge handling in the internal layers in the model. So this might not be fixable with the safetensor format.
3) This is an issue with the original SDXL VAE. I will try to use the modified SDXL VAE which have been fixed for FP16 as the base if I make a new VAE in the future.
@spillerrec 1) Before your work, I do only know model & lora can make pixel art.😂
Maybe other people who are interested in pixel art have not thought of it either.
I discovered your work by chance —— I was suddenly curious about what was in the VAE category.
Got to say, this is pretty cool.
Do you think you can make a 16 bit version so I don't have to upscale it if I want higher resolution?
I have considered it and I think it would work, but I don't have much incentive to work on it right now. I will post it here on Civitai if I make it at some point.
@spillerrec Thanks. Of course, there is no real major need. If it's something you don't want to get at now or later, it is your project after all.
@Diamond_Letter A version with twice the resolution is now available here: https://civitai.com/models/2299499?modelVersionId=2587460
@spillerrec Hey cool! Thank you!
I will be busting that out when I get any ideas.
Fantastic work, thank you!
Never thought that I could convert pixel style through VAE encoder - VAE decoder.
Very great work!
It creates consistent pixel art of 8x8 tiles, by default it's really dirty with jpeg artifacts, then I just run it through pixelate filter in paint.net (cell size 8, nearest neighbor scaler) and poof, all jpeg is gone completely without interfering with image structure at all... 10/10
ОПЯТЬ СЖИМАЕШЬ ШАКАЛ ЕБУЧИЙ