<h1 id="heading-12">QR Code Conditioned ControlNet Models for Stable Diffusion 1.5 and 2.1</h1><h2 id="heading-13">Model Description</h2>These ControlNet models have been trained on a large dataset of 150,000 QR code + QR code artwork couples. They provide a solid foundation for generating QR code-based artwork that is aesthetically pleasing, while still maintaining the integral QR code shape.The Stable Diffusion 2.1 version is marginally more effective, as it was developed to address my specific needs. However, a 1.5 version model was also trained on the same dataset for those who are using the older version.<h2 id="heading-14">Performance and Limitations</h2>These models perform quite well in most cases, but please note that they are not 100% accurate. In some instances, the QR code shape might not come through as expected. You can increase the ControlNet weight to emphasize the QR code shape. However, be cautious as this might negatively impact the style of your output.**To optimize for scanning, please generate your QR codes with correction mode 'H' (30%).**To balance between style and shape, a gentle fine-tuning of the control weight might be required based on the individual input and the desired output, aswell as the correct prompt. Some prompts do not work until you increase the weight by a lot. The process of finding the right balance between these factors is part art and part science. For the best results, it is recommended to generate your artwork at a resolution of 768. This allows for a higher level of detail in the final product, enhancing the quality and effectiveness of the QR code-based artwork.<h2 id="heading-1239">Installation</h2>The simplest way to use this is to place the .safetensors model and its .yaml config file in the folder where your other controlnet models are installed, which varies per application.For usage in auto1111 they can be placed in the webui/models/ControlNet folder. They can be loaded using the controlnet webui extension which you can install through the extensions tab in the webui (<a target="_blank" rel="ugc" href="https://github.com/Mikubill/sd-webui-controlnet">https://github.com/Mikubill/sd-webui-controlnet</a>). Make sure to enable your controlnet unit and set your input image as the QR code. Set the model to either the SD2.1 or 1.5 version depending on your base stable diffusion model, or it will error. No pre-processor is needed, though you can use the invert pre-processor for a different variation of results. 768 is the preferred resolution for generation since it allows for more detail.Make sure to look up additional info on how to use controlnet if you get stuck, once you have the webui up and running its really easy to install the controlnet extension aswell.

ControlNet QR Code

controlnetQRCode_sd15V1.yaml

controlnetQRCode_sd15V1.safetensors

SD1.5 v1

SD2.1 v1

<h3>STOP! THESE MODELS ARE NOT FOR PROMPTING/IMAGE GENERATION</h3>These are the <a target="_blank" rel="ugc" href="https://github.com/lllyasviel/ControlNet-v1-1-nightly">new ControlNet 1.1 models</a> required for the <a target="_blank" rel="ugc" href="https://github.com/Mikubill/sd-webui-controlnet">ControlNet extension</a>, converted to Safetensor and "pruned" to extract the ControlNet neural network. Also Note: There are associated .yaml files for each of these models now. Place them alongside the models in the models folder - making sure they have the same name as the models! These models are embedded with the neural network data required to make ControlNet function, they will not produce good images unless they are used with ControlNet. To be honest, there isn't much difference between these and the <a target="_blank" rel="ugc" href="https://civarchive.com/models/9251/controlnet-pre-trained-models">OG ControlNet V1's</a>. These models are further trained ControlNet 1.0 models, with an additional 200 GPU hours on an A100 80G.<ul><li>control_v11p_sd15_seg</li><li>control_v11p_sd15_softedge</li><li>control_v11p_sd15_normalbae</li><li>control_v11p_sd15_scribble</li><li>control_v11p_sd15_mlsd</li><li>control_v11p_sd15_openpose</li><li>control_v11p_sd15_depth</li><li>control_v11p_sd15_canny</li></ul><ul><li>control_v11e_sd15_ip2p</li><li>control_v11e_sd15_shuffle</li><li>control_v11p_sd15_inpaint</li><li>control_v11p_sd15_lineart</li><li>control_v11p_sd15s2_lineart_anime</li><li>control_v11u_sd15_tile</li></ul>Download these models and place them in the <code>\stable-diffusion-webui\extensions\sd-webui-controlnet\models</code> directory. Please consider joining my Patreon! Advanced SD tutorials, settings explanations, adult-art, from a female content creator (me!) <a target="_blank" rel="ugc" href="http://patreon.com/theally">patreon.com/theally</a> - I also have a write-up of ControlNet and will be updating with the latest news/developments!

ControlNet 1.1 Models

controlnet11Models_scribble.yaml

controlnet11Models_scribble.safetensors

Scribble

Tile (e)

AnimeLine

Lineart

Inpaint

Shuffle

Pix2Pix

OpenPose

MLSD

Normal

Softedge

Depth

Canny

controlnet11Models_normal.yaml

controlnet11Models_normal.safetensors

controlnet11Models_seg.yaml

controlnet11Models_seg.safetensors

controlnet11Models_lineart.safetensors

controlnet11Models_lineart.yaml

controlnet11Models_depth.safetensors

controlnet11Models_depth.yaml

<h3 id="heading-1005">This is a ControlNet model! This model requires ControlNet!</h3><h3 id="heading-1007">Model Details</h3>This model aims to allow users to modify what a subject is wearing in a given image while keeping the subject, background and pose consistent.I've produced good results in txt2img, img2img as well as inpainting. I've produced good results with images generated with Stable Diffusion as well as with pictures I've taken myself.<h3 id="heading-1006">Installation</h3>Place the .safetensors file into ControlNet's 'models' directory. To use the model, select the 'outfitToOutfit' model under ControlNet Model with 'none' selected under Preprocessor.<h3 id="heading-1008">Tips for use</h3><ul><li>Images with a clearly defined subject tend to work better.</li><li>This model tends to work best at lower resolutions (close to 512px)<ul><li>If you run into trouble at higher resolutions, try running a first pass at a lower resolution and then using img2img (or txt2img w/ Hires.fix) with a lower denoising strength to upscale to a higher resolution while continuing to use your original input image as input to this ControlNet model</li></ul></li><li>I recommend starting with CFG 2 or 3 when using ControlNet weight 1<ul><li>Higher CFG values when combined with high ControlNet weight can lead to burnt looking images.</li><li>Experiment with ControlNet Control Weights 0.4, 0.45, 0.5, 0.6, 0.8 and 1.<ul><li>Lower weight allows for more changes, higher weight tries to keep the output similar to the input</li><li>Anything below 0.5 seems to rely more on the Stable Diffusion model whereas anything 0.5 and up seems to weight the ControlNet model more heavily</li></ul></li></ul></li><li>When using img2img or inpainting, I recommend starting with 1 denoising strength<ul><li>Experiment with 0.75 denoising strength</li></ul></li><li>When inpainting, I recommend trying "latent nothing" under Masked content</li><li>Consider lowering the model's weight when generating higher resolution images<ul><li>The higher the resolution of the output image, the more difficult it tends to be to alter the content of the image from the input image</li></ul></li><li>If the output isn't changing enough from the input, try increasing the weight of the prompts or decreasing the Control Weight of the ControlNet Unit</li><li>Can work well with other models such as OpenPose ControlNet</li></ul>

Outfit To Outfit - ControlNet (outfit2outfit)

Trained this model using a completely new dataset using more real data.Main changes:<ul><li>Overall improvements in character consistency and quality.</li><li>Discontinued the use of "the same man" / "the same woman" in prompts</li><li>Reduced NSFW generations</li></ul>Known issues: Improvements could be made in maintaining color consistency as well as improving clothing generation overall. Some of this may just be a matter of curating more and higher quality data. Other improvements may require tweaking the training script

outfitToOutfit_v20.yaml

outfitToOutfit_v20.safetensors

v2.0

v1.0

controlnet11Models_canny.safetensors

controlnet11Models_canny.yaml

controlnet11Models_openpose.safetensors

controlnet11Models_openpose.yaml

controlnet11Models_mlsd.safetensors

controlnet11Models_mlsd.yaml

controlnet11Models_softedge.safetensors

controlnet11Models_softedge.yaml

controlnet11Models_pix2pix.safetensors

controlnet11Models_pix2pix.yaml

controlnet11Models_inpaint.safetensors

controlnet11Models_inpaint.yaml

controlnet11Models_animeline.safetensors

controlnet11Models_animeline.yaml

<pre><code>https://huggingface.co/thomaseding/pixelnet

--- license: creativeml-openrail-m ---

# PixelNet (Thomas Eding)

### About:

PixelNet is a ControlNet model for Stable Diffusion 1.5

It takes a checkerboard image as input, which is used to control where logical pixels are to be placed.

This is currently an experimental proof of concept. I trained this using on around 2000 pixel-art/pixelated images that I generated using Stable Diffusion (with a lot of cleanup and manual curation). The model is not very good, but it does work on grid sizes of about a max of 64 checker "pixels" when the smallest dimension is 512. I can successfully get the model to understand 128x128 checkerboards for image generations of at least 1024x1024 pixels.

The model works best with the "Balanced" ControlNet setting. Try using a "Control Weight" of 1 or a little higher.

"ControlNet Is More Important" seems to require a heavy "Control Weight" setting to have an effect. Try using a "Control Weight" of 2.

A low "Control Weight" setting seems to produce images that resemble smooth paintings or vector art.

Smaller checker grids tend to perform worse (e.g. 5x5 vs a 32x32)

Too low or too high of a "Steps" value breaks the model. Try something like 15-30, depending on an assortment of factors. Feel free to experiment with the built-in A1111 "X/Y/Z Plot" script.

### Usage:

To install, copy the `.safetensors` and `.yaml` files to your Automatic1111 ControlNet extension's model directory (e.g. `stable-diffusion-webui/extensions/sd-webui-controlnet/models`). Completely restart the Automatic1111 server after doing this and then refresh the web page.

There is no preprocessor. Instead, supply a black and white checkerboard image as the control input. Various control image grids can be found in this repository's `grids` directory. (https://huggingface.co/thomaseding/pixelnet/resolve/main/grids/grids.zip)

The script `gen_checker.py` can be used to generate checkerboard images of arbitrary sizes. (https://huggingface.co/thomaseding/pixelnet/blob/main/gen_checker.py) Example: `python gen_checker.py --upscale-dims 512x512 --dims 70x70 --output-file control.png` to generate a 70x70 checkerboard image upscaled to 512x512 pixels.

The script `controlled_downscale.py` is a custom downscaler made specifically for this model. You provide both the generated image and the control image used to generate it. It will downscale according to the control grid. (https://huggingface.co/thomaseding/pixelnet/blob/main/controlled_downscale.py) Example: `python controlled_downscale.py --control diffusion_control.png --input diffusion_output.png --output-downscaled downscaled.png --output-quantized quantized.png --trim-cropped-edges false --sample-radius 2`. See `--help` for more info.

### VAE:

https://civarchive.com/models/366568/vae-teding-aliased-2024-03

### FAQ:

Q: Are there any "Trigger Words" for this model?

A: Not really. I removed all words pertaining to style for my training data. This includes words like "pixel", "high quality", etc. In fact adding "pixel art" to the prompt seems to make the model perform worse (in my experience). One word I do find useful is to add "garish" to the negative prompt when the output coloring is hyper.

Q: Png or Jpeg?

A: Use Png. Jpeg's compression algorithm is terrible for pixel art.

Q: Is there special A1111 user-interface integration?

A: Yes... but not yet merged into the standard ControlNet extension's code. See (https://civarchive.com/posts/371477) if you want to integrate the changes yourself in the meantime.

Q: Why is this needed? Can't I use a post-processor to downscale the image?

A: From my experience SD has a hard time creating genuine pixel art (even with dedicated base models and loras), where it has a mismatch of logical pixel sizes, smooth curves, etc. What appears to be a straight line at a glance, might bend around. This can cause post-processors to create artifacts based on quantization rounding a pixel to a position one pixel off in some direction. This model is intended to help fix that.

Q: Should I use this model with a post-processor?

A: Yes, I still recommend you do post-processing to clean up the image. This model is not perfect and will still have artifacts. Note that none of the sample output images are post-processed; they are raw outputs from the model. Consider sampling the image based on the location of the control grid checker faces. The provided `controlled_downscale.py` script can do this for you. You can take the output of this script (presumably the `--output-downscaled` file) and then run it through a different post-processor (e.g. to refine the color palette). I only tested the script for a few generated images, so it might still be a bit buggy in the way it computes the sample locations. So for now, compare the output of the script. You may find that supplying an alternative control grid image may be beneficial, or may find that using some alternative post-processing method may be better.

Q: Does the model support non-square grids?

A: Kind of. I trained it with some non-perfect square grids (when pre-upscaled checkerboards are not a factor of the upscaled image size), so in that sense it should work fine. I also trained it with some checkerboard images with genuine non-square rectangular faces (e.g. double-wide pixels).

Q: Will there be a better trained model of this in the future?

A: I hope so. I will need to curate a much larger and higher-quality dataset, which might take me a long time. Regardless, I plan on making the control effect more faithful to the control image. I may decide to try to generalize this beyond rectangular grids, but that is not a priority. I think including non-square rectangular faces in some of the training data was perhaps harmful to the model's performance. Likewise for grids smaller than 8x8. Perhaps it is better to train separate models for very small grids (but at that point, you might as well make the images by hand) and for non-square rectangular grids.

Q: What about color quantization?

A: Coming soon, "PaletteNet".

### Sample Outputs:

![sample1](https://huggingface.co/thomaseding/pixelnet/resolve/main/example-outputs/20230703102437-64a98cdc-3566720748-1259.png)

![sample2](https://huggingface.co/thomaseding/pixelnet/resolve/main/example-outputs/20230703091940-d7d11138-2383291623-524.png)

![sample3](https://huggingface.co/thomaseding/pixelnet/resolve/main/example-outputs/20230703083502-89f714b7-2908299568-164.png)</code></pre>

PixelNet (ControlNet for Pixel Art)

pixelnetControlnetFor_v00Experimental.safetensors

pixelnetControlnetFor_v00Experimental.yaml

v0.0-experimental

Files with SHA256: 1afc9fcb11bc58a8ad4c667d3f3e723182d3f852edd0c12d0e6b082b1f7a9be6