Demos:<ul><li>multimodalart: <a target="_blank" rel="ugc" href="https://hf.co/spaces/multimodalart/stable-cascade">https://hf.co/spaces/multimodalart/stable-cascade</a></li><li>ehristoforu: <a target="_blank" rel="ugc" href="https://hf.co/spaces/ehristoforu/Stable-Cascade">https://hf.co/spaces/ehristoforu/Stable-Cascade</a></li></ul><h1 id="heading-1071">Stable Cascade</h1>This model is built upon the <a target="_blank" rel="ugc" href="https://openreview.net/forum?id=gU58d5QeGv">Würstchen</a> architecture and its maindifference to other models like Stable Diffusion is that it is working at a much smaller latent space. Why is thisimportant? The smaller the latent space, the faster you can run inference and the cheaper the training becomes.How small is the latent space? Stable Diffusion uses a compression factor of 8, resulting in a 1024x1024 image beingencoded to 128x128. Stable Cascade achieves a compression factor of 42, meaning that it is possible to encode a1024x1024 image to 24x24, while maintaining crisp reconstructions. The text-conditional model is then trained in thehighly compressed latent space. Previous versions of this architecture, achieved a 16x cost reduction over StableDiffusion 1.5. &lt;br&gt; &lt;br&gt;Therefore, this kind of model is well suited for usages where efficiency is important. Furthermore, all known extensionslike finetuning, LoRA, ControlNet, IP-Adapter, LCM etc. are possible with this method as well.<h2 id="heading-1072">Model Details</h2><h3 id="heading-1073">Model Description</h3>Stable Cascade is a diffusion model trained to generate images given a text prompt.<ul><li>Developed by: Stability AI</li><li>Funded by: Stability AI</li><li>Model type: Generative text-to-image model</li></ul><h3 id="heading-1074">Model Sources</h3>For research purposes, we recommend our <code>StableCascade</code> Github repository (<a target="_blank" rel="ugc" href="https://github.com/Stability-AI/StableCascade">https://github.com/Stability-AI/StableCascade</a>).<ul><li>Repository: <a target="_blank" rel="ugc" href="https://github.com/Stability-AI/StableCascade">https://github.com/Stability-AI/StableCascade</a></li><li>Paper: <a target="_blank" rel="ugc" href="https://openreview.net/forum?id=gU58d5QeGv">https://openreview.net/forum?id=gU58d5QeGv</a></li></ul><h3 id="heading-1075">Model Overview</h3>Stable Cascade consists of three models: Stage A, Stage B and Stage C, representing a cascade to generate images,hence the name "Stable Cascade".Stage A &amp; B are used to compress images, similar to what the job of the VAE is in Stable Diffusion.However, with this setup, a much higher compression of images can be achieved. While the Stable Diffusion models use aspatial compression factor of 8, encoding an image with resolution of 1024 x 1024 to 128 x 128, Stable Cascade achievesa compression factor of 42. This encodes a 1024 x 1024 image to 24 x 24, while being able to accurately decode theimage. This comes with the great benefit of cheaper training and inference. Furthermore, Stage C is responsiblefor generating the small 24 x 24 latents given a text prompt. The following picture shows this visually.<img src="https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/8715570f-2c8a-4955-b10f-ee76bdd3de3a/width=525/8715570f-2c8a-4955-b10f-ee76bdd3de3a.jpeg" />For this release, we are providing two checkpoints for Stage C, two for Stage B and one for Stage A. Stage C comes witha 1 billion and 3.6 billion parameter version, but we highly recommend using the 3.6 billion version, as most work wasput into its finetuning. The two versions for Stage B amount to 700 million and 1.5 billion parameters. Both achievegreat results, however the 1.5 billion excels at reconstructing small and fine details. Therefore, you will achieve thebest results if you use the larger variant of each. Lastly, Stage A contains 20 million parameters and is fixed due toits small size.<h2 id="heading-1076">Evaluation</h2><img src="https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/e45be2de-66a4-4295-99f1-bd9273ae4b60/width=525/e45be2de-66a4-4295-99f1-bd9273ae4b60.jpeg" />According to our evaluation, Stable Cascade performs best in both prompt alignment and aesthetic quality in almost allcomparisons. The above picture shows the results from a human evaluation using a mix of parti-prompts (link) and aesthetic prompts. Specifically, Stable Cascade (30 inference steps) was compared against Playground v2 (50 inferencesteps), SDXL (50 inference steps), SDXL Turbo (1 inference step) and Würstchen v2 (30 inference steps).<h2 id="heading-1077">Code Example</h2>⚠️ Important: For the code below to work, you have to install <code>diffusers</code> from this branch while the PR is WIP.<pre><code>pip install git+https://github.com/kashif/diffusers.git@wuerstchen-v3</code></pre><pre><code>import torch

from diffusers import StableCascadeDecoderPipeline, StableCascadePriorPipeline

device = "cuda"

num_images_per_prompt = 2

prior = StableCascadePriorPipeline.from_pretrained("stabilityai/stable-cascade-prior", torch_dtype=torch.bfloat16).to(device)

decoder = StableCascadeDecoderPipeline.from_pretrained("stabilityai/stable-cascade", torch_dtype=torch.float16).to(device)

prompt = "Anthropomorphic cat dressed as a pilot"

negative_prompt = ""

prior_output = prior(

prompt=prompt,

height=1024,

width=1024,

negative_prompt=negative_prompt,

guidance_scale=4.0,

num_images_per_prompt=num_images_per_prompt,

num_inference_steps=20

)

decoder_output = decoder(

image_embeddings=prior_output.image_embeddings.half(),

prompt=prompt,

negative_prompt=negative_prompt,

guidance_scale=0.0,

output_type="pil",

num_inference_steps=10

).images

#Now decoder_output is a list with your PIL images</code></pre><h2 id="heading-1078">Uses</h2><h3 id="heading-1079">Direct Use</h3>The model is intended for research purposes for now. Possible research areas and tasks include<ul><li>Research on generative models.</li><li>Safe deployment of models which have the potential to generate harmful content.</li><li>Probing and understanding the limitations and biases of generative models.</li><li>Generation of artworks and use in design and other artistic processes.</li><li>Applications in educational or creative tools.</li></ul>Excluded uses are described below.<h3 id="heading-1080">Out-of-Scope Use</h3>The model was not trained to be factual or true representations of people or events,and therefore using the model to generate such content is out-of-scope for the abilities of this model.The model should not be used in any way that violates Stability AI's <a target="_blank" rel="ugc" href="https://stability.ai/use-policy">Acceptable Use Policy</a>.<h2 id="heading-1081">Limitations and Bias</h2><h3 id="heading-1082">Limitations</h3><ul><li>Faces and people in general may not be generated properly.</li><li>The autoencoding part of the model is lossy.</li></ul><h3 id="heading-1083">Recommendations</h3>The model is intended for research purposes only.<h2 id="heading-1084">How to Get Started with the Model</h2>Check out <a target="_blank" rel="ugc" href="https://github.com/Stability-AI/StableCascade">https://github.com/Stability-AI/StableCascade</a>

Stable Cascade

Official controlnet checkpoints for Stable Cascade.

Files with SHA256: e34eb825274ba01522af4b5ed3b84a3937902f4ecdcba0c6a3d7f133091250aa