<h3 id="this-is-not-a-stable-diffusion-model.-it-requires-its-own-web-ui.-read-below-for-more-information.-6budy5uay"><strong><span style="color:rgb(250, 82, 82)">This is not a stable diffusion model. It requires its own web UI. Read below for more information.</span></strong></h3><p>note: I am not the author of the model, the link to the original HF repository is below.</p><p>Kandinsky 3.1 is available for use now:</p><ul><li><p>The source code for the model, released under the Apache 2.0 license, can be found on GitHub: <a target="_blank" rel="ugc" href="https://github.com/ai-forever/Kandinsky-3">https://github.com/ai-forever/Kandinsky-3</a></p></li><li><p>The model is available for download and use on Hugging Face: <a target="_blank" rel="ugc" href="https://huggingface.co/ai-forever/Kandinsky3.1">https://huggingface.co/ai-forever/Kandinsky3.1</a></p></li><li><p>The model has its own "AUTOMATIC1111": Kubin, for local use: <a target="_blank" rel="ugc" href="https://github.com/seruva19/kubin">https://github.com/seruva19/kubin</a></p></li><li><p>You can test Kandinsky's capabilities on the FusionBrain website: <a target="_blank" rel="ugc" href="https://www.sberbank.com/promo/kandinsky/">https://www.sberbank.com/promo/kandinsky/</a></p></li></ul><h1 id="the-original-description-of-the-repository:-7mrfaov7e">The original description of the repository:</h1><pre><code>license: apache-2.0</code></pre><h1 id="kandinsky-3:-text-to-image-diffusion-model-r4oltfn1k">Kandinsky-3: Text-to-image diffusion model</h1><p><a target="_blank" rel="ugc" href="https://huggingface.co/ai-forever/Kandinsky3.1/blob/main/assets/title.jpg"><strong><u><img src="https://huggingface.co/ai-forever/Kandinsky3.1/resolve/main/assets/title.jpg" alt /></u></strong></a></p><p><a target="_blank" rel="ugc" href="https://habr.com/ru/companies/sberbank/articles/775590/"><strong><u>Kandinsky 3.0 Post</u></strong></a> | <a target="_blank" rel="ugc" href="https://habr.com/ru/companies/sberbank/articles/805337/"><strong><u>Kandinsky 3.1 Post</u></strong></a> | <a target="_blank" rel="ugc" href="https://ai-forever.github.io/Kandinsky-3"><strong><u>Project Page</u></strong></a> | <a target="_blank" rel="ugc" href="https://fusionbrain.ai/"><strong><u>Generate</u></strong></a> | <a target="_blank" rel="ugc" href="https://t.me/kandinsky21_bot"><strong><u>Telegram-bot</u></strong></a> | <a target="_blank" rel="ugc" href="https://arxiv.org/pdf/2312.03511.pdf"><strong><u>Technical Report</u></strong></a> | <a target="_blank" rel="ugc" href="https://huggingface.co/kandinsky-community/kandinsky-3"><strong><u>HuggingFace</u></strong></a></p><h1 id="kandinsky-3.1:-5ayv3x4z7">Kandinsky 3.1:</h1><h2 id="description:-7r638f4p3">Description:</h2><p>We present Kandinsky 3.1, the follow-up to the Kandinsky 3.0 model, a large-scale text-to-image generation model based on latent diffusion, continuing the series of text-to-image Kandinsky models and reflecting our progress to achieve higher quality and realism of image generation, which we have enhanced and enriched with a variety of useful features and modes to give users more opportunities to fully utilise the power of our new model.</p><h2 id="kandinsky-flash-(kandinsky-3.0-refiner)-0h0k33c02">Kandinsky Flash (Kandinsky 3.0 Refiner)</h2><p><img src="https://huggingface.co/ai-forever/Kandinsky3.1/resolve/main/assets/butterly_effect.jpg" /></p><p>Diffusion models have problems with fast image generation. To address this problem, we trained a Kandinksy Flash model based on the <a target="_blank" rel="ugc" href="https://arxiv.org/abs/2311.17042"><strong><u>Adversarial Diffusion Distillation</u></strong></a> approach with some modifications: we trained the model on latents, which reduced the memory overhead and removed distillation loss as it did not affect the training. Also, we applied Kandinsky Flash model to images generated from Kandinsky 3.0 to improve visual quality of generated images.</p><h3 id="architecture-ohgss9yi7">Architecture</h3><p>For training Kandinsky Flash we used the following architecture of discriminator. It is the half of Kandinsky 3.0 U-Net encoder with additional head predictions.</p><p><img src="https://huggingface.co/ai-forever/Kandinsky3.1/resolve/main/assets/architecture.png" /></p><h3 id="how-to-use:-n0opq59mf">How to use:</h3><p>Check our jupyter notebooks with examples in <code>./examples</code> folder</p><pre><code>from kandinsky3 import get_T2I_Flash_pipeline

device_map = torch.device('cuda:0')
dtype_map = {
    'unet': torch.float32,
    'text_encoder': torch.float16,
    'movq': torch.float32,
}

t2i_pipe = get_T2I_Flash_pipeline(
    device_map, dtype_map
)

res = t2i_pipe("A cute corgi lives in a house made out of sushi.")
</code></pre><h3 id="kandinsky-inpainting-je2cvvn2b">Kandinsky Inpainting</h3><p>Also, we released a newer version of inpainting model, which we additionally trained the model on the object detection dataset. This allowed to get more stable generation of objects. The new weights are available at <a target="_blank" rel="ugc" href="https://huggingface.co/ai-forever/Kandinsky3.1"><strong><u>ai-forever/Kandinsky3.1</u></strong></a>. Check the usage <a target="_blank" rel="ugc" href="https://github.com/ai-forever/Kandinsky-3?tab=readme-ov-file#2-inpainting"><strong><u>example</u></strong></a>.</p><h2 id="prompt-beautification-26dgl95tp">Prompt beautification</h2><p><img src="https://huggingface.co/ai-forever/Kandinsky3.1/resolve/main/assets/prompt_beautifcation.png" /></p><p>Prompt plays crucial role in text-to-image generation. So, in Kandinsky 3.1 we decided to use language model for making prompt better. We used Intel's <a target="_blank" rel="ugc" href="https://huggingface.co/Intel/neural-chat-7b-v3-1"><strong><u>neural-chat-7b-v3-1</u></strong></a> with the following system prompt as the LLM:</p><pre><code>### System: You are a prompt engineer. Your mission is to expand prompts written by user. You should provide the best prompt for text to image generation in English.
### User:
{prompt}
### Assistant:
{answer of the model}
</code></pre><h2 id="kandisuperres-6nqar4kt0">KandiSuperRes</h2><p><img src="https://huggingface.co/ai-forever/Kandinsky3.1/resolve/main/assets/superres.png" /></p><p>To learn more about KandiSuperRes, please checkout: <a target="_blank" rel="ugc" href="https://github.com/ai-forever/KandiSuperRes/"><strong><u>https://github.com/ai-forever/KandiSuperRes/</u></strong></a></p><h2 id="kandinsky-ip-adapter-and-kandinsky-controlnet-sdv5o7oa1">Kandinsky IP-Adapter &amp; Kandinsky ControlNet</h2><p><img src="https://huggingface.co/ai-forever/Kandinsky3.1/resolve/main/assets/ip-adapter.png" /></p><p>To allow using image as condition in Kandinsky model, we trained IP-Adapter and HED-based ControlNet model. For more details please check out: <a target="_blank" rel="ugc" href="https://github.com/ai-forever/kandinsky3-diffusers"><strong><u>https://github.com/ai-forever/kandinsky3-diffusers</u></strong></a></p><h1 id="kandinsky-3.0:-tepp71ipp">Kandinsky 3.0:</h1><h2 id="description:-smlct922z">Description:</h2><p>Kandinsky 3.0 is an open-source text-to-image diffusion model built upon the Kandinsky2-x model family. In comparison to its predecessors, Kandinsky 3.0 incorporates more data and specifically related to Russian culture, which allows to generate pictures related to Russin culture. Furthermore, enhancements have been made to the text understanding and visual quality of the model, achieved by increasing the size of the text encoder and Diffusion U-Net models, respectively.</p><p>For more information: details of training, example of generations check out our <a target="_blank" rel="ugc" href="https://huggingface.co/ai-forever/Kandinsky3.1/blob/main/README.md"><strong><u>post</u></strong></a>. The english version will be released in a couple of days.</p><h2 id="architecture-details:-3kqgktxmd">Architecture details:</h2><p><a target="_blank" rel="ugc" href="https://huggingface.co/ai-forever/Kandinsky3.1/blob/main/assets/kandinsky.jpg"><strong><u><img src="https://huggingface.co/ai-forever/Kandinsky3.1/resolve/main/assets/kandinsky.jpg" alt /></u></strong></a></p><p>Architecture consists of three parts:</p><ul><li><p>Text encoder Flan-UL2 (encoder part) - 8.6B</p></li><li><p>Latent Diffusion U-Net - 3B</p></li><li><p>MoVQ encoder/decoder - 267M</p></li></ul><h2 id="models-36yz0dh4w">Models</h2><p>We release our two models:</p><ul><li><p><a target="_blank" rel="ugc" href="https://huggingface.co/ai-forever/Kandinsky3.1/blob/main/README.md"><strong><u>Base</u></strong></a>: Base text-to-image diffusion model. This model was trained over 2M steps on 400 A100</p></li><li><p><a target="_blank" rel="ugc" href="https://huggingface.co/ai-forever/Kandinsky3.1/blob/main/README.md"><strong><u>Inpainting</u></strong></a>: Inpainting version of the model. The model was initialized from final checkpoint of base model and trained 250k steps on 300 A100.</p></li></ul><h2 id="installing-t1183v02f">Installing</h2><p>To install repo first one need to create conda environment:</p><pre><code>conda create -n kandinsky -y python=3.8;
source activate kandinsky;
pip install torch==1.10.1+cu111 torchvision==0.11.2+cu111 torchaudio==0.10.1 -f https://download.pytorch.org/whl/cu113/torch_stable.html;
pip install -r requirements.txt;
</code></pre><p>The exact dependencies is got using <code>pip freeze</code> and can be found in <code>exact_requirements.txt</code></p><h2 id="how-to-use:-yhv4wkm4h">How to use:</h2><p>Check our jupyter notebooks with examples in <code>./examples</code> folder</p><h3 id="1.-text2image-enpmu4aoc">1. text2image</h3><pre><code>import sys
sys.path.append('..')

import torch
from kandinsky3 import get_T2I_pipeline

device_map = torch.device('cuda:0')
dtype_map = {
    'unet': torch.float32,
    'text_encoder': torch.float16,
    'movq': torch.float32,
}

t2i_pipe = get_T2I_pipeline(
    device_map, dtype_map,
)
res = t2i_pipe("A cute corgi lives in a house made out of sushi.")

res[0]
</code></pre><h3 id="2.-inpainting-oc6usg4ei">2. inpainting</h3><pre><code>from kandinsky3 import get_inpainting_pipeline

device_map = torch.device('cuda:0')
dtype_map = {
    'unet': torch.float16,
    'text_encoder': torch.float16,
    'movq': torch.float32,
}

pipe = get_inpainting_pipeline(
    device_map, dtype_map,
)

image = ... # PIL Image
mask = ... # Numpy array (HxW). Set 1 where image should be masked
image = inp_pipe( "A cute corgi lives in a house made out of sushi.", image, mask)
</code></pre><h2 id="authors-3z23vruyf">Authors</h2><ul><li><p>Vladimir Arkhipkin: <a target="_blank" rel="ugc" href="https://github.com/oriBetelgeuse"><strong><u>Github</u></strong></a></p></li><li><p>Anastasia Maltseva <a target="_blank" rel="ugc" href="https://github.com/NastyaMittseva"><strong><u>Github</u></strong></a></p></li><li><p>Andrei Filatov <a target="_blank" rel="ugc" href="https://github.com/anvilarth"><strong><u>Github</u></strong></a>,</p></li><li><p>Igor Pavlov: <a target="_blank" rel="ugc" href="https://github.com/boomb0om"><strong><u>Github</u></strong></a></p></li><li><p>Julia Agafonova</p></li><li><p>Arseniy Shakhmatov: <a target="_blank" rel="ugc" href="https://github.com/cene555"><strong><u>Github</u></strong></a>, <a target="_blank" rel="ugc" href="https://t.me/gradientdip"><strong><u>Blog</u></strong></a></p></li><li><p>Andrey Kuznetsov: <a target="_blank" rel="ugc" href="https://github.com/kuznetsoffandrey"><strong><u>Github</u></strong></a>, <a target="_blank" rel="ugc" href="https://t.me/complete_ai"><strong><u>Blog</u></strong></a></p></li><li><p>Denis Dimitrov: <a target="_blank" rel="ugc" href="https://github.com/denndimitrov"><strong><u>Github</u></strong></a>, <a target="_blank" rel="ugc" href="https://t.me/dendi_math_ai"><strong><u>Blog</u></strong></a></p></li></ul><h2 id="citation-yx896njg3">Citation</h2><pre><code>@misc{arkhipkin2023kandinsky,
      title={Kandinsky 3.0 Technical Report}, 
      author={Vladimir Arkhipkin and Andrei Filatov and Viacheslav Vasilev and Anastasia Maltseva and Said Azizov and Igor Pavlov and Julia Agafonova and Andrey Kuznetsov and Denis Dimitrov},
      year={2023},
      eprint={2312.03511},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}</code></pre>

Files with SHA256: 47c9e62b1b230806a85edd1f04700bbb6f37d48b2a982afb6be94b6875be488c