ksu_v10.safetensors

v1.0

This is the official DMD2 implementation LoRA from the Improved <a target="_blank" rel="ugc" href="https://arxiv.org/abs/2405.14867">Distribution Matching Distillation for Fast Image Synthesis</a> paper.<h1 id="dmd2_sdxl_4step_lora">dmd2_sdxl_4step_lora</h1>Original model at <a target="_blank" rel="ugc" href="https://huggingface.co/tianweiy/DMD2">Huggingface tianweiy/DMD2</a>.<h2 id="recommended-settings">Recommended Settings</h2>Steps: 8LoRA weight: 0.7Sampler: LCMScheduler: Normal, SimpleCFG: 1.4 - 1.8I use similar settings on both a low resolution base image, and the upscaled refinement pass (hires fix). The settings above are my personal recommendations, but I suggest you play around with the settings to find what works for you and the model you are using.<h2 id="combining-both-loras">Combining Both LoRAs</h2>It's also been reported to perform better to use both the fp32 and fp16 at the same time at 0.5 weight each.<h2 id="recommended-lora-weight">Recommended LoRA weight</h2>Load the LoRA like any other. I personally recommend a weight of 0.7, but others swear it's better at 1.0 in weight.<h2 id="usage">Usage</h2>You should be able to use it with any SDXL or similar checkpoint. Use the settings above.You can also merge this into a checkpoint and save that, to include the result of the LoRA into the checkpoint directly. This means you no longer need to load this as a LoRA, and you can still use it with LCM and low steps for good result.The LoRA itself doesn't add any speed improvement, it just allows you to generate at fewer steps, with better result.If the results are washed out, desaturated or bland, try increasing the CFG.In the comments below there are a bunch of interesting discussions, so please explore them. Here are a few noteworthy:<a target="_blank" rel="ugc" href="https://civarchive.com/models/1608870/dmd2-speed-lora-sdxl-pony-illustrious?commentId=991371&amp;dialog=commentThread">https://civarchive.com/models/1608870/dmd2-speed-lora-sdxl-pony-illustrious?commentId=991371&amp;dialog=commentThread</a><a target="_blank" rel="ugc" href="https://civarchive.com/models/1608870/dmd2-speed-lora-sdxl-pony-illustrious?commentId=1179404&amp;dialog=commentThread">https://civarchive.com/models/1608870/dmd2-speed-lora-sdxl-pony-illustrious?commentId=1179404&amp;dialog=commentThread</a><a target="_blank" rel="ugc" href="https://civarchive.com/models/1608870/dmd2-speed-lora-sdxl-pony-illustrious?commentId=1034300&amp;dialog=commentThread">https://civarchive.com/models/1608870/dmd2-speed-lora-sdxl-pony-illustrious?commentId=1034300&amp;dialog=commentThread</a><a target="_blank" rel="ugc" href="https://civarchive.com/models/1608870/dmd2-speed-lora-sdxl-pony-illustrious?commentId=1123128&amp;dialog=commentThread">https://civarchive.com/models/1608870/dmd2-speed-lora-sdxl-pony-illustrious?commentId=1123128&amp;dialog=commentThread</a><h2 id="licence">Licence</h2><a target="_blank" rel="ugc" href="https://huggingface.co/tianweiy/DMD2#license">Creative Commons Attribution Non Commercial 4.0</a>I am not the creator of the model, just uploading the model here to make it easier to download and reference in creations. Full credit goes to <a target="_blank" rel="ugc" href="https://huggingface.co/tianweiy">Tianwei Yin</a>.

DMD2 Speed LoRA [SDXL, Pony, Illustrious]

dmd2_sdxl_4step_loradmd2_sdxl_4step_lora_fp32

dmd2_sdxl_4step_lora.safetensors

dmd2_sdxl_4step_lora_fp32

dmd2_sdxl_4step_lora_fp16

Forward from ://<a target="_blank" rel="ugc" href="http://huggingface.co/tianweiy/DMD2/tree/main">huggingface.co/tianweiy/DMD2/tree/main</a>这个模型在 cfg=1 ,lcm karras 下工作良好，另外在在线生图器的加速模型可以使用Euler a作为替代step设置为10-12步可以获得非常好的效果。SDXL和pony均可使用。Forward from ://<a target="_blank" rel="ugc" href="http://huggingface.co/tianweiy/DMD2/tree/main">huggingface.co/tianweiy/DMD2/tree/main</a>This model works well under cfg=1 and lcm karras, and Euler a can be used as a substitute for the acceleration model in online graph generatorsSetting step to 10-12 can achieve very good results.Both SDXL and Pony can be used.===================================================<a target="_blank" rel="ugc" href="https://arxiv.org/abs/2405.14867">Improved Distribution Matching Distillation for Fast Image Synthesis</a>, Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Frédo Durand, William T. Freeman<h2 id="contact-47w9r1z1g">Contact</h2>Feel free to contact us if you have any questions about the paper!Tianwei Yin <a target="_blank" rel="ugc" href="mailto:tianweiy@mit.edu">tianweiy@mit.edu</a><h2 id="usage-kzyt8cgo5">Usage</h2>We can use the standard diffuser pipeline:4-step UNet generation<pre><code>import torch
from diffusers import DiffusionPipeline, UNet2DConditionModel, LCMScheduler
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
base_model_id = "stabilityai/stable-diffusion-xl-base-1.0"
repo_name = "tianweiy/DMD2"
ckpt_name = "dmd2_sdxl_4step_unet_fp16.bin"
# Load model.
unet = UNet2DConditionModel.from_config(base_model_id, subfolder="unet").to("cuda", torch.float16)
unet.load_state_dict(torch.load(hf_hub_download(repo_name, ckpt_name), map_location="cuda"))
pipe = DiffusionPipeline.from_pretrained(base_model_id, unet=unet, torch_dtype=torch.float16, variant="fp16").to("cuda")
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
prompt="a photo of a cat"

# LCMScheduler's default timesteps are different from the one we used for training 
image=pipe(prompt=prompt, num_inference_steps=4, guidance_scale=0, timesteps=[999, 749, 499, 249]).images[0]
</code></pre>4-step LoRA generation<pre><code>import torch
from diffusers import DiffusionPipeline, UNet2DConditionModel, LCMScheduler
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
base_model_id = "stabilityai/stable-diffusion-xl-base-1.0"
repo_name = "tianweiy/DMD2"
ckpt_name = "dmd2_sdxl_4step_lora_fp16.safetensors"
# Load model.
pipe = DiffusionPipeline.from_pretrained(base_model_id, torch_dtype=torch.float16, variant="fp16").to("cuda")
pipe.load_lora_weights(hf_hub_download(repo_name, ckpt_name))
pipe.fuse_lora(lora_scale=1.0) # we might want to make the scale smaller for community models

pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
prompt="a photo of a cat"

# LCMScheduler's default timesteps are different from the one we used for training 
image=pipe(prompt=prompt, num_inference_steps=4, guidance_scale=0, timesteps=[999, 749, 499, 249]).images[0]
</code></pre>1-step UNet generation<pre><code>import torch
from diffusers import DiffusionPipeline, UNet2DConditionModel, LCMScheduler
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
base_model_id = "stabilityai/stable-diffusion-xl-base-1.0"
repo_name = "tianweiy/DMD2"
ckpt_name = "dmd2_sdxl_1step_unet_fp16.bin"
# Load model.
unet = UNet2DConditionModel.from_config(base_model_id, subfolder="unet").to("cuda", torch.float16)
unet.load_state_dict(torch.load(hf_hub_download(repo_name, ckpt_name), map_location="cuda"))
pipe = DiffusionPipeline.from_pretrained(base_model_id, unet=unet, torch_dtype=torch.float16, variant="fp16").to("cuda")
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
prompt="a photo of a cat"
image=pipe(prompt=prompt, num_inference_steps=1, guidance_scale=0, timesteps=[399]).images[0]
</code></pre>4-step T2I Adapter<pre><code>from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter, AutoencoderKL, UNet2DConditionModel, LCMScheduler
from diffusers.utils import load_image, make_image_grid
from controlnet_aux.canny import CannyDetector
from huggingface_hub import hf_hub_download
import torch

# load adapter
adapter = T2IAdapter.from_pretrained("TencentARC/t2i-adapter-canny-sdxl-1.0", torch_dtype=torch.float16, varient="fp16").to("cuda")

vae=AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)

base_model_id = "stabilityai/stable-diffusion-xl-base-1.0"
repo_name = "tianweiy/DMD2"
ckpt_name = "dmd2_sdxl_4step_unet_fp16.bin"
# Load model.
unet = UNet2DConditionModel.from_config(base_model_id, subfolder="unet").to("cuda", torch.float16)
unet.load_state_dict(torch.load(hf_hub_download(repo_name, ckpt_name), map_location="cuda"))

pipe = StableDiffusionXLAdapterPipeline.from_pretrained(
 base_model_id, unet=unet, vae=vae, adapter=adapter, torch_dtype=torch.float16, variant="fp16", 
).to("cuda")
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
pipe.enable_xformers_memory_efficient_attention()

canny_detector = CannyDetector()

url = "https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/org_canny.jpg"
image = load_image(url)

# Detect the canny map in low resolution to avoid high-frequency details
image = canny_detector(image, detect_resolution=384, image_resolution=1024)#.resize((1024, 1024))

prompt = "Mystical fairy in real, magic, 4k picture, high quality"

gen_images = pipe(
 prompt=prompt,
 image=image,
 num_inference_steps=4,
 guidance_scale=0, 
 adapter_conditioning_scale=0.8, 
 adapter_conditioning_factor=0.5,
 timesteps=[999, 749, 499, 249]
).images[0]
gen_images.save('out_canny.png')
</code></pre>For more information, please refer to the <a target="_blank" rel="ugc" href="https://github.com/tianweiy/DMD2">code repository</a><h2 id="license-4nf7hal1i">License</h2>Improved Distribution Matching Distillation is released under <a target="_blank" rel="ugc" href="https://creativecommons.org/licenses/by-nc-sa/4.0/deed.en">Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License</a>.<h2 id="citation-2jd3jtnf6">Citation</h2>If you find DMD2 useful or relevant to your research, please kindly cite our papers:<pre><code>@article{yin2024improved,
 title={Improved Distribution Matching Distillation for Fast Image Synthesis},
 author={Yin, Tianwei and Gharbi, Micha{\"e}l and Park, Taesung and Zhang, Richard and Shechtman, Eli and Durand, Fredo and Freeman, William T},
 journal={arXiv:2405.14867},
 year={2024}
}

@inproceedings{yin2024onestep,
 title={One-step Diffusion with Distribution Matching Distillation},
 author={Yin, Tianwei and Gharbi, Micha{\"e}l and Zhang, Richard and Shechtman, Eli and Durand, Fr{\'e}do and Freeman, William T and Park, Taesung},
 booktitle={CVPR},
 year={2024}
}
</code></pre><h2 id="acknowledgments-93kxxby66">Acknowledgments</h2>This work was done while Tianwei Yin was a full-time student at MIT. It was developed based on our reimplementation of the original DMD paper. This work was supported by the National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, <a target="_blank" rel="ugc" href="http://iaifi.org/">http://iaifi.org/</a>), by NSF Grant 2105819, by NSF CISE award 1955864, and by funding from Google, GIST, Amazon, and Quanta Computer.

Files with SHA256: a374289e9446d7f14d2037c4b3770756b7b52c292142a691377c3c755010a1bb