@KandooAI ok. thank you

I feel like the answer to this question is always "try both and see what you prefer". There's no right or wrong, only what makes you happy!

That said, changes to how CLIP works in SDXL make clip skip less important than it is for SD1.5. Most of what it used to do can now be done by adjusting CFG (although some anime checkpoints are still being trained specifically for clip skip 2, probably out of habit).

No refiner needed!

@RunDiffusion Thanks, but will it be sharper/more defined with the Refiner or worse? I know for things like nipples the refiner sucks but it sometimes makes a clearer image. I wonder if anyone will train the refiner?

@EricRollei21 A refiner is always a great idea. You can even use an SD 1.5 model as a refiner (like in Fooocus). A refiner is just a second step in the diffusion pipeline. I would recommend trying it but with low strength and work up from there.

@EricRollei21 The results of my tests with comfyui show that there are a lot of concepts missing from the refiner model, and when you include concepts in your Prompt that it doesn't have, you will get unsatisfactory images. The most impressive ones are the NSFW content and the negative expressions of the characters. refiner will refine your otherwise obvious features into mediocrity.
My experience has been to use refiner when all of your Prompt can be accurately fed by sai's base model, and most of the time I only use it as a model for Upscale.

@suede2031691  I'm surprised to read you use the refiner for upscale, but maybe I didn't understand. I use the refiner in most workflows at base, then upscale with the base model and controlnet tile if it's sd 1.5 or if SDXL then I use IPAdapter instead. Mostly I upscale using tiled sampler.

I was going to train Lora on V6 for richer facial expressions, but checking the changelog for Kohya-ss's script, I see that it still doesn't separate the two text encoders for SDXL, so I'll just have to wait for the OP's next release, and look forward to progress on character expressions and violent horror-type NSFW content.

For some great photo realism, use prompts with the person at the front then say what that person is doing, then add the filler tokens. e.g. beautiful woman playing tennis in the rain, stormy weather, dark clouds, action shot, intense scene.

Natural language prompts don't work well with SDXL in general.

@RunDiffusion This is not true. It's simply that these custom models are not trained with Natural Language prompts.

This custom model has been trained on short keyword combos rather than natural language. The majority of SDXL models that I have used respond better to natural language prompts than to any other form of prompting.

In my experience only those custom models trained using the old SD1.5 captioning methodology struggle to use natural language prompts.

That said, there is a good reason for there to be models that can still use the old ways. As most people, like yourself, mistakenly have the same belief (A belief perpetuated by SD1.5 creators rushing their SDXL models for the CIVITAI competition and not taking the time to research and develop new methods for the new model but instead using their existing experience) and will find this model more effective when used in conjunction with the old methods.

Further, I'm relatively certain that if you tested this in Automatic1111, currently they only make use of one of the 2 SDXL text encoder clip models. Whereas other UI, like ComfyUI/Foooocus act upon both encoders for most accurate testing.

@Triple_Headed_Monkey But comfyui's TextEncode is not so simple to divide into natural language and phrases when you use it, some prompts are better in CLIP_G even if they are phrases, and CLIP_L feels more suitable for some quality control prompts. anyway, it's quite complicated.

@suede2031691 Sure. Personally I only use the simple text encoder view with one prompt box. However this sends the prompt to both text encoders in ComfyUI.

With that I get pretty good results. My prompts are usually worded like the caption for an image in a museum display exhibit.

A title/concept "A Digitial anime artwork of" followed by the prompt itself "a woman with beautiful eyes eating cornflakes" then afterwards follow up with a few keywords/phrases to bolster the context and setting "she is sat in a dark room lit by a single torch,".

For your positive prompt, use "perfect eyes" or specify a color of eyes. That should help.
For the neg, "bad eyes", "deformed eyes", etc

Magic! We’ll create magic!

Don’t use SDXL for 512x512 images. It will look horrible.

Try generating at 768x1168, then simply downscale them with your preferred tool of your choice.

does upscaler work for sdxl ? last time I checked iirc, upscaling didn't do much quality improvement for images, you had to start with higher res images above 1024px

Can confirm: Yes, it does

@KandooAI thanks

Or if using a1111, leave your vae setting at None. It is rare you need to replace the baked in vae. Using sdxl_vae.safetensors you are literally replacing the built-in vae with itself. The one exception would be if you wanted to save some vram and use the fp16 sdxl vae. Look for sdxl-vae-fp16-fix from madebyollin on huggingface

@joeuser12 Thank you so very much! :D

Wow, that was fast, thanks ;D

This explains the weird artifacts I have in my image and that won't go away even with the high offset noise. I have to play around a bit, maybe with lower discount values. So far I cannot train on the checkpoint without either getting weird results or getting noise artifacts from the pyramid noise.

FYI: I made that observation too.

I agree. This is probably one of the best models available as of November, 2023. Well done!

What do you use? Automatic 1111? Which resolution are you using and which VAE?

@Aerth If I use Automatic1111, the vae I use is vae-ft-ema-560000-ema-pruned.safetensors, my PC is an RTX 3070 8GB, I5 11000, I clarify that English is not my native language

@SrGatoide It's not mine either, it's ok. The VAE you use is for models based on SD1.5. This one is a SDXL and requires other VAE. Luckily it is baked in, meaning it's already included.

Choose Automatic in your A1111 and you should be good.

@Aerth I'm going to try it 👍

@Aerth thank you i turned off VAE and it help me.

@restartinstar423 Although it worked for me, I stopped using it because it causes my PC to freeze

@SrGatoide It's probably due to the way A1111 loads the models which can be 'heavy' and is certainly slow. You should try ComfyUI, it''s lighter in resource usage.

@SrGatoide Try watching this video to optimise SDXL for A1111. It worked for me.

https://www.youtube.com/watch?v=7mlJQ6viH20

@Aerth I'll try it, I just found out about its existence

Same problem for me, I am trying to reproduce your tree house picture, and it looks beautiful to near the end when... pixels.

@decourcy3348  What do you use? A1111? Generally 'pixels in the end' are a problem when the engine tries to decode the latent space to convert it to an image, which is done with a VAE.
Possibly you use a wrong VAE, one for SD1.5 for example.
This model does not need external VAE because it is already included in. So VAE parameter should be 'Automatic' or 'None'

@KandooAI Any update on that? I'd love to try it out as well

I have found the best way to fix that it is retrain your lora against this model instead

Any update?!

Just point pretrained_model_name_or_path directly at this safetensors file. You don't need to do anything special. You train against it like you would train against the base sdxl model.

@joeuser12 you think its better to make checkpoints in Dreambooth on this model or Loras?

Loads fine in a1111. Make sure your a1111 is up to date.

Check the file hash, your download may have been corrupted.

i have same problem. Failed to create model quickly; will retry using slow method.

Use the LCM LoRA

same bro, i hope they can help us.

You are probably running out of vram and your Windows drivers are using system ram which is really slow. Try using the fp16 sdxl vae instead of the baked in one. Look for madebyollin/sdxl-vae-fp16-fix

I didn't have that issue on Automatic1111, but I had a similar one in my custom AI app, which was a similar error. Don't use SD1.5 loras etc. In other words, it might be because you have incompatible Loras. Try without any loras, then if it works download and use SDXL compatible Loras. Cheers

It is fine for me too. ^^

checkout the filter Eddie

@mramer723 It appeared for me when the new model version came out 👍