@Joschek so clip skip 2 cuts off the final layer of the clip network, and this is what anime models use, IIRC the historical reason is due to an accident in the training a huge anime model that many others have as a base. the language control is less precise because the final layer is cut off, but it's actually well suited for a big list of comma separated tags (such as what those models were trained on) like 1girl, hyperdetailed, absurdres, twintails, ...

clip skip 1 (it should be called clip skip 0, because no layers of the clip network are cut off, but the name stuck around for historical reasons) is more suited to natural language descriptions like "a woman is hogtied on the floor in an elegant dungeon. she is wearing a red ballgag and quivering in anticipation of what comes next."

most users of realistic models use clip skip 1, because this is how those models were trained... even if your prompt is just a comma separated list of tags, clip skip 1 setting is still perfectly capable of interpreting that. it can handle the extra precision of natural grammar, but it doesn't NEED it. under the hood, when the clip model processes that "list of tags" prompt and hands it off to the cross attention layer in the U-net, it will still "talk" to the U-net using the clip skip 1 language that the U-net understands.

the biggest issue occurs when trying to use any clip skip 2 model (checkpoint or LORA) while you have the clip skip 1 configuration. clip skip 2 models were not trained to handle that final layer of the clip network, so when you pass it that kind of output from the prompt, it spits out garbage that looks like a deep fried meme. if you add any clip skip 2 LORA, you have to change the entire configuration to use clip skip 2 to avoid these bad results, even if you are using a checkpoint that was trained at clip skip 1 and 8 other LORAs trained with clip skip 1.

now, you can use clip skip 2 configuration with realism models and LORAs that were trained at clip skip 1, but.... there is a cost. the images it generates won't match the user's prompts as closely due to losing the precision of the last clip layer... so clip skip 2 LORAs "can" be used with realism models, but it forces the user to enable the clip skip 2 setting.

for best prompting results, LORAs intended for realism should be trained with clip skip 1 and natural language style captions, and LORAs intended for anime should be trained with clip skip 2 and booru style tags.

in summary, stable diffusion translates prompts into embeddings that steer the generation process. higher levels of clip skip cut off more layers of the network that translates prompts into embeddings, effectively limiting the complexity of the communication at the interface between the vision-language model (CLIP) and the denoiser (U-net).

anime model U-net has the stable diffusion equivalent of a 3rd grade reading level. it can't handle the richer clip skip 1 language embeddings. clip skip 2 LORA is more "broadly compatible" with the current ecosystem of anime models, but it forces the user to enable the clip skip 2 setting in their WebUI, dragging everything else down to its same level of being basically retarded.

@disapproving_daddy Thanks for the info on clip skip, very helpful.