<h1 id="astolfomix-xl">AstolfoMix-XL</h1><a target="_blank" rel="ugc" href="https://github.com/6DammK9/nai-anime-pure-negative-prompt/blob/main/ch01/della.md#spinoff-dgmla">"256c" and "215c" consist of multiple merge methods</a> (<a target="_blank" rel="ugc" href="https://github.com/6DammK9/nai-anime-pure-negative-prompt/blob/main/ch01/della.md">215a</a> "DELLA" and <a target="_blank" rel="ugc" href="https://github.com/6DammK9/nai-anime-pure-negative-prompt/blob/main/ch01/rebasin.md#spinoff-permutation-from-the-fermat-point">215b </a>"Git-Rebasin") of <a target="_blank" rel="ugc" href="https://github.com/6DammK9/nai-anime-pure-negative-prompt/blob/main/ch05/xl_docs/discovered_models.md">257 (216) discovered SDXL models (unfiltered)</a>. See <a target="_blank" rel="ugc" href="https://civarchive.com/articles/3409/diagram-of-unet-and-exotic-merging-methods">this article</a> for description. <a target="_blank" rel="ugc" href="https://huggingface.co/6DammK9/AstolfoMix-XL">Go to HuggingFace model page for sneek peek before "kind of official release"</a>. Contents / theories will not repeated with the <a target="_blank" rel="ugc" href="https://civarchive.com/models/155604/astolfomix">SD1 version</a> or <a target="_blank" rel="ugc" href="https://civarchive.com/models/255754/astolfomix-sd2">SD2 version</a>, contents below are exclusive for SDXL. <a target="_blank" rel="ugc" href="https://github.com/6DammK9/nai-anime-pure-negative-prompt/blob/main/ch05/README_XL.MD">Full docuementation / materials are in Github.</a><ul><li>Use CFG++ / PAG / SEG combo for sharper image.</li><li>For CivitAI users: CFG 3.0, CLIP skip 2, default VAE.</li><li>(215c) No quality tags are required, but excessive tags will not break either.</li><li>(256c) Slightly less consistint then 215c, because content seems less focus on human characters.</li><li><s>(255c) It won't obey your prompt. It would be a nice base model for training. 這個破模型比 SDXL1.0 更欠擬.</s> 255c is confirmed has broken recipe and picked wrong base model.</li></ul><h2 id="abstract">Abstract</h2>I present AstolfoMix-XL, a merge model focusing on "exploring <a target="_blank" rel="ugc" href="https://github.com/ljleb/sd-mecha/blob/main/sd_mecha/merge_methods.py">merging options</a>", while being stuck in merging <a target="_blank" rel="ugc" href="https://civarchive.com/models/257749/pony-diffusion-v6-xl">the popular model</a> and <a target="_blank" rel="ugc" href="https://www.reddit.com/r/StableDiffusion/comments/1ciyzn5/sd3_weights_are_never_going_to_be_released_are/">waiting for SD3</a>. Currently it is in anime style. Welcome back, Astolfo you are so cute!<h2 id="introduction">Introduction</h2>AstolfoMix-XL is the "scale up" merge from my previous models (SD1 / SD2), and hopefully find more useful ideas to expand <a target="_blank" rel="ugc" href="https://civarchive.com/articles/3409/diagram-of-unet-and-exotic-merging-methods">my article</a>, and reuse the valueable contents (<a target="_blank" rel="ugc" href="https://huggingface.co/collections/osanseviero/model-merging-65097893623330a3a51ead66">and concepts</a>) in the internet. The <a target="_blank" rel="ugc" href="https://wealthie.works/just-buy-the-haystack-investing-advice-that-works/">haystack</a> is too large for me to merge by hand. With a <a target="_blank" rel="ugc" href="https://github.com/ljleb/sd-mecha">dedicated merger</a> (like <a target="_blank" rel="ugc" href="https://github.com/arcee-ai/mergekit">mergekit</a>) doing the procedure in auto without saving intermediate models, I expect it can produce more general but high quality contents out of trained materials.<h2 id="related-works">Related Works</h2>This is tough. 這條學術路線究竟是復興, 還是倒退?Model merging is an extented topic on <a target="_blank" rel="ugc" href="https://en.wikipedia.org/wiki/Ensemble_learning">ensemble learning</a>, there are quite a bit of merging algorithms for AI models, and it is being <a target="_blank" rel="ugc" href="https://huggingface.co/docs/peft/developer_guides/model_merging">discussed formally</a> because LLM models are generally huge and too expensive to train. Many merging algorithms <a target="_blank" rel="ugc" href="https://github.com/martyn/safetensors-merge-supermario">are proposed in paper's own code repo (DARE)</a>, or <a target="_blank" rel="ugc" href="https://github.com/samuela/git-re-basin/blob/main/src/mnist_mlp_wm_many.py#L211">being forggen (MergeMany in Git Rebasin)</a>, or <a target="_blank" rel="ugc" href="https://huggingface.co/Enferlain/218xl/blob/main/test/umbra_mecha.safetensors">being undocuemented ("rotate" with more "matrix operation")</a>, meanwhile dedicated merger (<a target="_blank" rel="ugc" href="https://github.com/arcee-ai/mergekit">mergekit</a>, or <a target="_blank" rel="ugc" href="https://github.com/hako-mikan/sd-webui-supermerger">supermerger</a>) is being devloped.With a slight belief (<a target="_blank" rel="ugc" href="https://stats.stackexchange.com/questions/23789/is-ar1-a-markov-process">MDP=AR(1)</a> i.e. LLM merge to SD merge is viable), quite a bit of experiments (SD1 as <a target="_blank" rel="ugc" href="https://arxiv.org/abs/2203.05482">ModelSoup</a>, SD2 as model selection without <a target="_blank" rel="ugc" href="https://www.reddit.com/r/StableDiffusion/comments/xysavk/has_anyone_tried_the_novelai_leak/">alignment</a>, then <a target="_blank" rel="ugc" href="https://github.com/ogkalu2/Merge-Stable-Diffusion-models-without-distortion/pull/46">PR</a>, <a target="_blank" rel="ugc" href="https://github.com/ljleb/sd-mecha/pull/24">PR</a>, and <a target="_blank" rel="ugc" href="https://github.com/Xerxemi/auto-MBW-rt/pull/4">PR</a> because there is no codebase in the wild), and some luck, I am brave enough to "get it done" and publish it. It is very hard to with <s>little to none</s> community support (especially most model merger are disappeared after NAI v3, or the finetune hype, or realized that <a target="_blank" rel="ugc" href="https://civarchive.com/articles/3409/diagram-of-unet-and-exotic-merging-methods">MBW theory is actually invalid</a> <s>it works because of introducing parameters with optimization loop only, which is not artistic at all</s>).For example, from the released recipe of <a target="_blank" rel="ugc" href="https://civarchive.com/models/9409/or-anything-xl">AnythingXL</a>, I can interprept it as "average of 7 models, with 14.2857% of each model, meanwhile taking the favourite model as double i.e. 28.5714%". Meanwhile, <a target="_blank" rel="ugc" href="https://civarchive.com/models/391659?modelVersionId=436920">PonyMagine</a> is successfully apply DARE on top of a custom recipe.<h2 id="methodology">Methodology</h2>融模可以不玄, 只是門檻有點高 (魔改算式) 罷了.I will make a seperated discussion here, or an article in this platform, or see <a target="_blank" rel="ugc" href="https://github.com/6DammK9/nai-anime-pure-negative-prompt/blob/main/ch01/ties.md#spinoff-ties-soup">my article in Github</a> (<a target="_blank" rel="ugc" href="https://github.com/6DammK9/nai-anime-pure-negative-prompt/blob/main/ch01/dare.md#spinoff-drop-without-rescale">this also</a>), or <a target="_blank" rel="ugc" href="https://civarchive.com/articles/3409">seperated article in CivitAI</a> if it is not written yet.Since model merging in SD at this level is lack of discussion, I have nothing to reference but carefully make justification and make insight. From the study with "related works" (I read for multiple times because it is close to ML / Math discussion), I expect that algorithm modification will be essential.Therefore I first get the original implementation done (took months), and finally perform analysis on the inherited mathmatical properties. Soon I found that <a target="_blank" rel="ugc" href="https://arxiv.org/abs/2212.04089">task vector</a> should be <a target="_blank" rel="ugc" href="https://arxiv.org/abs/2311.03099">normalized</a> (subset of rescale), meanwhile <a target="_blank" rel="ugc" href="https://arxiv.org/abs/2306.01708">sign election</a> should be based form identity instad of signed movement, because <a target="_blank" rel="ugc" href="https://stats.stackexchange.com/questions/23789/is-ar1-a-markov-process">MDP under SD is suffering gradient problems like RNN does</a>.Meanwhile, I don't have resources to either train model (hardware / time / human resources, or interest), or even conduct a <a target="_blank" rel="ugc" href="https://arxiv.org/abs/2309.14859">throughful evaluation</a> (like <a target="_blank" rel="ugc" href="https://civarchive.com/articles/2468/lycoris-guidelines">team lycoris</a> and <a target="_blank" rel="ugc" href="https://civarchive.com/articles/2479/2023-10-7-survey-of-v14-training-automation-and-planning-of-version-v15">deepghs</a>) of a model. All I can do is conduct a subjective <a target="_blank" rel="ugc" href="https://practicalpie.com/house-person-tree-test/">HTP Test</a> towards the model, assuming it projects its natural behaviour. Therefore you will see "a pink hair boy interacting with a car, with random but filled backgound".<h2 id="experiements">Experiements</h2><a target="_blank" rel="ugc" href="https://github.com/6DammK9/nai-anime-pure-negative-prompt/blob/main/ch05/recipes/x191a-AstolfoMix-24083001-3360d18.mecha">(215a) Discovered models, filtering process, the merging script (End-to-End with single click!) and the 387-line recipe are generated.</a><a target="_blank" rel="ugc" href="https://github.com/6DammK9/nai-anime-pure-negative-prompt/blob/main/ch05/amplify_diff.ipynb">(215b) Extended use of the merger.</a><a target="_blank" rel="ugc" href="https://github.com/ogkalu2/Merge-Stable-Diffusion-models-without-distortion">(215c) The dedicated merger which was left uncompleted.</a><h3 id="parameter-searching">Parameter Searching</h3>Parameters for merging algorithms are found by randomly selecting 10% models out of the model pools (e.g. 20 out of 192) for viewing the effect. It requires much 20x less time to merge, and still able to represent the estimated final result.<h3 id="prompts">Prompts</h3><a target="_blank" rel="ugc" href="https://civarchive.com/articles/5149/untitled-denoising-to-the-random-content">Even no prompts works</a>. 質量詞真的需要嗎?I have tested it will long prompts, it works fine. In contrast, you will see most of my posted images are just a few words, and no negative prompt (because I seldom need to exclude things). However, when I add quality tags, it may produce worse or even broken images, becuase the recipe models are fighting each other with contradicted knowledge.<h3 id="cfg-steps-additives">CFG / STEPS / Additives</h3>It is as wide as the <a target="_blank" rel="ugc" href="https://civarchive.com/models/155604/astolfomix">SD1 version</a>. Currently I find that "CFG 3.0 + PAG 1.0 + mimic 1.0 phi 0.3 + FreeU Default" will be good. "48 steps of Euler" is enough for generation, however I still favor 256 STEPS + 64 Highres.<h2 id="discussion">Discussion</h2>Even SOTA merging algorithms cannot learn all concepts from all models, it shouldn't be a replacement of Trained Models / LoRAs, more like a base model for further developement. 更好的底模都因為社群的誤解, 或者短視, 被忽略了. 從 SD1 / NAIv1走來, 大家學會了甚麼?It is a pity, or "final nail to the coffin", that Pony is accepted just because of its NSFW ability, overriding any technical consideration, making resources not sustainable.I am aware of low attention throughout the journey (that is huge improvement since Baseline Model, the image quality is increased, with less "halo effect"), but I must get it done to leave a mark in the (art) history. I know no one will be interested on developing open source model, because the <a target="_blank" rel="ugc" href="https://www.theblockbeats.info/en/news/53623">incentive</a> is way too low, no matter material or just spiritual support. There is no more <a target="_blank" rel="ugc" href="https://huggingface.co/cagliostrolab/animagine-xl-3.0">animagine</a>, meanwhile some <a target="_blank" rel="ugc" href="https://civarchive.com/articles/4733/yuno779-the-response-to-yuno779-and-byebye">famous modellers</a>, or <a target="_blank" rel="ugc" href="https://github.com/6DammK9/nai-anime-pure-negative-prompt/blob/main/ch02/any4_qqdoc.md">less famous</a>, are gone.I expect SD community should, or will be forced, to consider merging thousands of LoRAs back to the base model, and keep the "artistic movement" rolling. 將來應該會有人需要大批融 LoRA, 甚至底模, 這總需要有一點辦法吧.<h3 id="pseudorandom-in-extreme-condition">Pseudorandom in extreme condition</h3>In 255c, it looks like the "noise across model weights" is coming back because of extreme normalization across model weights. It resembles <a target="_blank" rel="ugc" href="https://en.wikipedia.org/wiki/Vanishing_gradient_problem">vanishing gradient problem</a> in the sense of training trajectory. It marks the end of this series, and I should train the model properly... <a target="_blank" rel="ugc" href="https://github.com/6DammK9/nai-anime-pure-negative-prompt/blob/main/ch06/readme.md">and I did</a>. 終於融模融到撞牆了.<h2 id="conclusion">Conclusion</h2>The new merger gives me ability to carry on the research on fancy merging algorithms on large number of models, and keeping the model structure same and convenient. I may update this article when I successfully produce and test models based from different merging algorithms.<h2 id="appendix">Appendix</h2>See Experiment session for recipes.<ul><li>My <a target="_blank" rel="ugc" href="https://github.com/6DammK9/nai-anime-pure-negative-prompt/blob/main/ch04/ice_lake_ws.md">Workstation</a> for this mix (merge time is 36.2 hours, peak usage of RAM for DGMLA-216 is <s>1.446</s>3.500TB, scale with total of model counts).</li></ul>License: <a target="_blank" rel="ugc" href="https://freedevproject.org/faipl-1.0-sd/">Fair AI Public License 1.0-SD</a> For more details, please see the license section of <a target="_blank" rel="ugc" href="https://civarchive.com/models/260267">ANIMAGINE XL 3.0</a> / <a target="_blank" rel="ugc" href="https://civarchive.com/models/257749?modelVersionId=290640">Pony Diffusion V6 XL </a>/ <a target="_blank" rel="ugc" href="https://civarchive.com/models/833294/noobai-xl-nai-xl">NoobAI-XL</a><ul><li>Hint: It is impossible to proof that you have merged / finetuned my model even it is (1-<a target="_blank" rel="ugc" href="https://en.wikipedia.org/wiki/Planck_constant">h</a>) similar. Remember use <a target="_blank" rel="ugc" href="https://github.com/silveroxides/stable-diffusion-webui-model-toolkit-revisited">toolkit</a> to fix VAE<s> (metadata is not preserved)</s></li><li>Hint: My scripts are MIT. <s>I'm happy to sell you my workstation.</s></li></ul>

AstolfoMix-XL

Uniform merge of 32 UNETS + (19+26) CLIPS

astolfomixXL_baseline.safetensors

Baseline

256cR-Evo

256c-Evo

256c

DGMIZ-AK7c

215cR-Evo

255c

215c

DGMLA-216

TGMD-192

TGMD

TIES-SOUP

Extended-FP64

Files with SHA256: 82f53a8fe100b87179c78e6a3f74557e19f3e61d1fd5b3e1ce5e8893a687334d