Why this topic?

This started, simply because ComfyUI wouldn't detect my variants of Lycoris using the weight adapter system that BlueLeaf installed - which is a fairly good approximation of the solution.

It no work.

It works, and then... it doesn't. So, the BlueLeaf variation is quite good... at approximating the LOHA response, while... only spotting models by layers.

Each model variation - like say SDXL and SD15 have different variations of sub-layer names, and those sub-layer names all have consequences to the formulas.

It works...-ish, because ComfyUI uses additive inclusion, and the weight_adapter does a FAIR approximation of those layers - IF IT SPOTS THEM.

So it... does work a little?

Yes, and no. The formulas for many of these models, REQUIRE a form of pre-calculation as inverse to the sd-scripts structure. Which... does work, when applied to the sd-scripts variations.

SimpleTuner however, is based almost purely on the formulas. Meaning, I cannot really improve those variations, nor can I distill them in a more efficient manner... I thought, until I plugged the formulas into LOHA and LOCON for a test.

It also means they are more robust in many ways, while simultaneously more fragile in other ways.

So what are you saying?

Modification of SOME sd-scripts LYCORIS models, DOES include the correct response behavior from the sd-scripts trained variations.

I'm uncertain why, but it seems to be a consistent-inconsistent version mismatch paradox - where some sd-scripts trained models follow certain LyCoRis paradigms specifically correctly, and yet some like say full vs attention only, produce very very different formulas.

Can't you just invert the formulas?

Yes, in theory. I've managed to invert LOHA and LoCoN. Both of which are more similar to standard LORA and the approximations generated ARE fairly similar in this implementation, however due to pre-calculation these operate at a much faster rate, and can be distinctly modified at runtime.

You said... LOKR TOO

Y... yeah... Not all models have... easily discernable formulas. The models WERE trained to be applied multiplicatively, based on a certain paradigm - but the paradigm has been... updated a few times over the years.

The older paradigm is more stable with response and uniformity, while the fidelity of the model falls into a different categorical basis.

Newer variations have more... diverse inclusions, such as text_embedding additions, layered interpolation, gradient accumulation preservation, and so on. There are MANY subtle things that torch has done over the past couple years for optimization, and many of which went under the radar for DIRECT gradient accumulation calculation.

The newer LOKR from SimpleTuner are more inline with the actual LOKR ratios, while the earlier variations from sd-scripts and other tuners are actually... an approximation. They are KIND OF like the correct formula, but the fidelity loss I believe is heavily due to the models only being approximated and saved in a certain way that doesn't conform to the product valuation correctly... yet.

I believe there are some edge-case rounding issues that are being broken by this formula and that is likely one of the prime culprits to these failing. The resizing and structural problems really speak not to LOSS of fidelity but increasing detail, and in that light the increased fidelity turns into burned edging, incorrect scattered pixelated learning, and a multitude of other variances that do not conform to the goal.

This is partly due to the trainer not approximating the math correctly through rounded torch values with float16 and part due to the loaders not loading the valuated formulas correctly.

What is the solution?

Model graphing. I've determined the most deterministic and careful approach, is a full reinversion of the model graph and an approximation of the exact structure in the correct order, which isn't inherently an easy to employ process at times.

Now... I say sometimes, because it really isn't very hard at it's core. You can easily just make a model builder, or import the lycoris model structure INTO comfyui and have it autobuild the models. That's not the core problem right.

It's the translation matrix TO the way the ComfyUI pipeline HANDLES those weights being moved from point A to B that causes the sincerest problems.

What are you even talking about?

Progress my friends. Scientific progress.

I've determined that not only do we have various different versions of these sorts of models floating around, but I've figured out a way to identify them and coordinate their differences into a an equation that I call the "precalculated_distillation" process.

This is much easier in diffusers, since I have full control of the entire pipeline - but we don't have that convenience here, not if we want to have access to all the conveniences and codebases attached to comfyui.

Interpolative Distillation

An interpolative distillation isn't the same as interpolating a model from A to B, or interpolating the neurons, or interpolating anything in a specific format.

In this case, we are interpolating a LOKR directly into the intended parent model, rather than causing the approximated form to exist AS AN APPROXIMATION.

This is not fast.

The formulas involved are complex. However, the baseline outcomes are showing promise. The generated and pre-distilled layer interpolations are more robust to request so far, and have shown more careful attention to detail around edges with less tearing and edge failing.

BeatriXL began through interpolation

She was quite the model and she wasn't good enough. The gaps and non-cemented learning showed that she lacked the fundamental understanding of the topics to reaffirm or generate upon them.

The more finetuning that was added, the more edge-detection failed. Everything slowly reverted to flat over time, rather than generationally improving the fidelity... Which should have happened.

Beatrixl was constructed through nearly 50 LyCoRis before interpolating the outcomes using an interpolation teacher

It was built from 5 models interpolated from a teacher model. Each subsequent interpolation has failed, only the initial Omega V0001 with the correctly selected settings based on the feature extractions worked successfully.

Omega was fragile though. I could not figure out why until I started tracing back the exacting route through what made those core models behave and what made them not behave.

A failed beauty - BeatriXL.

She was learning, until she was not. It became clear that each subsequent generation of merged lycoris was causing damage, but the damage did not manifest until the later outcomes started to show cracks and edges.

Eventually, the manifested merges broke down her fidelity centers, and everything became more akin to flat and pixelated edges on realism, blurry depth, incorrect associations, and sometimes flat ignoring prompting.

What I learned from this in terms of Lycoris

Lycoris is flawed, but not at the implementation or training... I mean it might have some cracks, but I sincerely doubt the core structure of Lycoris is to blame here. I blame myself for blindly trusting the merge tools, and then continually training iterations based on those merge tools using tools that I wasn't in full control of.

I was working full time at the time and I really wanted to get the automated structure operational with minimal effort, and that makes me to blame for it.

Lycoris wasn't to blame - I am to blame. Not because of my training process, nor was it the lycoris, nor was it ComfyUI; it was the concatenation and merging process employed by ComfyUI that I simply did not scrutinize enough. The error happened in a deep-rooted core issue, and it was ComfyUI's methodology in merging those weights into the core of a model, which is considered common practice when merging.

The conclusion

It must be redone. All of it. Top to bottom, the entire merge system must be re-envisioned. The entire interpolation system must be reforged. The models must load correctly based on the formulas. The models must inference correctly based on the formulas. The models must have a solid form of masked learning and a solid form of attention masking on every level required to teach them.

Then and only then, can I cook.