Some time ago I wrote an article about the discovery; but didn't have the knowledge nor understanding of the mechanisms to explain WHY it worked. I could only say YEAH THIS THING WORKS. Now I have a fair understanding of the internal mechanisms, so I can give a metaphorical and pragmatic analysis.

Boiled completely down; it comes down to the T5 having general knowledge routes along with direct knowledge routes. I've seen it directly with sd35 and indirectly with flux beforehand.

I knew what to avoid, what to do, and why to avoid things or do things, all because of my assumption that plain English had an intelligent response from the T5. I just wasn't sure what sort of response until I dove really far into the depths of these things.

The general training parameter templates used to condition them were some evidences originally but I wasn't sure how they were used. Instruct parameter sets and instruct schema is another methodology that is akin to what I'm describing here, and it's quite an interesting set of mixes and findings.

Metaphorical understanding.

I do apologize to the more technical here - as this is a terrible breakdown, but it's a very rudimentary assessment for people. Keep in mind, this isn't fully accurate.

Alright simply put; when Flux takes an image in while in update gradient mode, it sees that whole image for a brief moment before it diffuses it for learning. Depending how you timestep it from 0 or higher, to 1000 or lower; it'll retain the information based on those valuations; you can be saving more or less water in the pond. Sometimes you're introducing oil, sometimes you're introducing a method to rain that water.

First it takes your prompt right; and then it converts it to a bunch of numbers that it uses as a kind of accessor map. It uses this map to define subtext unto itself and attach to certain things within itself. Simply put, it's complicated.

The more complicated and more accurate your text_encoder's response to the tags and the bigger your model's unet is and how much information the model has been taught; the more you can jam in the square hole, like inertia.

There are a series of complex formulas and sub-systems built entirely on translating that image input into something useful neurally that can be directly inversely recalled in different ways later on; this is DIFFUSION in a nutshell. Learning a bunch of things one way, and then recalling different things and those different things overlapping as they are recalled.

Metaphorically; Enough water gathers on a surface and it glops into a larger water droplet, until eventually you form a pool, and someday maybe even an ocean. So you tossing a few images in Flux at low strength is like shooting a BB gun at a freight train. It won't matter. However, you shoot a few hundred thousand at the thing, and you're going to make some dents. Do this a few hundred times and the Flux train will likely be derailed.

In any case, the way it learns is by trickling correct or incorrect based on the difference of what it guessed and what it needs to know; which is a complicated math formula called loss.

Trigger Directives and why they work

Alright, well with Flux 1D and the like, the UNET and T5 subsystems are... big. Very big. Training the entirety of the internals of Flux involves many variations of words in a large amount of ways.

As you trickle image learning data down the systems, you find that there are ponds. Many ponds of smaller information. Many of the ponds have built in safeguards to prevent them from draining too much, causing cascade floods, and many simply drain to nothing with almost no effort.

Well, a directive, is akin to attaching a funnel to a specific set of ponds. The original LORA "trigger word" spigots were very direct, akin to literally running a hose to a specific area within the ecosystem to begin from, then flooding it outward naturally from there. It's not exact, but similar to the Flux variation; as the text_encoder will respond similarly, but in a different kind of way.

In Flux it's similar, but not exactly the same. The T5 is a regulator and a powerful condenser, as it's complexity is quite high, so you can attach to MANY ponds carefully and with many options with the right directives.

A directive like; "Focus entirely on the taco," will cause the T5's text encoder to fixate on a multitude of potentials within itself automatically - COMPLETELY TACO FIXATED. It's akin to asking a question in a lot of ways, a question that the T5 can answer if asked, and the UNET has training with focus, tacos, and whatnot.

So, those tokens line up in interesting ways, and unet slowly learns how to focus on any taco in any image that you're recording loss from - whether tacos exist or not.

Your training will likely result IN TACOS showing up everywhere, from paintings, to walls, to chickens with taco shoes, and so on. Even if you're just teaching it about swimming pools, or about a single chair type, or about something entirely unrelated.

It doesn't matter, because by telling the T5 to expect to focus on tacos, your training will automatically reflect that outcome based on what was already learned.

In this instance, a directive would be like telling your bb firing machine where to put the BBs to do the most damage to the Flux freight train. The T5 and CLIP_L being internally a pair of hot messes will automatically start guiding the prompt in the absolute direction whether it be directly or indirectly towards those goals.

By feeding it those words, it begins to expect those words; as the T5's entire purpose and training process was REMOVING words from segments of text and then it either correctly or incorrectly filling in those responses based on those changes.

If the T5 does not know those words, it cannot produce the correct summarizations.

Trigger Directives and why they won't always work

The directive depends on whether or not the T5 can rationalize something into something useful, and whether or not the UNET can contextualize that useful behavior or not.

So, in the end it comes down to pure utilitarian combination. If the T5 knows it, the UNET may not know it. If the UNET knows it, the T5 may not know it.

This is generally why you don't train the two of them together after a large scale fitting on something like SD35 or Flux, at least that was my assumption originally.

The reality I think we have here is a simple one; the fragile ceiling.

What once was done, was done for X reason but without stipulations for Y solution.

The Y solution in this case is high-grade finetuning and refitting of the pair of models together. In reality, the X reason does not conform to the need; and instead we have the conundrum of risk that comes with learning and adapting something more complicated.

1. It CANNOT work, if the T5 does not KNOW the information, but the UNET only knows direct information with little linkages. If you teach it this way, you are teaching the T5 in unknown ways.

2. It CAN work, if the T5 knows the information and the UNET does NOT know the information. This is the foundation of diffusion, as you're teaching the diffusion model to behave.

3. It CAN work, if the T5 does not know the information and the UNET knows some associative information. Directives will be hit or miss, but it can pragmatically link the various tokens in the UNET over a long period of time.

For some reason, the entire community at large defaulted to 3.

"It CAN work, if the T5 does not know the information and the UNET knows some or associative information."

Which to me is blatant disregard of all the SD15 and SDXL CLIP training done beforehand. I dug into this for a long time until I realized that this WAS NOT A SOLUTION, it was gatekeeping.

This is not a compromise. This is akin to yielding the power of these models to whatever the hell is listed in the text_encoder, without ever updating the text_encoder.

These TEXT ENCODERS are NOT accurate enough to give that power to. They MUST be finetuned to fit trains, whether before or after the UNET is fitted, but they MUST be finetuned.

ESPECIALLY THE T5. The T5 is VERY dispersed due to the large amount of tokens. This, when paired with LARGE unets causes a huge cascading series of problems when attaching more information.

If you don't train the T5, you're potentially WASTING large amounts of compute time and power to fit a unet that may or may not even fit, due entirely to the complexity within the models themselves being so much higher realistically than can form.

What I think really happened with the T5 learning and why it was arbitrarily gated.

There was a heavy push to PREVENT people from fitting the UNET in ways that could be utilized in negative or society-wide detrimental ways.

Sentencepiece is was trained without "naughty" words, and many other elements required for an unchained dataset to exist and be trained into a diffusion model. Not to mention the actual T5 being forcefed information and automatically tokenized in ways that made it very difficult to LEARN those subsequently difficult to access pieces of information.

There are many obvious reasons for this, but the solution spawned people like me, so it turned into a very nearsighted solution to prevent something that may or may not have happened; and instead turned it into something potentially worse.

Others figured out ways anyway, without showing their papers, their work, or their findings. They simply did trainings that worked because these machines are too complicated to simply MAP the outcome for.

Directives; are how Flux learned.

Everything associative was automatically learned and associated in unison together. In a cascading pool of liquids that fell from layer to layer until it ended up learning everything associatively required to make images. Likely after a massive bulk and costly undertaking, as described in another one of my articles. The cascading learning eventually manifested routes of generalization that fit similar pathways to the T5; like a cascading symphony of harmony that turned text to images in ways that never could be done before.

One directive at a time it built them; until it understood every little piece over time. Likely using generalized templates and instruct-style (not the same by a long shot) templates to assign captions and tokens and tags in ways that taught the engine directly how to behave with those images.

I could be entirely wrong though, and they could have just shoved it all the square hole, and this is the result.