The T5xxl's true unchained and how I plan to train it.

To include the micro resnet layers and a classifier pool to capture the token entropy;

We're looking at 12.4 days according to GPT which really isn't bad on a budget for an epoch with a tb of text using this process.

I'll get started next weekend on the prototype.

https://arxiv.org/pdf/1910.10683v4

So here's the T5xxl google paper. Simply put, it's full of how they made the thing from basically start to finish. Datasets, implementations, training parameters, methodologies, and strategies employed to make the thing behave.

They had a series of datacenters, a series of ai experts, and a series of complex learnings and teachings employed by in-house technicians, developers, and you name it. It's google.

We got, little ole me. So, I'll just be all of that in one.

The paper makes it doable.

The T5 has a blueprint here. The entire series of layers from top to bottom, left to right, accuracy tested against their canary prompts and the returned output from the testing.

We have the information, and it shows the T5 is actually kind of stupid. It doesn't understand even the basic concepts it was taught, but instead is trickle burning out new concepts as things progress along the chain.

This tells me a series of obvious factoids about this summarizer; and the factoids are basically... it's substandard for the current day.

This model pales in comparison to differently trained and differently fitted models of similar categories and calibers... but damn does it summarize. One would argue, it's considerably less accurate than it's supposed to be. You can see it, when you attempt to summarize directly with it. It will fail, often.

So that begs the question; how do we make it BETTER than it used to be, and is it worth it?

I think, it'll be a combination of text encoder embeddings, custom tokens with special parameters, and a big shift of the unchained vocab into a direction that better conforms to the necessary data; without altering the tokenization in a biased direction that impacts the T5 in heavily negative ways like what happened with my SD35 finetune and the subsequent insanity that it can produce on other models such as Flux.

It became - too different. Causing cascade overlapping vectors, leading to area within flux that simply do not have the necessary learning, but are mathematically accurate to the T5's learning. Which begs the question; where exactly is that information going?

Incorrect places.

Those vectors were too hard set. The encoders and the trajectory results in a series of internal offsets; upsetting the entire internal balance of the T5xxl, leading to a cascade series of incorrect placements within Flux and the like. As they assume certain details and the T5xxl is now summarizing in a different way, the entire system is now - Shifted. Not in a seeable manner either, we're talking in a trajectory beyond rationality - only I can offset correctly with the correct encoder inversions... because I have the full images and captions still.

However, that's not what I'm talking about. I'm talking about fixing the T5xxl-unchained, and adding specific informations in specific patterns to better condense and follow the necessary routes trained within it - thus preserving a large percentage of what was trained in other machines, while simultaneously allowing for internal self attention and masking via that attention to be a very reasonable and even required process in preserving the T5xxl.

Our required starting data

Simply put; we need the C4-noblock dataset to get kicked off. Just like the original learned from the C4, we can mask what it already knows from the C4 and include the non-filtered variant for a powerful baseline infusion of additional information and summarization power before we start fitting with more specific information like I kinda just shoved in earlier.

https://github.com/allenai/allennlp/discussions/5056?utm_source=chatgpt.com

This thing isn't exactly the most useful set of information here, but the information DOES EXIST. So I need to track down the 1tb of plain English, and then devise a methodology to train the difference using masking in an efficient manner.

I have multiple methodologies for this process; many of which included in multiple papers since the T5xxl was introduced.

My method, and the method I'm fairly certain will work; is going to be masking training using a combination dreambooth-style and tokenization weights.

Okay, so simply put, I don't have a datacenter. However, I know how to mitigate this issue, by using simplistic means. This isn't actually that tough for me as I've always managed to make due in the past.

So here's the full breakdown on how this will work.

Cutting our data.

We cannot train this entire series of data for a multitude of epochs. It would be expensive and require much hardware. Likely A100s or something that can process in a high fp route.

However, we can REDUCE this data almost immediately using mechanisms in the encoder directly.

There are a series of self attentions, tokenizations, and encoder ratings we can use to simply omit data that meets those highly trained tokens - aka the tokens already learned in the T5xxl originally - and then only train those necessary tokens that are NEW or untrained - then concatenate old informations with new informations as the trains progress.

Like, blending the differences of old with new - unlocking the potential of those new tokens by connecting places within the old.

As the train progresses, I can reduce this mask learn rate and strength, causing the internal structure to slowly conform in a more meaningful manner, and given enough time removing the masking training wheels entirely to allow full tokens to be omitted.

Thus, we will only need to train a percentage of our data, and we only need to train that data a specific amount of iterations, and with those iterations we only need to train those iterations to encompass those necessary masked tokens - completely bypassing the tokens that aren't attention sought based on our layering parameters and rules.

In a sense, we will let the T5 decide what it wants to do with that data, and then in time it will produce the necessary results dynamically with our new tokenization and new process akin to a full and true unchained process.

The LOHA and the purpose of the LOHA

Following suit to our attention masking goal, we can use many adapted methods from dreambooth and diffusion; to trickle learn our attention into the process. Not in the same way as a fully masked and auto-completed token would be taught, but in a way of reducing the strength of that learned token as it's processed; akin to diffusing it using a different form of layers alongside the originals like a kind of... overlapping and intrusive subsystem.

The more closer it can identify those routes using my interpolation process and a bit of traditional math, the closer we can get to fully unmasked training the model into our LOHA and LYCORIS.

I know this might sound odd, as it's a bit new in this realm; but the odds of dropout of tokenization with the T5 in this case would be like teaching diffusion to the the T5 directly. It's a very odd cross-entropic principle that can be summarized based on a series of principles founded in AI throughout the years; primarily teaching a model to learn in this case.

Teaching the T5 to understand it's own former learning, along with teaching the T5 to understand the new learning.

You can see a similar process was performed on the SimV4 epsilon training; where over time the planned and trickle learning met expectations and the timesteps recalculated to fit the intended goals until the timestep formula fit the needed outcome.

This isn't sorcery

This is math. Plain and simple. Teaching the T5 how to learn, by using a LOHA that is built to teach the T5 how to learn, while the t5 is learning. This will not need images, it will need text.

This is pure unadulterated entropic science and will fit perfectly in the square hole.