How powerful is this model REALLY!?

Can it handle one man?

Can it handle the front line?

Can it handle the brass, the strings, the entire chorus of over a hundred?

How about the audience of thousands?

How about the whole damn thing? Can we feed it a thousand faces and it respond with a thousand accurate responses?

When I TRULY approached Flux, I approached it with a singular mindset;

I know nothing.

I thought I knew some things. I have been making games since before I was 10. I was developing PC games before I was 15. I was creating software before I was 20. At near 40 I expected to know a thing or two. I DID NOT. I knew NOTHING, and I had some painful experiments that proved I knew nothing, and hello my old friend Dunning Kruger I haven't seen you in a while. Thanks universe for the kick in the ass I needed.

I'm returning to this mindset to skip the learning pain.

Like a child, I must understand what heat is, what dangers there are around me, the sounds, the smells, the colors.

This machine. I must understand what makes it tick to improve it.

The Great Orchestra Experiment

As a researcher, this is based purely on the drive for empirical knowledge. I must, understand this machine.

As a pragmatic engineer I prod and jab at machines for a living, and I must know what makes this machine tick.

As a human, I want to understand what makes these machines similar, and what makes them different to us.

I will be running a series of experiments on no less than 50 sliced pieces of this video from this youtube clip here; to determine the flexibility and responsiveness of the model trained with many forms of captioning into the same lora.

I stand to gain no monetary award, profit nor valuation in any terms. I have the utmost respect for everyone who went into making this video, and if you wish to have your face, likeness, or appearance omitted from it; I have software that can do this exact thing.

1. Vanilla no captioning;

   • entire series of slices no captions.

2. Singular directive

   • shared directive:

     • the orchestra

3. Multi-directive

   • each slice is given it's own defining points, the actual names of the people in the orchestra that I can find, the conductor, every slice given it's own directive and the directives listed for testing.

4. Hand captioning each slice.

   • Each slice will be manually hand captioned, which will take a careful amount of planning to determine the actions in the scenes, the scenes themselves, and the importance of the actions WITHIN the larger scale of the scene.

     • Setting

     • Action

     • Scene

5. Auto-captioning using CLIP_L and T5

   • A little bit of unsupervised learning never hurt an experiment.

     • T5 identifies, CLIP_L verifies.

     • I'll use T5xxl for this one, since it's a lot smarter than my finetuned T5 small.

6. Auto-captioning using CLIP_L and LLAVA LLAMA

   • Similar to T5 we'll use the EXACT models paired with HunYuan

     • LLAVA identifies, CLIP_L verifies.

     • LLAVA is considered to be the "unabridged" text dataset, which it kind of is, but it also suffers a great deal when you don't cook a model long enough; so lets see if this one is cooked to perfection, or if it has burned ends.

7. Auto-captioning using auto-caption software for full bbox detection and positioning.

   1. Full kitchen sink bbox tagging

      • I'll run the full kitchen sink software suite on the clips.

      • Slice up the videos into segments, tag and identify bounding box locations

        • grid, zone, depth, offset, size, rotation, scale

      • Each slice will have it's individual frames each tagged using the identification software, and then the tags that have the most presence in the software will be the ones used in the final tag file.

   2. Full caption + bbox tagging

      • Similar to kitchen sink, but with caption dropout so individual tags are dropped along the route of training

   3. Split caption and tagging

      • A new type of training I'm working on, where the tagging and captioning is split down the middle and both are trained sequentially.

      • This multiplies the data by a divisor in tag length ratio to accepted attention valuation ratio.

        • T5 Attention Mask is normalized using the hyper fast finetuned T5 small for attention masking during training time. It runs batched multiprocessing and captions thousands of images per minute on a 4090 or hundreds on just 8 threads of cpu.

        • Essentially this will teach the lora only what the T5 deems necessary; overriding LLAVA's responses in a lot of cases.

8. Manual captioning for bboxes using one of the popular bbox software, potentially hundreds or thousands of small captions per clip.

   • Hand cleaning up the bboxes to be semi-perfect to where they should be

   • Filling in missing bboxes that were identified the frame before to correctly interpolate.

The most powerful, most responsive, and most utilitarian based outcome with the most correct accuracy to the caption systems used; will be dubbed the method of captioning HunYuan in my software and the entire baseline set up around these values.

If my software falls short of any of these concepts in terms of flexibility, utility, and reusability; I'll eat my hat.

Overall this should be about 400 or so loras, so it'll take a while;

The first 8 experiment outcomes should be quick, but I don't have the best hardware so we'll see how it goes.

I expect to run most of this on A100s attached to collab, so it shouldn't cost an arm an a leg at least; just uncertain how long it'll take using this process.