Claymation Transformation - MiniMax H3 / Wan2.2-2.1 14b I2V

Where objects melt, stretch, and squish into something entirely new!

This Minimax H3 Reference2Video LoRA aims to replicate the surreal, handcrafted look of plasticine/claymation animations, where objects morph into others with fluidity. Heavily inspired by the classic cartoon "Last Year's Snow was Falling" (LYS). (although the likeliness is obviously not 100% because of MH3's training rigidness)

Trained for 159 epochs using a musubi-tuner fork by AkaneTendo. Special thanks for the implementation of the stop-flag file as it allowed me to save the model at the minimum loss of a very unstable train. Training techniques: ostris de-distill adapter, CFG loss, base preservation loss.

MiniMax H3 Prompting Info

Example prompt:

LYS-style transformation video. <Subject 1> realistic cup of coffee is shown on a table. Suddenly, the realistic cup of coffee plasticine-transforms into a realistic parrot.

The two necessary trigger words are: "LYS-style", "plasticine-transforms"/"plasticine-melts".

This model is trained in reference2video-audio_omni mode and is very versatile! You can throw many reference images, or 1 or 0 for text2video.

Workflow in JSON form: https://huggingface.co/kabachuha/mh3-r2va-claymation-transformation/blob/main/mh3-r2va-lys-workflow.json.

Wan2.2 I2V Prompting Info

While this doesn't generate videos from scratch, it leverages the power of Wan.2.2 I2V 14b to create really stunning transformations. Just input the first or the last frame.

If you have no idea where to source them, you can use the version for Wan2.1 text2video (read below) in the "text2image" mode when you set the input latent frame count to 1 and then save this 1 resulting frame as a picture.

The trigger words are 'Claymation transformation' and 'plasticine-transforms'. The best prompts are:

Claymation transformation. The video begins with an [object] standing before a [describe background]. [optional object description]. Then the [object] plasticine-transforms into an [another object, object description].

Simplified:

Claymation transformation. The video begins with an [object]. Then the [object] plasticine-transforms into [another object].

Bonus. You can specify multiple transformations in the same video!

Claymation transformation. The video begins with an [subject] sitting on a table. Then the [subject] [does triggering action]. The [subject] squishes, shrinks and the [subject] then plasticine-transforms into an [object_1], which then plasticine-melts into a [object_2].

This LoRA should be quite beginner-friendly.

Wan2.1 T2V Info

Works great both standalone and with VACE for first-last-frame selection and image to video!

The examples illustrate each mode: 3 x pure text2video, 2 x image2video (set start, set end), 1 x flf2video (interpolation).

The generation resolution is advised to be 480p or close (e.g. 512x512). The trained durations are from 1 to 5 seconds, with the most tested one of 3 seconds. (48 frames)

The t2v training was made using diffusion-pipe for 135 epochs and flow shift of 4.0 on the dataset of metamorphoses from the legendary plasticine animation "Last Year's Snow Was Falling".

Due to the specifics of the train material, you may sometimes encounter a hut-like interior with four-sectional windows in the default mode, so it's encouraged to write the background appearance in the text2video mode manually. Though less often, some parts of the main character's appearance can happen (like the villager hat), in this case add 'lys-man' to the negative prompt.

For text2video I tested with cfg 6.0, shift 5.0. For image2video/flf2video recommended to use with the kijai VACE workflows, standard 1.0 lora weight, 6.0 cfg, 16.0 shift + cfg_zero_star. (see videos meta)