A quick merge of the Indecent and Love of the Earth models which, when combined with a few embeddings, help to create surprisingly realistic one-shot animatediff results.
This very simple model is exclusively intended for use with the recommended textual embeddings to produce 512 (that's where Hotshot gives the most motion) pixel images, one of which is then directly turned into a video using HotshotXL and finally sampler-upscaled to 1k. This T2I2V process helps maintain Hotshot consistency (to some degree) over multiple contexts, at the expense of coherent motion. For the exact workflow, download the Training Data and load the Json in ComfyUI. It also contains the embeddings you'll need, which you'll have to place in ComfyUI/Models/Embeddings.
You can download and drop the movies into ComfyUI for that specific workflow. The workflows should be identical in terms of nodes, but may have some different values for noise, samplers, LORAs, embeddings, etc. They should work fine with most other pony checkpoints as well, realistic or otherwise, though parameters, embeddings and noise values may need tuning.
You'll need ComfyUI with HotshotXL, some utility models(VFI) and a GPU with a minimum of 10GB of VRAM to produce 8-64+ frames(1-8 seconds) of 1K video in roughly 2-8 minutes.
The temporal coherency of the movies is so-so, sadly. It's somewhat a fight between motion and coherency, and you'll need luck to get both at the same time. It's often better to switch seeds and try a different scene than to try and tweak values to get a good result.
Find HotshotXL here:
https://huggingface.co/hotshotco/Hotshot-XL/tree/main
Download the hsxl_temporal_layers.f16.safetensors file and place it in your ComfyUI/Models/animatediff_models folder.
Description
The TrainingData contains an example workflow (as do the preview movies) + the required embeddings (these go into ComfyUI/Models/Embeddings/)
FAQ
Comments (5)
You have to provide some getting started comfyui workflows or images with embedded workflow. Otherwise is just very abstract the use of your model.
I've uploaded a workflow and the embeddings required as Training Data in the files section. In addition, the individual preview movies should each contain a version of the workflow, and these can be dragged and dropped into ComfyUI.
@Tragdar Drag-and-drop thing does not seem to work.
Nice check point with no need to do any animated scene
"Unable to find workflow in..." is what ComfyUI sais when fed your videos. Could you please publish your workflows separately?