Important Notes:
This LoRA is created as part of a fan project for research purposes only and is not intended for commercial use. It is based on the TV series called Arcane which are protected by copyright. Users utilize the model at their own risk. Users are obligated to comply with copyright laws and applicable regulations. The model has been developed for non-commercial purposes, and it is not my intention to infringe on any copyright. I assume no responsibility for any damages or legal consequences arising from the use of the model.
Compatibility:
HunyuanVideo
ID Token / Trigger word(s):
Use these in your prompt helps providing the character. See example prompt above.
csetiarcane, nfjinx, blue hair, black top
Please consider the following:
If it doesn't produce the character, try increasing the lora strength to 1.2
With some seeds it doesn't work well, it simply doesn't produce either the style or the character
I definitely recommend using at least the following trigger words in your prompt: 'csetiarcane, nfjinx, blue hair' .This is probably due to how the dataset was assembled, I'd like to fix this in later versions"
Acknowledgment:
Thanks to the Tencent team for making this great model available
Thanks to tdrussel for the diffusion-pipe that helps us making these LoRAs.
Thanks to Kijai for his great ComfyUI integration
Thanks to POM for providing the computing resources. Without this, these LoRAs could not have been created.
Trainig details:
LR: 2e-5
Optimizer: adamw
steps: 6000
dataset: 40 (33x704x352) videos
rank: 32
batch size: 1
gradient accumulation steps: 4
Description
FAQ
Comments (30)
do you have a workflow where youve adapted LORAs to for the video flow?
I'm not at pc atm, but you can find t2v example workflow in Kijai's wrapper node's example folder. Then you can connect a hunyuan lora node to the lora connector. It will be quite selfexplanatory if you know ComfyUI.
I just saw comfy supports hunyuan video natively now, so you should be able to use the default lora loader with that. I haven't tested this yet though: https://comfyanonymous.github.io/ComfyUI_examples/hunyuan_video/
hmm alright yeah some of those nodes are causing issues, was hoping you were using a different work around, thanks though.
It says "See example prompt above." but there are no example prompt above
download the video ,it's in the file
Can you make more fictional characters? Very good job with this one!
Thanks, I'm glad you like it! You mean more characters from this show or from others?
@Cseti Just other fictional characters in general! Disney characters, cartoon characters and such! Moana, Elsa, Mavis would be really great for example!
@AshleyArnes Yes. I'm currently experimenting with style loras, but I'd like to make more character loras. By the way, the model knows older, famous characters quite well, so they can probably be generated with proper prompting, in which case there might not necessarily be a point in making a lora for them
@Cseti Thats kinda wierd, because I tried prompting disney style and certain disney characters, but it always got me realistic people looking nothing like the original character lol. Do you know how to prompt them? Also looking forward to your loras!
@AshleyArnes I recommend using the Kijai's wrapper nodes, then you can set "prompt_template" option to "disabled" in the HunyuanVideo TextEncode node, and use this prompt template for example:
main_content_theme: A whimsical Disney-inspired scene featuring Belle in an enchanted library, capturing the magical essence of Beauty and the Beast with a blend of classical fairytale elements and warm, inviting atmosphere. The style emphasizes the signature Disney animation aesthetic with flowing movements and rich, painterly details.
object_properties: Belle wears her iconic yellow ball gown with intricate gold embroidery that catches and reflects candlelight. Her brown hair is styled in the classic loose half-updo with soft curls framing her face. Her expressive brown eyes and gentle smile convey intelligence and warmth. She holds a leather-bound book with gilt edges and ornate clasps. Floating around her are magical sparkles and animated books with subtle golden glows.
actions_events_changes: Belle is caught in a moment of wonder as she discovers a new story, with her dress swirling gracefully around her. Several books float and spiral through the air, their pages gently fluttering. Dust motes and magical particles dance in the air, creating patterns of light and movement. Some books gracefully open themselves, their pages turning as if by an invisible hand.
background_atmosphere: The vast castle library features towering oak bookshelves that stretch beyond view, with ornate Gothic architecture details. Warm candlelight from brass chandeliers and wall sconces creates a cozy golden ambiance. Velvet curtains in deep burgundy frame tall arched windows, through which soft afternoon light streams in. The overall color palette combines rich woods, warm golds, and deep jewel tones typical of Disney's classic animation style.
camera_techniques: The scene employs a dynamic low-angle shot that emphasizes the library's grandeur while keeping Belle as the focal point. The composition follows Disney's characteristic sweeping camera movements, with a subtle spiral arrangement leading the eye upward. Atmospheric perspective creates depth in the library's endless rows of books. The lighting emphasizes rim lighting on Belle's figure, creating that characteristic Disney glow around the main character.
@Cseti Wow! That's quite the prompt! Thank you for the detailed explanation, I tried it and it does work like magic! I used different Stable Diffusion versions and I got used to prompting only words and short overall prompts, but I guess Hunyuan needs detailed explanations. Still, I experienced, that combining action loras with prompts it will change back to realistic looks and it will forget the characteristics of the character, so it will be still cool to have loras like your Jinx ones!
@Cseti When you're using this prompt template do you include the object_properties: etc in the actual prompt passed to the model or are they there just to illustrate what you're covering?
@Cseti vi emo gothic will be awesome
By far the best hunyuan video character lora out there, absolutely insane results, thanks so much for adding your training details into the description, can't wait to train some character beginning from next week when i got more RAM too.
Do you remember the amount of repeats and epochs you had? And with "dataset: 40 (33x704x352) videos" do you mean that the resolution was 704x352 with 33 frames? how long were your videos?
The videos were around 2-6 sec long. The script automatically adjust them to the frame bucket that was 33 frames. This is similar with the resolution. The videos were bigger but the script resized them to that resolution when using resolution bucket of 512
Repeats were 10, and the epoch I don't remember but can be calculated by the other values
@Cseti great, thank you so much for the insight! the epoch should‘ve been 15 then.
How do you usually tag the prompts for your hunyuan loras? Do you only enter a few trigger words or do you do full text prompts?
@vslinx I use detailed captions for the dataset with 2-3 trigger words mostly.
@Cseti Awesome! thank you so much for all the info! 🫡♥️
This is the best Hunyuan Lora by far not even close. I don't understand why all the other Loras mess up the motion of videos but this one does not
I would like to ask what video card are you using? I'm using a 4090 on my side, 48 videos at around two seconds each, 480p resolution, practicing at 512×512, and basically it's going close to 70 seconds a STEP, and with repetitions of 1 and bs of 1, it's very slow
I used 4090 for this lora. And yes, it was quite slow. I don't remember the s/it values but it took me 2-3 days constant running if I recall properly
@Cseti Thanks for your reply, looks like I'll just have to wait a bit more too, but I also practiced a test lora, only used 6 videos, each one was also very short, also 480p, about 5s/it, and the 6 epoch fit was pretty good, I feel like I could break up the 40+ videos and practice it over multiple times each time with less than 10 videos, and then next time, continue on the practiced lora, and it might be quicker! The next time you continue on the practiced lora, it may be faster.
Details
Files
Available On (1 platform)
Same model published on other platforms. May have additional downloads or version variants.