Hey there! Happy to share!
Well, during training, it was pretty tight in term of VRAM (31GB/32GB), but I managed to do it with those configurations:

For the toml (no sub lengths, I cured it manually to have only the movements I wanted to train):
---
[general]

resolution = [480, 270]

batch_size = 1

enable_bucket = true

bucket_no_upscale = false

[[datasets]]

video_jsonl_file = "./use-case/config.jsonl"

cache_directory = "./cache/use-case"

frame_extraction = "full"

max_frames = 65

source_fps = 16.0

[[datasets]]
<same for other datasets>

---

Videos were regrouped in 7 sub datasets (max 20 videos per dataset, max 4s each, representing a use case), no repeat, everything was pre-cached on an SSD before training.
It is notable that the pre-caching downscaled a bit to 464x272 (to match a good number for Wan).

Train config:
---
accelerate launch --num_cpu_threads_per_process 1 --mixed_precision bf16 wan_train_network.py \

--task i2v-14B \

--dit ./models/diffusion_models/wan/wan2.1_i2v_480p_14B_bf16.safetensors \

--dataset_config ./deepthroat.toml --sdpa --mixed_precision bf16 --fp8_base --fp8_scaled \

--optimizer_type adamw8bit --learning_rate 2e-4 --gradient_checkpointing \

--max_data_loader_n_workers 1 \

--blocks_to_swap 0 --network_args loraplus_lr_ratio=4 \

--network_module networks.lora_wan --network_dim 24 \

--timestep_sampling shift --discrete_flow_shift 4.0 \

--max_train_epochs 60 --save_every_n_epochs 1 --seed 989 \

--logging_dir ./logs/ \

--output_dir ./deepthroat/ --output_name jfj-deepthroat
---

Over the default musubi-tuner configuration, I removed --persistent_data_loader_workers (free up VRAM for not so much loss in time), used --fp8-base & --fp8_scaled
No block swap needed, so I can achieve ~6.1s/it
Diminishing the --network_dim helped too I think (and it was great for a concept LoRA).

Hope it helps!

@CyberAImania 3600 steps to be precise. The iteration time decreased with time (started around 6.32s/it and got stable at 6.07s/it starting epoch 28).

To be honest, I'm far from being an expert. I have been fiddling around with musubi-tuner for 2 weeks since I received my (fucking pricey) 5090, and it's the first time I train on Wan (and I have little experience on other models too). But let me tell you how I got here.

During my initial tests at 160p, I played around with the learning rate (1e-4, 5e-5) & LoRA+ (4, 8, 16) at 40 epochs with only 20 videos and a network dim set at 32. Wasn't that great, lots of over-fitting, overcooking, the concept was difficult to trigger and results were... Meh.

I initially wanted to first train on a small dataset to get the concept, and try to fine tune it multiple times with new datasets and differents use cases, which I tried but the concept wasn't fully there yet.

So, I got frustrated a bit, and just packed all my datasets that represent the action in all angles ("Wan will maybe get it if I do a 360° of the concept?"), pruned datasets about very specific use cases that wasn't necessary, reworked most captions to be as precise and descriptive as possible (I helped myself with Florence2 visual model). I decided to use 24 dim since it created better results in my previous steps (and I remembered that concept = low dimension from SDXL), and I pumped the epochs count to 60 because my last run at 40 was under-trained.
All of that on a 360p run to test my assumptions (~3h). The result was very good finally ! This forsaken model got it !
But hairs were messed up by the original resolution, so I did yesterday the 480p run, and here it is.

I also think that my dataset selection was pretty important, once everything was trained together, it always went great. I hand picked the movement in each source video with different speeds (with same captions but changed the speed word, I though that the model would understand the delta better), also selected where it started and finished (from the base, from the top, etc), in all angles, positions, etc.

It's still lacking on details though, since I brute forced it without fine tuning. I wonder if I can fine tune it without overcooking it...

Well, I hope it helps! I still have a lot to learn!

@CyberAImania Thanks, I appreciate it! The VRAM headroom is indeed nice aha! I miss my kidneys though.

About Florence2, it's just a plugin in ComfyUI you can install (comfyui-florence2). I load the video in ComfyUI, extract one frame, and run it though Florence2 in "more_detailed_captions" mode, it output a detailed description of the image that I use as a base to add movement captions & my trigger words (double check the output though, it sometimes hallucinate).
I didn't automated it to create the .txt file or update the .jsonl, but I think it's doable. I just copy paste the captions in my .jsonl file, tweak it, and load the next video. I don't recommend using the captions as-is though.

It's a bit tedious though, took 2/3h for the 60 videos. Some videos were extracted from the same source video, so I didn't run ALL 60 videos through Florence2, just copy pasted and quickly changed it manually. But I think it's something we need to go through for good results.

For video splitting, I use Shotcut, it's free, works well.
For dataset management (transcoding, etc) I just use a lil' script I made using ffmeg, always in lossless. I can convert my whole datasets to another resolution quickly with that, so I can try things out and iterate.

And shit tons of tea and funny videos to watch while the GPU is melting under all the training iterations ahahaha

@CyberAImania I'm surprised that those cloud based AI are not filtered. The caption is a bit short though, and danbooru tag style, I think, doesn't work that well with Wan since it seems to prefer verbose captions, and it's difficult to describe a movement with only tags.

Here is an example of a raw Florence2 output on one of my training video's frame, if it can helps those that will read this discussion:
---
a high-resolution, explicit image depicting a sexual act, the central figure is a young woman with light skin and long, brown hair, kneeling on a beige carpeted floor, she has a slender physique with small to medium-sized breasts, and her nipples are visible, her facial expression is one of intense focus, with her eyes half-closed and her lips parted, as she performs oral sex on a man whose erect penis is prominently displayed in the foreground, the man's skin tone is light, and his pubic hair is visible, the background includes a white towel on the left side and a black power cord on the right side, suggesting a domestic setting, the image is taken from a low angle, looking up at the woman and the man, emphasizing the intimate and explicit nature of the act
---

Which I manually transformed to:
---
blowjob, deepthroat, pov; a POV on a young woman with light skin and long, brown hair, kneeling on a beige carpeted floor, she has a slender physique with nude medium-sized breasts, and her nipples are visible, her facial expression is one of intense focus, with her eyes half-closed and her lips parted, as she performs a deepthroat blowjob on a man whose erect penis is prominently displayed in the foreground, she is bobbing her head back and forth while sucking the man's penis, the man's skin tone is light, and his pubic hair is visible, the background includes a white towel on the left side and a black power cord on the right side, the video is taken from a POV, looking down at the woman and the man's penis
---

@tensor_fanatic See my last message in this discussion.

@_Species8472_ Hey there. First of all, note that this discussion was about the Wan 2.1 version. Wan 2.2 I2V is similar, but I did added penises datasets to it, because 2.1 I2V suffered from deformed penises when they were introduced to the image (so not present originally). I wanted to circumvent that a bit, so I trained the LoRa on it too (it had the funny side effect to attach penises to all new characters introduced to the scene aha, some may like it ahah).

Wan 2.2 I2V has the same datasets as Wan 2.1 I2V, with an addition of 27 static images of penises (erect or not), and 14 videos of moving penises during different manipulation and state, to teach the model what is a penis first, and then how is moves and react to stimulus (does it bounce? is it hard? elastic? etc). I built this added dataset with this goal in mind.
The final result is not perfect tbh, but seemed fair enough, especially in the interaction with it & its physicality, which to me convey the scene more convincingly (I hated those "monolithic" penises in other LoRa, they felt like wood or rubber).

Wan 2.2 T2V is a whole another subject (especially on the configuration). Same datasets as Wan 2.2 I2V though but now with vulva datasets (27 static images & 32 videos for physicality). Why do that for a Blowjob LoRa you would say? Because I wanted the lora to be self contained, and be able to output a good video by itself (and woman are often nude in those). It also allowed me to eliminate the side effect of all characters having penises aha (well at least partially). Not perfected too, but does the job good enough to me.

So to make it short, I overcame this by training with my dataset concept, mixed with related concept that were missing in the base model (penises & vaginas), so Wan can get wtf I'm trying to teach it directly. I believe it output slightly better output than mixing a LoRa about genitals & the concept LoRa, but I have nothing to back it up than my appreciation of the results.

Hope it helps.

@_Species8472_ Hey, unfortunately I can't share it.
I'm happy to share the process, the configurations & my approach though.

The static images is basically a bunch of cleaned-up photos of penis of various sizes, colours, with or without foreskin, flaccid or erect, from various angle (front, above, below, left/right, three-quarter, etc). I tried to include as much variability as possible. I try to cement the look of a penis with those.

The videos describes various interactions with the penis from different angles and framing (so it's correctly placed between the legs), like handling with a hand, moving it erect or flaccid, pulling the foreskin back, etc.
I considered that adding this context to the training would improve the data learned from the original blowjob dataset, since the model would now know what the keyword "penis" is in the training captions, and spend more time refining the concept of penis and the interaction it has with the subject's mouth, which it would categorise as "blowjob".

If the base Wan 2.2 models was NSFW, it wouldn't be necessary, but it isn't, so I feel like we need to include those kind of datasets in all XXX loras if want them to be usable with the base model.

By the way, I'm sidetracking a bit, but I tried my lora on the the recent Smoothmix NSFW Wan 2.2 model, it works sooooo much better in T2V, amplitude, generalisation, control, everything is awesome. I guess the success of an XXX lora on the base model is mainly based on its ability to jailbreak the model out of SFW with good and profuse nudity datasets? Seems inefficient though...

Wow that's awesome to read! So it does works on T2V, I didn't even tried, I though it was a lost cause aha! I wonder if I can convert it by pruning some blocks to avoid those warnings?

I'm happy also to read that it triggers well in most of the use cases, that was my original intention & motivation.
The first DT T2V LoRA that went out a month ago is pretty good, but difficult to trigger and alter the original image too much to my taste. I was also frustrated by the ex-nihillo penis lengthening, or the disappearance of the glans when the subject pulled back. I think I partially solved it (at least, most of the seeds, some are still cursed), so mission (partially) accomplished! :D

I was also surprised that breasts & penis physics went out great, thus making this LoRA good by itself. I'm not sure why yet, but I take it!
I even tried to do an I2V on a random image of someone sitting on the ground, the LoRA triggered so well that the subject started to suck its own fingers ahahah! This LoRA definitely needs to chill, where is the horny police...

All in all, thanks! And happy to share :) Apes strong together.

Hey! Yes, I2V only, since there is already quite some T2V LoRA available. Another commenter told me it works pretty good on T2V though, but looking at your comment, it seems it's far from perfect.

But I'll take a look at training a T2V version, not sure when though :)

@WhatTheGuy I'm working on a T2V version, I've had a 8h train that just finished, no guarantee when it will be out though, as T2V seems more difficult to train than I2V (or I got lucky on I2V?).

@WhatTheGuy Yeah, I noticed that I need to fill the gap on how a male genital looks, I've got some pretty cursed inferences after my firsts tests ahahah (the great curse of glans disintegration returned). So far, I've tried still image of penises alone, to anchor their concept, and let the original I2V videos stitches the rest together, there were some improvements, but the LoRa is still too finicky without specific trigger words and not very malleable. Noticed also that inferences were more dynamic, and prone to hallucination, without still images.
I'm still exploring... Maybe including DT still images instead of penises alone ? Or solo penises videos ?

Ooooh... I think I've got it.
I'm realising writing this that I should probably take this T2V training as a regular stable diffusion training (to be able to generate a good first frame), with added movements information to it to infer the rest of the video from the first frame. That would make a lot of sense. :o

Thanks! I appreciate the comment :)
Yes, I'm currently looking into building a T2V version. I'll release it if it have the same qualities as I2V (easy trigger, all angles, leave the model free enough to hallucinate, etc).
Currently doing the test runs at low res.

Inference for images without penises already present is a bit more difficult to trigger cleanly, I recommend including one in the source image by in-painting if necessary for cleaner results. But you can play with the Shift & CFG & Seed to try to get a somewhat good result out of it. Coupling my LoRa with a Penis LoRa could also improve the result (so Wan know better what a penis looks like).

About the head appearing out of nowhere, it can happens if the CFG or the Shift are too high (Wan hallucinates too much) or you are on a cursed seed, at least to my experience. You can try lowering a bit the LoRa weight as a last resort if it's still broken (down to 0.7 minimum, lower is broken).

It could be also the prompt being too evasive, did you try to use my helper phrases I describe in the LoRa presentation ?

You have a working example here: https://civitai.com/images/75908889
No penis in the original image, but it could be hallucinated by coupling a dick lora, and a careful prompting.

I'm not sure, but I think it's because the model identifier per version changed. I've just updated mine to represent "Wan i2v 480p", I suppose it should solve the problem (at least I can select it in the cloud generation UI)

For what I've tried, the current LoRa version can output some pretty good results on Wan 2.2, though not perfect, but the movement is here. You just need to push the LoRa weight to 2 (on the high noise and low noise).

It's out! I hope it'll be as good as the 2.1 version :D