This workflow uses SAM to invert the colors of the character to be replaced. By doing so, it makes character replacement significantly more consistent that Minimax H3 alone.
How to use it
You must
(v2 and above) Install https://github.com/kijai/ComfyUI-SolAttn_triton if you want the speedup.
(v3) Install https://github.com/1038lab/ComfyUI-QwenVL for QwenVL video to text description.
(v1 only) Change the text prompt to describe both the scenario and the subject. Your goal is that Minimax H3 reproduces the original video as similar as possible.
Change the video and the picture, and make sure to change as well the resolution to match the original video.
(Optional, v2) change the SAM prompt until it masks only the subject to replace.
(Optional, v3) change the hint given to the Qwen node.
Why SAM and masking with H3
When using H3, it seems that the model will either recreate the original video as it is, without changing the subject at all (follows the <Video 1> too literally), or it will create a totally new video (follows the <Picture 1> too literally).
However, when inverting the color of the subject in the video, H3 is forced to produce a new video because the subject looks weird. So it can't follow the <Video 1> too literally anymore, otherwise it would produce a video with inverted colors.
Description
FAQ
Comments (24)
Awesome, working great !
Work it with Turbo Lora ?
I must be missing something. I'm using the workflow and have all of it the same, but the output video is nothing like the original. Am I missing some setting? or the prompt need to be changed? Yours just says "the woman"
You need to change the prompt on the left hand side, in the text (multiline) node next to the video loader, i missed that in my first run as well lol
@TheJake619 so which is the input for the actual video, the boxes on the left in his workflow seems to have the input data, also do we need to describe what the woman does in the video or is it not just going to replace her, seems little confusing
Also, the previews look to all have the same exact body shape. Can you replace characters that actually look different? Taller, shorter, skinnier, fatter, etc..
The prompt must describe both the scene and the description of the subject 1
The workflow for some reason has 4 text prompts, but only the top one is the one used for the rendering. I believe he added the extra prompts as examples for the video he is referring too.
So only focus on the top prompt, works fine for me.
@civitai7_ I have updated the description of the workflow explaining why it is done like this. It is possible to change the subject but if the subject is too different in proportions it will probably create a totally different video.
@rgfxsdg Yes exactly. I left these for reproducibility and as an example of how to use them.
I’ve revised this prompt; I think it can be used for all characters and videos.
subject_definitions:
<Subject 1> is the whose facial features, hair style, skin tone, and outfit are provided by <Picture 1>, while her physical body actions, expressions, and posture sequence are derived from <Video 1>.
<Picture 1> is the character identity reference image defining the appearance of <Subject 1>.
<Video 1> is the reference video defining the background environment, lighting setup, framing, camera motion, and action timing for the target video.
summary:
[reference generation + attribute transfer] The target video replaces the original subject in <Video 1> with <Subject 1> (defined by <Picture 1>), while precisely preserving the background scene, ambient lighting, camera movement, and motion sequence from <Video 1>.
retention_analysis:
<Subject 1> (appears in [Shot 1]): attribute_transfer - facial appearance, hair, and clothing from <Picture 1> are transferred onto the performing character, taking over the motion path of <Video 1>.
<Video 1> (background, motion, camera, lighting): fully_preserved - the environment setting, camera trajectory, action timing, and movement sequence are preserved 1:1 from <Video 1>.
<Picture 1> (character identity): fully_preserved - the characters's face, hair style, and visual identity are retained.
detailed_description:
The target video matches the cinematic style, spatial layout, and color grading of <Video 1>.
[Shot 1] The scene is set in the exact environment shown in <Video 1>. <Subject 1>, featuring the distinct facial features, hair, and attire from <Picture 1>, occupies the central framing position previously held by the original subject in <Video 1>. <Subject 1> executes the exact sequence of body movements, gestures, and facial expressions from <Video 1> in perfect synchronization. The camera replicates the motion path, focal distance, and pan/tilt angles of <Video 1>, keeping <Subject 1> consistently framed as the dynamic lighting and environmental elements unfold identically to <Video 1>.
@taoucchelamroi153 I'll take a look, have you tried this with different scenarios and subjects?
@lomote worked for me for one 10 sec video (funny, first I tried), then it doesn't on 25 secs. Sam3 do it's part good, but h3 somewhy not always want to change negative char (or it changes him for a 10 secs, then - original char). So I start savaging workflow, changed this and that and almost was ready to try to fix the prompt to tell h3 to forgot everything but CHANGE THAT DAMN INVERTED PERSON. And I switched to ref2va_pruned_bf16 model, turned off turbo lora, 20 steps on, er_sde (default seed btw), and it fckng works (because of turbo loras)! Yeah, still not tested much, I just want tell that prompt is working, but need some adjustments sometimes, and still better than every god damn time describe character, scene and, which is worsest - action in video.
Tried with ref2va_int8_convrot - yep, still working, but without turbo loras (clip 16 secs). Seems to be it struggles to produce more than 10 secs if turbo lora is on... Too late for tests today.
Works like a charm! Fantastic work mate.
For me I even changed the prompt to "replace the woman in the video with the woman in the image" and it also works nicely.
I tried this prompt and it did not work lol
@darkwaterramen the best prompt for this to work is that you describe the scene in detail, and the subject in detail. Look at the four examples.
sure but takes forever to swap a 5 sec video tho. (rtx 4090)
The reference image should be under 2048×2048. If it's larger than that, it takes forever.
The video quality also seems to matter, I think. I'm still testing it to figure out the best settings.
From my tests, video can be scaled down significantly up to 0.15mp and the final output won't get affected. If we don't downscale the video it will take significantly longer. So scaling down helps with speed and it doesn't seem to affect quality.
@lomote Thank you for the info! I'll try this! I ran the workflow with the default settings. The positive thing about this workflow is that I didn't have to install 300 custom nodes! 😂
@ikiru99percent thank you for this!
@private_joker There is now v2 with turbo lora and sol attnt. These are custom but optional and they speed up significantly the time.