Two-hand finger-frame gesture becomes a window into an AI character-swapped version of the same footage — generated entirely locally, no API key, no cloud billing.
Local ComfyUI port of the gesture-tracking pipeline from sophiamyang's finger-frame-effect-ai. Where the original sends your clip to Gemini per generation, this workflow runs on your own GPU with Wan2.1 SCAIL-2: it generates a cropped region covering everywhere the window travels (not the whole frame), pastes it back with head-drift auto-align, and reveals it through the tracked quad with rim-only refraction.
Requires: ComfyUI-WanVideoWrapper, mediapipe (auto-installed), Wan2.1 14B SCAIL-2 checkpoint + LoRAs (see workflow notes). Tested on RTX 4090 24GB.
What it's NOT good at: multiple gesture sessions in one clip, non-continuous takes.
==
Usage notes: Pick a head image and a body image of the character you want(Also generate an identical version without clothes.) , load your own driving video, and optionally describe the appearance/outfit in the prompt. Auto-resolution in the pipeline works with both horizontal and vertical video. No length limit on the clip — tested with 30+ second videos. Just feed the input at a format/resolution ComfyUI can handle. That's it — the pipeline is fully automated, nothing else to configure. works with anime and real characters. Requires the ComfyUI-FingerWindow-Local custom node: https://github.com/stark622/ComfyUI-FingerWindow-Local-SCAIL2 (see README for install — copy the ComfyUI-FingerWindow-Local subfolder into your custom_nodes)
-tools\converter.bat if you need convert iphone or another type video to comfyui standart in one click
Description
Comments (2)
To first create a reference video for the finger animations, how do you prompt (and maybe what loras) did you use to tell the model to use her fingers? Or did you record yourself, then swap with a character, and the use that version as a reference?
Just: record a video - SCAIL-2 character swap - Flux generates the same character in a different outfit - run this workflow. (For me it was just faster to film a video than write prompts.)
Gemini's way simpler, but not free. Also — here you can pick your own anime/human reference; Gemini currently only takes a prompt for this specific effect.