This is a motion transfer workflow in ComfyUI using DiffusionGemma PromptBuilder nodes + LTX 2.3 to transplant movements from a source video onto a still image character. It allows movement while maintaining the appearance with concise instructions. DiffusionGemma NVFP4 automatically provides precise prompting at the node level, which allows you to achieve the desired result effortlessly.
Essentially, you can just use a video and an image as a prompt. Optionally, you can add additional context - but it isn't necessary.
While the image I used in the example was busier/noisier than it should be for this type of workflow, DiffusionGemma was still able to guide the generation nicely.
You'll need the nodes for this workflow; you can get them here: https://github.com/exportAnything/ComfyUI-DiffusionGemmaPromptBuilder
The nodes are still a WIP. I'll upload more workflows for different use-cases here in the coming days.
Description
FAQ
Comments (4)
I was really looking forward to testing this workflow, but unfortunately it didn't work for me. I downloaded the files (8 and 10 GB), created the folder in "LLM," but had no luck. I got an error related to the transformers package, saying it was an older version (that’s the explanation an LLM gave me). When it reached the “DiffusionGemma CoT Generator” node, that’s when I got the error. Congratulations on the effort you put into creating this workflow. LTX 2.3 has huge potential when it comes to motion control. I also tested SCAIL 2, but it didn’t impress me.
Once you download the DiffusionGemma NVFP4 model, you must create an addition folder in /LLM (the instructions in the github repo show you what to name it exactly). You place the model in that folder. If the model is just inside the /LLM folder, it will not work.
@exportAnything It’s important to know that you need to download several files into the newly created folder in LLM. Since I was using an older version of ComfyUI, I got that error related to “DiffusionGemma CoT Generator.” To resolve this error, I had to update to the latest version of Transformers. I updated it and fixed the error. The workflow is now running fine. However, it’s also worth noting that this workflow is recommended for those with RTX 50 series graphics cards (Blackwell architecture). NVFP4 is a 4-bit quantization format designed by NVIDIA specifically for RTX 50 series cards (Blackwell architecture). That’s why I kept getting errors like “[WARNING] NVFP4 scaled_mm failed: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasLtMatmulAlgoGetHeuristic( ltHandle, computeDesc.descriptor(), Adesc.descriptor(), Bdesc.descriptor(), Cdesc.descriptor(), Ddesc.descriptor(), preference. descriptor(), 1, &heuristicResult, &returnedResult)". It may also run on older graphics cards other than the RTX 50 series, but there will be some errors.
@drak0n Yeah you're right. I had addressed this in r/ComfyUI several times, and I took for granted that people already knew that NVFP4 is Blackwell. I was confused about fp16/8/4 when I started out with my 3090. I'll emphasize that moving forward. I did add an additional toggle in the DiffusionGemma model loader for "low" VRAM (18GB is the least amount of VRAM possible to run the model). You can also offload the model after it's finished before moving on to the rest of the LTX workflow. I'm glad you were able to still get it running.