Score to Screen — One prompt → one song → a music video (YuE2 × MiniMax H3)

Write one prompt and pick your images. You get an original song made with YuE2 and a music video made with MiniMax H3. The singer lip-syncs, the band plays along with the music, and every scene change lands exactly on a bar line.

🥁 v6.2: the drummer now plays the song's own drum part, drum by drum. Ride or hi-hat, which toms in which order, a crash only where one is heard, a rest where the drums stop. 🔒 And the song is now H3's own audio track (Audio Lock): H3 draws the picture on the song's clock instead of on a soundtrack of its own, so lips and drums land on the music. 🎸🎻 The hands follow the score too: the fretting hand changes chord shape on the chord changes, a violinist's bow and fingers follow the length of the notes, and the camera moves in to the fingers when a solo starts. Plus 🔁 make only one scene again instead of the whole video, and 🥁 an estimate of how late the drummer's strokes land.

v6.2 Simple: 🥁 drum cues by instrument (adaptive), 🔒 Audio Lock (the song is H3's own audio track), 🎸 the fretting hand timed, 🎻 bow and fingers from the score, 🎥 a close-up of the fingers when a solo starts, 🔁 make only some scenes again, 🥁 drum timing estimated from the picture, plus everything from v6.1: 1–4 scenes that can continue one into the next (⛓) with the same main character in every scene (🧑), a low-memory join (💾 each scene saved to a file), the prompt LLM inside ComfyUI (no LM Studio needed), a 🎞 timeline editor that shows what was found in the song and lets you fix it by hand, 🔬 a frame-by-frame sync check, scene transitions, optional hand check of the prompts before H3
Older versions on this page: v6.1 Simple, v6.0 Simple, v5.9 Simple, v4.1 Simple (1–4 scenes, LM Studio) and v3.9 Simple (2 scenes)

Each version comes in ONE download: the workflows (English + Japanese) and the custom nodes. Only a few switches; everything else is automatic, or set to the values that worked best.

New in v6.2 (2026-10-25): 🔒 Audio Lock. Until now the song reached H3 only as a reference: H3 made its own soundtrack next to the picture and moved to that, and the finished video put the real song under it, 0.2 s later on average, with a drift that changed from seed to seed. Now each scene's piece of the song is encoded into H3's own audio track and kept there while the picture is made (ComfyUI's native H3 can mask the audio and video streams separately), so the mouth and the sticks follow the song itself; 🎚 A/V sync becomes 0 (with frozen). A drummer made this way had the strokes within ±0.01 s of the kicks. frozen (default) / anchor / off, in 🔧 Details. And the drums, drum by drum. With STRUM installed, set the drum cue style to adaptive and the drummer is told what the song's drums really play: the beat on the ride or on the hi-hat (and the moment it moves between them), tom fills by name in the order heard (high tom, low tom, floor tom), a crash only where a crash is heard (with the kick only when the kick is heard too), and a rest with the sticks held still where the drums are silent. 🔁 After a 4-scene run, make only scene 3 again: the other scenes are read from the files the run kept, so H3 runs for one scene instead of four. 🥁 When the scenes are joined, the moment the sticks and the kick foot come down is estimated from H3's picture and compared with the kick in the song (📺 Join report and 🔬). Also new: 🥁 the steady beat follows what STRUM heard (no snare hits where the song has no snare, a kick on every beat where it has one), 📘 STRUM is now part of the install, with a detailed step-by-step guide in the zip, ⛓⚠ a warning when a continuing scene brings in another instrument (the drummer may come in still holding the guitar), and 🎸 the guitar lead keeps the picking hand over the body unless the scene asks for two-hand tapping. And the hands: 🎸 the fretting hand changes chord shape at the score's chord changes and moves along the neck where a solo jumps, 🎻 a violinist's bow and fingers follow the note lengths (quick notes, note by note, held notes with vibrato; slurs in one bow), and 🎥 the camera moves in to the fingers when a solo starts. See 🎸🎻 THE HANDS IN v6.2.

New in v6.1 (2026-10-22): measure the sync. 🔬 After a run, open the 🎞 Timeline and press "🔬 Sync check": step through the finished MV or a scene's "H3 only" video one frame at a time (each frame is taken straight from the file and numbered), mark where the mouth really starts moving or the stick really comes down, and see how early or late it is against the song, both as H3 was told and as you see and hear it in the finished MV (🎚 A/V sync taken in). Marks are kept per video, so seeds, prompts and settings can be compared with numbers instead of a feeling. It only looks: nothing is changed and no AI is used. v6.0 added ⛓ scenes that continue, 🧑 one main character and 💾 a low-memory join; see the version history at the bottom.

🔒 AUDIO LOCK (v6.2)

Why: H3 makes video and audio together. When the song is only a reference (<Audio 1>), H3 composes its own soundtrack from it and animates to that; the finished MV then plays the real song, so every motion is off by however far H3's soundtrack drifted, differently in every take (measured drum strokes: +0.14, -0.14, -0.09 s). The old 🎚 +0.2 s only fixed the average.

What it does: the scene's piece of the song (exactly what plays under that scene in the finished video) is encoded with H3's audio VAE into the audio part of H3's latent, and that part is masked so sampling keeps it (audio mask 0, video mask 1; a ⛓ guide's video mask is kept). H3 draws the picture on the song's own clock. Three settings in 🔧 Details:

frozen (default): the song IS the audio track, kept as it is
anchor: the song as a keyframe at frame 0 (ComfyUI's Add Guide with audio; H3 still makes its own audio). Known to misbehave on some ComfyUI versions: keep frozen unless you are comparing
off: as before (v6.1 behaviour; 🎚 A/V sync 0.2 s applies again)

With frozen, 🎚 A/V sync is 0 by itself; anchor and off use the 🎚 value (anchor only guides H3's own audio, so measure first). 🔁 needs the same lock setting as the run it reuses (it says so, and ⤵ Load that run's settings sets it). 🔬 and the 🥁 estimate use the shift the video really had. Still hints for the motion itself: H3 decides how the lips and hands move, but now on the right clock.

🥁 THE DRUMS IN v6.2

Until now the drummer got one steady beat for everything, and the fills were "both sticks sweep across the drums". v6.2 adds a third drum cue style, adaptive, that uses what STRUM (an AI drum transcriber) tells apart: kick, snare, hi-hat, ride, crash and three toms.

Ride or hi-hat: when the beat is kept on the ride, the right stick taps the ride (on the drummer's right). When it moves from the hi-hat to the ride inside a shot, one line says when ("From 00:09.000 S1 moves the hi-hat hand over to the ride cymbal")
Tom fills by name: "a fill around the toms in this order: high tom, low tom, floor tom", only the toms heard, in the order heard. Never a mid tom STRUM did not hear
Crashes only where one is heard: a new section with no crash in the song gets none (simple puts one on every section start). "Together with the kick" only when the kick is heard with it. At most 2 crashes a shot
Rests: where the drums are silent for 2 s or more, "S1 rests in this shot: ... holds both sticks still just above the drums", or "stops at 00:08.000, plays again at 00:11.000". Before, the drummer kept drumming through the silence
Follows the real groove (simple and adaptive): no snare hits where STRUM hears no snare (the left stick stays just above it), a kick on every beat where the song has four on the floor
Kept from simple: the beat positions (snare on 2 and 4, kick on 1 and 3), the kit layout (also left-handed), and short shots keep the groove
The 🎼 report says what was taken from STRUM: "🥁 adaptive drum cues (STRUM): S1: ride from 00:09.000; tom fills: 00:07.500 high tom → low tom → floor tom; rests (silent): 00:12.000-00:15.000; section crashes not heard (left out): 00:04.000"
STRUM is needed (install guide in the download: STRUM_INSTALL.md). Without it (Demucs only, or drum sync off) adaptive gives exactly the simple cues, and the report says "⚠ STRUM is not working". simple stays the default
🔁 + adaptive = an easy A/B test: make the video with simple, switch to adaptive and make only the drum scene again with the same seed, then compare the two with the 🥁 estimate and 🔬

These are still hints. Whether H3 really draws a ride and a floor tom differently is what v6.2 is for finding out: please share what you see.

🎸🎻 THE HANDS IN v6.2

The players' hands now get times from the score, not only words. Three settings on the 🥁🎤 node in 🔧 Details, all on by default:

🎸 fret_cues: the fretting hand changes to the next chord shape at each chord change of the score while the player strums or sings ("At 00:04.000 S1's fretting hand changes to the next chord shape"; a bassist moves to the next root note). In a solo, the hand moves along the neck where the melody jumps (guitar / bass: 5 semitones or more; violin: when a note leaves the hand's position)
🎻 bow_finger_cues (violin, viola, cello): from the length of the notes. Quick notes = "the left-hand fingers drop onto the strings rapidly, one after another"; note by note = "the bow changes direction with each note, about every 0.7 s"; a held note = "one finger stays pressed down with a slow vibrato ... the bow draws slowly in one direction". Slurs are played in one bow
🎥 hand_closeup: when a solo really starts, "the camera moves in to an extreme close-up of S1's left-hand fingers on the fingerboard, the face out of frame" for 2.5 to 4 s, then back out. The fingers are a few pixels in a medium shot; big, H3 draws them much better

Still hints: H3 decides the fingers, and correct fingering is beyond any video model today. What changes is a hand that moves when the music changes. Hands look best with ⚡ Turbo off (20 steps).

About this workflow

This is a personal, experimental workflow. It started from wanting to make songs with YuE2 and have MiniMax H3 move in time with them, and from wondering whether a short, simple prompt could be expanded by a local model (such as Gemma 4) into prompts tuned for H3.

For the local LLM, I recommend a model that can understand images (a vision-capable model), but which model you use is entirely up to you.

Because it grew out of experiments, some features may seem odd and some things may not behave as expected. Please treat it as something to experiment with, not a finished tool. If that sounds interesting to you, give it a try!

このワークフローについて

これは個人的な実験として作っているワークフローです。YuE2 で曲を作り、その曲に H3 の動きを合わせたい。そして短く簡潔に書いたプロンプトを、Gemma 4 などのローカルで動かすモデルで H3 向けに最適化できないか試してみたい。そんな思いから始まりました。

ローカル LLM は画像を理解できるモデルをおすすめしますが、どのモデルを使うかはご自由に選んでください。

実験の積み重ねでできているので、変わった機能があったり、思った通りに動かないところもあります。完成品ではなく、あくまで実験目的での使用をおすすめします。こういったことに興味のある方は、ぜひ試してみてください。

⚠ THE CUSTOM NODES ARE NOT IN COMFYUI MANAGER

It is normal that Manager can't find them. They are inside the download: unzip it, then copy the 4 folders from custom_nodes/ into ComfyUI/custom_nodes/ (see QUICK START).

"Image Saver Save Video" comes from ComfyUI-Video-Saver (by diodiogod), NOT from ComfyUI-Image-Saver.

In Manager use "Install via Git URL": https://github.com/diodiogod/ComfyUI-Video-Saver

Searching "Image Saver" in Manager finds the image-only package, which has no video node. If it is installed, remove it (the node names overlap).

🎬 WHAT IT DOES

One prompt, two models: a local LLM splits your prompt into YuE2 style tags and lyrics, plus an H3 video prompt for each scene
🧠 LLM inside ComfyUI (NEW in v5): put a GGUF (the same files LM Studio uses) in ComfyUI/models/LLM/ and pick it. It runs on llama.cpp's official llama-server, downloaded automatically the first time, started only for the split and stopped right after, so YuE2 and H3 keep their VRAM. The default, Qwen3-VL-4B-Instruct Q4_K_M (about 3.3 GB, sees your images), downloads itself on the first run. Hugging Face models (transformers) and LM Studio (also on another PC) still work
🎞 Timeline (NEW in v5.5): the 🎞 Timeline button at the bottom left opens a video-editor style strip. Drop an image file on a scene clip, drag a clip's right edge for its seconds, drop a song file on the audio track and drag it sideways to choose where it starts, click ◆ for the transition, ▶ Play to check the flow with the stills and the song. It edits the same node settings, so nothing else changes
🔎 Analysis on the timeline (NEW in v5.7): under the waveform, what the last run found, in video seconds: 🎬 the real cuts, camera shots and sections, 🎤 when each singer sings, 🥁 every drum hit coloured by drum (▼ = the drums come in; from STRUM, Demucs or the score), 🎻 when the strings are heard, 🎹 piano key strikes. Players the audio can't check (guitar, bass) are marked "no audio check". It reloads after every run
💾 Kept with the song (NEW in v5.9): the analysis is saved in song seconds as <song>.s2s.json next to the song (copy it along with the song and its .abc to another PC). The same song again, any start and any scene lengths: the scenes are cut from it and STRUM / Demucs are skipped. A new render of the same score (another vocals seed) keeps its own analysis and hand edits. Parts not analysed are marked. Picking a pre-made song shows its analysis on the timeline at once; "🔎 Analyse this song" analyses just the song (no LLM, no H3) in ComfyUI's queue, never at the same time as a run; hand edits are saved with the song, come back when it is picked again and come off for another song
✎ Fix it by hand (NEW in v5.8): drag the analysis on the timeline. 🎤 singing and 🎻 strings spans: drag the ends (start / stop) or the middle (move), double-click an empty spot to add one, a span to delete it. 🥁 ▼: drag for when the drums come in. From the next run, the H3 cues (sing / wait with lips closed, bow up / down, drums in) follow your marks, only where the scenes had an analysis when you drew; the rest stays as analysed. Strings can be drawn in songs with vocals too, where there is no audio check. ↺ = back to the analysis. Mouse wheel = zoom (0.1 s ruler, finer drags), Space = play / stop (v5.8.1)
🔬 Sync check (NEW in v6.1): on the 🎞 Timeline, pick a run and a video (the finished MV or a scene's "H3 only" video) and step it frame by frame (← → one frame, Shift + ← → one second; the frame comes straight from the file, "frame 68 / 288"). 📍 Mark where singing, drums, strings or piano really start or stop (or one drum hit) and for whom (S1–S4). The error against that run's music analysis is shown in seconds and frames, as H3 was told and as the finished MV plays (A/V sync taken in), and the marks sit on their own lane under the analysis (the music moment each is measured against as a pink line). ⚠ on marks inside a crossfade or dip. Marks keep the video and the frame and stay after a restart; a video made again is another video
🔁 Make only some scenes again (NEW in v6.2): every run keeps its scenes in output/score2screen_runs/ (the newest 5 runs, about 50–200 MB per scene). On the 🎞 Timeline press 🔁 Remake, pick the run, "⤵ Load that run's settings", press 🔁 on the scene to make again, change its text, picture or seed and run. The other scenes are only read from their files (no H3, no decode, read frame by frame so the RAM doesn't grow), everything is joined again and saved as a new video; the old one stays. A scene that ⛓ continues from the one made again is made again too. A different song, start, scene length or size stops the run with the reason (scenes are never reused on a guess)
🥁 Drum timing from the picture (NEW in v6.2): when the scenes are joined, every scene with a drum analysis is checked: the moment the sticks and the kick foot come down in H3's picture (optical flow, camera pans taken out) against the song's kicks, averaged over all the beats. The 📺 Join report and 🔬 say how late the strokes land, in H3's video and in the finished MV, with an 🎚 A/V sync to try and how well it matched (good / fair / weak). It is an estimate from the motion, not a stick detector: check weak ones with 🔬
🔒 Audio Lock (NEW in v6.2): the song is H3's own audio track, kept while the picture is made, so H3 draws to the real song (frozen / anchor / off, 🎚 A/V sync 0 with frozen). See 🔒 AUDIO LOCK above
🎞 Scene transitions (NEW in v5.5): cut / crossfade / dip to black / dip to white at each scene change. The video length never changes, so the song and the lip sync stay in step. From v6.0 the crossfade keeps the scene before moving while it dissolves (it uses the frames H3 made past the cut)
✍ Prompt check (NEW in v5.4): auto = the prompts go straight to H3. manual = the run pauses after the prompt split and a window opens with one text box per scene: fix anything, press ▶ Continue, and H3 runs with your text
One song, several scenes: YuE2 writes a single song. H3 renders each scene (1–4) from its own image, and the scenes are joined while the song plays straight through (no audible seam)
⛓ Scenes that continue (NEW in v6.0): for each of scenes 2–4 choose own picture (a separate shot) or continue: the frame right after the cut of the scene before is anchored as H3's first frame (H3's own Add Guide), so positions, poses, the place and the light go on across the cut. Handoff: 1 frame, or 5 frames to carry the direction of the motion too. A continuing scene needs no picture (picture 1 is used as the look). On the 🎞 timeline the ⛓ under each cut toggles it (blue = continues), the clip shows "⛓ continues 1", and the panel below has Start and ⛓ handoff for the selected scene
🧑 One main character (NEW in v6.0): a picture of the main character (face, hairstyle and clothes clearly visible) goes to every scene as <Picture 2>, with a line to keep the same face, hairstyle and clothes. Pick it in its own group above the scene pictures (with a preview), or with the 🧑 button on the timeline. Best together with ⛓
💾 Low memory (NEW in v6.0): each scene is written to temp/s2s_scenes/ the moment it is decoded, and the final mp4 is joined from the files one frame at a time, with 🔍 super resolution 24 frames at a time. Only one scene's frames are in RAM at any point. Disk: up to about 1 GB per scene (1280×720, 15 s), cleared when ComfyUI starts
📺 Join report (NEW in v6.0): frames per scene, how each scene started (own picture, or ⛓ with the frames asked, available and really used) and the RAM / VRAM peak of the join and of each scene's decode
⏳ Prompts first (NEW in v5.3): every scene's prompt is read before any scene is drawn, so H3's text encoder and the H3 model are swapped once per run instead of once per scene. Same pictures, less waiting with more scenes
The score drives the picture: YuE2 also writes an ABC sheet-music plan. The workflow reads its tempo, time signature and chords, then turns them into timed cues in each scene's H3 prompt
Bar-accurate cuts: every scene change snaps to the start of a bar
Band performance: write who plays what ("the white-haired woman plays the drums") and each player gets cues
🥁 Drum sync: an on-screen drummer follows the song's real drums (STRUM, which you install with the included guide; if it fails, the Demucs drum stem, then the score)
- A steady beat H3 can draw: one stick taps the hi-hat on every beat, the other comes down onto the snare on beats 2 and 4, the right foot plays the kick
- One stick in each hand, drums that come in later (the drummer waits through a guitar intro), kick-only intros and single crash hits
- Follows the real groove (v6.2): where STRUM hears no snare, the left stick stays above the snare; a kick on every beat when the song has one
- 🆕 adaptive cues (v6.2): ride or hi-hat, tom fills by name, crashes only where heard, rests where the drums are silent (see 🥁 THE DRUMS IN v6.2 above)
- Fill or no fill: a short fill, a big fill, or just one kick + crash (no made-up tom fill where STRUM heard none)
- Kit layout: tells H3 where the snare, hi-hat, crash, toms and ride are, as seen from the drummer's seat (standard / left-handed / off)
- The song is transcribed once; the later scenes reuse it, even after a ComfyUI restart
🎹 Piano sync: an on-screen pianist follows the real key strikes (STRUM + Basic Pitch, optional)
🎤 Waiting performers (NEW in v5.6): H3 tends to make anyone holding a microphone sing and anyone holding an instrument play. A performer who doesn't sing / play yet is told the posture and until when ("S1 keeps the lips closed and does not sing until 00:12.000"), and the LLM's words that make them start too early become that posture. The singing start can also be set by hand (vocal_in). The singer is found from "who sings", "vocalist", "singer", "lead vocals" and (v5.8) "S1 ... while singing" / "S1 sings ..."
🎸 Fretting hand timed (NEW in v6.2): chord-shape changes at the score's chord changes, hand-position shifts in a solo from the melody's pitch (guitar, bass, violin). See 🎸🎻 THE HANDS IN v6.2 above
🎻 Bow and fingers from the score (NEW in v6.2): a violinist's / cellist's two hands follow the note lengths: quick notes, note by note, held notes with a slow vibrato; slurs in one bow
🎥 A close-up of the fingers (NEW in v6.2): when a solo really starts, the camera moves in to the fingers (the face out of frame) for 2.5 to 4 s, then back out
🎻 Violin sync: an on-screen violinist (viola, cello) plays only while the strings are really heard, and keeps the violin down with the bow off the strings while they are silent. Automatic for instrumental songs
🔍 Super resolution (NEW in v5.2, optional): NVIDIA RTX Video Super Resolution enlarges and sharpens the finished video just before the song is put on. H3 is not run again, so movement, lip sync and timing don't change. From v6.0 it runs 24 frames at a time while the scenes are joined from their files. Off by default
🎚 Even loudness: every song is brought to -16 LUFS with a peak limiter
Singing on screen: auto / yes / no. A song can have vocals while nobody on screen lip-syncs
Instrumentals: write "Instrumental." The lyrics are removed, the melody moves to an instrument, and everyone's mouth stays closed
Skips the intro automatically: vocal onset detection (Demucs) starts the clip just before the singing
Use your own songs: load a song you made earlier (with its .abc score), or "(newest song in output/)", and ▶ Start at any second (the chorus, for example)
A/V sync: H3's motion lags the sound by about 0.2 s, so the final audio is shifted to match (adjustable)
Run log inside the workflow: 📺 Runs so far adds one line per run (video name, seeds, steps, memo), also saved as a CSV
Civitai-ready output: the MP4 is saved with metadata (all scene prompts, YuE2 style and lyrics, H3 + YuE2 model hashes)

⚠ The singing, drum, piano, violin and hand cues are hints

These features only give H3 hints: when to wait, when to sing or come in, where to hit. H3 decides how the scene is finally drawn, so it may not always follow them. The player may start a little early, keep playing through a rest, hit the wrong drum, or keep the hand in one place. Results vary from seed to seed, so if a take doesn't look right, try another seed. Shorter scenes and fewer camera changes also make the cues easier to follow.

歌・ドラム・ピアノ・バイオリン・手の動きの機能は、H3 に「いつ待つか・いつ歌う／入るか・どこを叩くか・いつ手を動かすか」のヒントを出すだけです。最後にどう描くかは H3 しだいなので、いつも思った通りになるとは限りません。うまくいかない時は seed を変えて何回か試してみてください。

⚙️ HOW IT FLOWS

Prompt Splitter (LLM, inside ComfyUI): makes the YuE2 style and lyrics, plus one H3 prompt per scene
YuE2: writes the ABC score, then sings the song (skipped when you pick a pre-made song)
Loudness + vocal onset: evens out the volume, trims the song, cuts the scenes on bar lines
ABC Timing: adds tempo, chord, band, drum, piano and violin cues to each scene's H3 prompt
✍ Prompt check: straight on (auto), or a pause to fix the prompts by hand (manual)
H3 ref2va: reads every scene's prompt first, then renders each scene from its image (⛓ a continuing scene waits for the scene before and starts from its frames; 🧑 every scene gets the character picture)
💾 Each scene is saved to a file as soon as it is decoded. 🎞 The scenes are joined from the files frame by frame (with an optional transition and 🔍 super resolution), the song is laid over them, and the video is saved
💾 The run keeps its scenes for 🔁, and 🥁 the drum timing is estimated from the picture when the scenes are joined
🔬 Afterwards, if you like: the Sync check on the 🎞 Timeline measures how well the finished video follows the song
🔁 Next time, make only the scenes you want again: the others come from the kept files

The internals are packed into subgraphs. Day to day, you only touch the scene columns (or the 🎞 Timeline), the 🧑 / ⛓ group above them, the 🎛 switches and, if you like, the 🧠 LLM settings.

🎛 THE SWITCHES

⚡ Turbo: fast drafts (off = 20 steps)
🎤 On-screen singing: auto / yes / no
🥁 Drum sync: auto / off (piano sync follows this switch too)
🎵 Song delay: seconds of silence at the start of the video
🎵 Song only: makes just the song; H3 and all video saving are skipped
🎬 Scene count: 1–4
✍ Prompt check: auto / manual (timeout_min: continue by itself after N minutes, 0 = wait)
🎞 Scene transition: cut / crossfade / dip to black / dip to white, seconds (0.5 s)
🔍 Super resolution: off / rtx (scale 1.5, quality ULTRA, strength 1.0)
🧑 Character picture: (none) / a picture (<Picture 2> in every scene)
⛓ Scene 2 / 3 / 4 start: own picture / continue, handoff 1 frame / 5 frames (also on the 🎞 Timeline)
🔁 Scene remake: (off) = make every scene / a kept run + the scenes to make again, seed same as that run / new, keep_runs (also on the 🎞 Timeline)
🧠 LLM settings: gguf (default) / transformers / lm_studio, model, GPU layers, context size

Seeds (video, vocals, score) can be locked separately in the 🎲 Seed details group. One video seed is used for all scenes of a run, so one number brings back the whole video. Rarely used settings live in 🔧 Details:

bar snapping (keep fps at 24: H3 makes 24 fps video, and from v6.0 any other value stops before the run)
🔒 Audio Lock (frozen / anchor / off), drum cue style (simple / adaptive / detailed; adaptive needs STRUM), 🎸 fret_cues (on / off), 🎥 hand_closeup (lead / off), 🎻 bow_finger_cues (on / off), when the drums come in (auto / start / a second in the song), kit layout, 🎤 when the singing comes in (auto / a second in the song), ✎ timeline marks (the singing / strings spans drawn on the timeline; filled in by the timeline)
A/V sync
song volume: even loudness (-16 LUFS, on by default) + optional volume curve

📦 REQUIREMENTS

Custom nodes (my own, included in the download):

ComfyUI-PromptSplitter v3.2: prompt split (lyrics in the language you ask for) with the LLM inside ComfyUI (GGUF / transformers / LM Studio), vocal onset (the separated vocals kept per song), song loudness. Run pip install demucs in ComfyUI's Python
ComfyUI-ABCTiming v2.50: score → rhythm, band, drum (simple / adaptive by instrument), piano and violin cues, 🎸 fretting-hand and 🎻 bow / finger cues, waiting cues for the singer and players, the analysis for the timeline (kept with the song), and the spans you fix by hand
ComfyUI-ABCInstrumentalize v1.5: instrumental score and bar-snapped scene cuts (1–4 scenes, 24 fps check)
ComfyUI-ABCScoreIO v1.46: 🔒 Audio Lock, 🔁 scene remake with the runs kept, 🥁 drum timing estimated from the picture, switches, scores, pre-made / newest song, run log, A/V sync, scene pictures, ⛓ scene continuity, 🧑 character picture, 💾 low-memory join from files with 📺 report, 🎞 transitions and timeline (with 🔎 analysis, 🔎 analyse-only, ✎ hand fixes kept with the song), ⏳ prompts first, ✍ prompt check, 🔍 super resolution, 🔬 sync check (frames read with PyAV, which comes with ComfyUI)

Other custom nodes:

ComfyUI-Video-Saver by diodiogod ("Image Saver Save Video", final save with Civitai metadata): https://github.com/diodiogod/ComfyUI-Video-Saver
Only for 🔍 super resolution = rtx: ComfyUI Nvidia VFX Nodes (official, in ComfyUI Manager): https://github.com/Comfy-Org/ComfyUI_Nvidia_VFX_Nodes . Needs an NVIDIA RTX GPU and CUDA 13 PyTorch (cu130 or newer)

Models:

diffusion_models: minimax_h3_ref2va_pruned_int8_convrot
loras: minimax_h3_ref2v_turbo_4step (⚡ Turbo)
text_encoders: qwen3vl_32b_minimax_h3_nvfp4_awq
vae: minimax_h3_video_vae_fp16, minimax_h3_audio_vae_fp32
checkpoints: yue2_3b_bf16 (default) or yue2_3b_int8_convrot (lighter)
LLM: models/LLM/ (any GGUF; put its mmproj file in the same folder so it can see your images). The default Qwen3-VL-4B-Instruct Q4_K_M downloads itself

LLM: no LM Studio needed any more. Nothing to pip install or compile: llama.cpp's llama-server is downloaded from GitHub the first time (CUDA build on NVIDIA, otherwise Vulkan). A vision model (Qwen3-VL, Gemma 4 …) with its mmproj is recommended so the LLM can see your images. Big MoE models can keep their experts on the CPU (extra_args, e.g. --n-cpu-moe 9). LM Studio on this or another PC still works: set backend to lm_studio.

Disk: free space for temp/ (up to about 1 GB per scene while a run is going; cleared when ComfyUI starts). 🔁 keeps the newest 5 runs' scenes in output/score2screen_runs/ (about 50–200 MB per scene; keep_runs = 0 keeps none). The 🔬 marks are small files in output/score2screen_sync/.

STRUM (needed for the drums from v6.2): an AI drum transcriber (free, MIT) that tells the kick, snare, hi-hat, ride, crash and toms apart. The drum cues are built on it: when the drums come in, fills, crashes, the steady beat that follows the song, and the adaptive cues. It runs in its own Python next to ComfyUI, so ComfyUI's Python is not touched. A detailed step-by-step guide with a check at every step is in the download: STRUM_INSTALL.md (Japanese: STRUM_INSTALL.ja.md). Without STRUM the workflow still runs (Demucs hits without drum names, or the score), and the report shows "⚠ STRUM is not working".

🚀 QUICK START

Download and unzip. Inside: workflows/, custom_nodes/, a README and the STRUM install guide (English + Japanese)
Copy the 4 folders inside custom_nodes/ into ComfyUI/custom_nodes/ (not one folder deeper). Updating? Replace the old folders, and reload the page after restarting (Ctrl+F5)
Install ComfyUI-Video-Saver via Git URL, and pip install demucs in ComfyUI's Python. Restart ComfyUI
Install STRUM by following STRUM_INSTALL.md: Git, Python 3.11 and ffmpeg, STRUM in ComfyUI/models/strum with its own venv and CUDA PyTorch, the drum weights, and a quick test that makes a notes.mid
Drag a workflow from workflows/ onto the canvas. Leave 🧠 LLM settings at the default (the first run downloads llama.cpp and the model once), or pick your own GGUF in models/LLM
In the scene 1 column write the scene AND the song (mood, genre, vocals or "Instrumental."). Later scenes: the scene only
Set the 🎬 scene count, then fill the columns left to right: image, seconds and text for scenes 1 → 4. Width / height are in ① Shared. Or open the 🎞 Timeline and drop your images onto the clips
- Optional: a 🧑 character picture and ⛓ continue for the scenes that should go on from the one before (the group above the scene pictures, or the ⛓ under each cut on the Timeline)
Run. Want to read the prompts first? Set ✍ Prompt check to manual: the run pauses after the split so you can fix them. Check the 📺 previews afterwards: the scene cut, the detected start, the prompts H3 got, and Runs so far
Optional: open the 🎞 Timeline and press 🔬 Sync check. Step to where the singing or the drums really start and press 📍 Mark here to see how early or late H3 was
Optional: one scene not right? Press 🔁 Remake on the 🎞 Timeline, pick that run, "⤵ Load that run's settings", 🔁 the scene, change it and run again: only that scene is made

💡 TIPS

Prompts can be in any language the LLM reads (English, Japanese, Russian ...). Add "Lyrics in Russian." (or any language) to choose the lyric language; without it the lyrics follow the language of your description
For a band, always write who plays what
🥁 Try adaptive drum cues: install STRUM (STRUM_INSTALL.md), set the drum cue style in 🔧 Details to adaptive and look at the 🎼 report ("🥁 adaptive drum cues (STRUM): ..."). To compare with simple, 🔁 make only the drum scene again with "same as that run" for the seed
🔁 Change only the text or the picture of a scene and keep "same as that run": the rest of the take stays the same. "new" gives another take of that scene
🔁 Keep the song, start and scene lengths of the run (⤵ Load that run's settings does it): the kept scenes only fit where they were in the song
🎸 Want two-hand tapping in a guitar solo? Write "two-hand tapping" (or ライトハンド / タッピング) in that scene's text. Without it the picking hand is kept over the guitar's body
⛓ Continue is for the same person going on (a wide shot, then a close-up of the same player). A scene with another person or instrument (guitar → drums): own picture, or the drummer may start holding the guitar. The orange ⛓⚠ on the timeline tells you
⛓ For a clean join, leave the continuing scene's picture at (none) (picture 1 is used as the look), or use a picture of the same people and place. A very different picture can pull against the frame handed over
⛓ alone can drift a little in the face over several scenes: add a 🧑 character picture with the face, hairstyle and clothes clearly visible
The 📺 join report tells you whether a 5-frame handoff really happened (it falls back to 1 frame when H3 made fewer frames past the cut)
Don't want someone (or a pet) to sing? Say so in the scene text ("the dog doesn't sing"): the LLM sometimes writes "... and singing" for everyone on screen
Guitar and bass have no audio check (the timeline says "no audio check (from the score)"): their cues come from the score. Drums, piano and violin are checked against the real song
Guitar intro before the drums? Leave "drums come in" on auto with drum sync on, or type the second in the song
🎻 Hands: render the final with ⚡ Turbo off (20 steps): fingers are fine detail. Ask for the fingers, not "the hands": "an extreme close-up of her left-hand fingers on the fingerboard, her face out of frame" ("a close-up of her hands" tends to become her face). A sharp picture with the neck and the hand clearly visible helps too
🎸🎻 Compare the hand cues: 🔁 make the scene again with the seed "same as that run" and fret_cues / bow_finger_cues off, then look at the chord changes and the held notes with 🔬
Drummer images: a kit with a cymbal low beside the snare on the drummer's left works best (it is played as the hi-hat)
Violinist: H3 tends to start playing when the camera cuts to a close-up of the player. Put the close-up where the violin comes in
✍ Prompt check (manual) is also a quick way to compare LLMs: look at the prompts, then ✖ Cancel before H3 runs
A take looks wrong? Open the 🎞 Timeline first. If the 🔎 analysis is off (the singing, the strings or the drums at the wrong time), drag it right there (✎) and run again. If the analysis is right but H3 didn't follow it, try another seed, a picture in a waiting pose, or a shorter scene
🔬 Compare takes with numbers: mark the same moments (the first sung word, the first drum hit) on two runs, and the errors tell you which seed or setting followed the song better
🔬 To judge H3 itself, use the scene's "H3 only" video (no crossfade, frames past the cut included); to judge what a viewer gets, use the finished MV. A mark inside a crossfade gets ⚠: two scenes are mixed there
🔒 Comparing with and without the lock: make the scene with frozen, then 🔁 make it again with off and "same as that run" for the seed. The 🥁 estimate and 🔬 show the difference in numbers
🎚 With 🔒 frozen, leave A/V sync alone (it is 0 by itself). If the lips still look early or late, measure with 🔬 first: a constant offset means the song's start, not the lock
🥁 The 📺 Join report already estimates how late the drummer's strokes land (e.g. "0.31 s after the kick in H3's video -> 0.14 s late in the finished video"). "close to half a beat: H3 may be playing between the beats" means a steady pattern looks the same half a beat later: check with 🔬
🔬 Every mark in the finished MV about the same amount late (say +0.1 s)? Then 🎚 A/V sync in 🔧 Details can go up by about that much. Videos made with v6.0 can be measured on their "H3 only" videos (the finished MV needs the scene borders v6.1 keeps)
Pre-made song: keep its .abc (and, from v5.9, its .s2s.json) next to it. Pick it, press "🔎 Analyse this song" once, fix the lanes by hand if needed, then run: the scenes use the saved analysis
Hand edits belong to their song: picking another song takes them off, picking the song again brings them back. ↺ clears them
Want to try a few songs first? Turn on 🎵 Song only (with 🎵 pre-made song on (none): with a song picked there, nothing new is made), then pick "(newest song in output/)" and make the video
Singer starts too early or too late? Set 🎤 vocal_in (the second in the song where the singing starts), or press "🎤 Singing starts at the playhead" on the Timeline
Transitions: crossfade for a soft change, dip to black for a new chapter. Keep the seed fixed and only change the transition: it finishes in moments
Turbo LoRA ON for drafts, OFF for final renders (H3 follows detailed cues better with more steps)
Scene length: up to 15 s per scene by default; up to 24 s has worked. For a long song, 4 shorter scenes usually hold up better than 2 long ones
🔍 Super resolution and the join now work from the saved scene files 24 frames at a time, so long songs no longer need the whole video in RAM. The 📺 join report shows the RAM / VRAM peak, handy if you want to measure
Resolution: multiples of 32, short side up to 768 (1024×768 recommended)
Errors? Paste the ComfyUI console log and the workflow .json into Claude or ChatGPT and ask what is missing

🗂 OLDER: v4.1 / v3.9 / v2.34

v4.1 (1–4 scenes) and v3.9 (2 scenes) use LM Studio for the prompt split; get them from the older versions on this page. The older "every option" v2.34 Full / Legacy workflow also still works with these custom nodes.

Version history

v6.2: 🔒 Audio Lock (the song is H3's own audio track: frozen / anchor / off, 🎚 A/V sync 0 with frozen); 🥁 adaptive drum cues (ride / hi-hat and when it moves, tom fills by name, crashes only where heard, rests where the drums are silent; needs STRUM, simple unchanged); 🔁 make only some scenes again (the runs keep their scenes, kept scenes read from file frame by frame, settings checked, ⛓ followers remade too); 🥁 drum timing estimated from H3's picture (📺 Join report, 🔬); 📘 STRUM needed, with a detailed install guide in the zip; 🥁 the steady beat follows STRUM (no snare where none is heard, kick on every beat); ⛓⚠ warning when a continuing scene changes instrument; 🎸 the picking hand stays on the guitar's body; 🎸 the fretting hand timed from the score (chord shapes, position shifts in a solo); 🎻 bow and fingers from the note lengths (slurs read); 🎥 a close-up of the fingers when a solo starts
v6.1: 🔬 sync check on the timeline: step the finished MV or a scene's "H3 only" video frame by frame, mark where singing or playing really starts or stops, and see the error against the song in seconds and frames (as H3 was told and in the finished MV, A/V sync taken in); marks kept per video; the join keeps each run's scene borders
v6.0: ⛓ scenes 2–4 can continue from the scene before (1 or 5 frames handed over, ⛓ under each cut on the timeline); 🧑 one main character picture for every scene (<Picture 2>); 💾 low memory: each scene saved to a file and the video joined from the files (super resolution chunk by chunk); crossfade keeps the scene before moving; 📺 join report (frames, real handoff, memory peaks); fps must be 24
v5.9: 💾 the analysis is kept with the song (<song>.s2s.json) and used again (no STRUM / Demucs for the same song); a pre-made song's analysis shows on the timeline at once; 🔎 analyse only the song (no LLM, no H3, in ComfyUI's queue); hand edits saved with the song; each recording of a score kept apart; "not analysed" / "older analysis" marked on the timeline
v5.8.1: 🎞 timeline wheel zoom (down to 0.1 s), Space = play / stop
v5.8: ✎ fix the analysis by hand on the timeline (🎤 singing and 🎻 strings spans, 🥁 drums in, only where the scenes had an analysis, in seconds of the song); the H3 cues follow it, ↺ back to the analysis; the singer is found from more wordings; the timeline panel resizes at any height
v5.7: 🔎 the last run's analysis on the timeline (singing, drum hits, strings, piano, real cuts); auto start shown where the last run started the song; only the newest song's analysis, each scene in its place, real cut lengths on the clips ("7.1s → 8.0s")
v5.6.1: fixes: the singing start set by hand works in short scenes, the LLM's other actions (waving, dancing) are kept when a singer is set to wait, "vocalist" / "singer" / "lead vocals" are recognised
v5.6: 🌐 lyrics in the language you ask for; 🎤 waiting performers are shown waiting (lips closed / hands still until their cue), the LLM's too-early words are fixed, singing start by hand (vocal_in)
v5.5: 🎞 timeline editor (drag & drop images and songs), 🎞 scene transitions, drum sync fixes (backbeat, no made-up fills, kick-only parts)
v5.4: ✍ prompt check (auto / manual): pause after the prompt split and fix each scene's prompt by hand in a window
v5.3: ⏳ every scene's prompt is read before any scene is drawn (one model swap per run); inputs laid out scene 1 → 4
v5.2: 🔍 optional RTX video super resolution on the finished video
v5.0: 🧠 the prompt LLM runs inside ComfyUI (GGUF via llama-server, transformers, or LM Studio); load / unload handled automatically
v4.1 / v3.9 (update 2026-10-14): 🎻 violinist plays only while the violin is heard; prompt split with thinking off and a 600 s timeout
v4.1: choose 1–4 scenes with all the v3.9 features
v3.9: one-zip download, even song loudness, song-only switch, newest-song pick, drums that wait / come in later, kick-only intros, fill or no fill, kit layout for H3
v4.0: choose 1–4 scenes (🎬 scene count switch)
v3.8: drum cues H3 can draw
v3.7: piano sync
v3.6: A/V sync
v3.4 / v3.5: run log in the workflow
v3.2: ▶ Start at for pre-made songs
v3.0: the new Simple line (4 switches)
v2.09: first public release

Files with SHA256: 72ba22f79f02411c1a6d989bb7ba95cfbcaaf5a02089cabe789a4f9486c19254