Audio, Images & Lyric Video — Workflow Guide

108 nodes · 7 groups · 30 unique node types
103 Eclipse Nova nodes, supported by ComfyUI's built-in nodes and Audio Separation.

Workflow: Audio_Transcript_Render_Lyrics_iv2v.json

What Is This?

Turn a song or spoken recording into a video with timed captions, using your own images or existing video clips as the background.

The workflow provides:

Audio loading and trimming, with optional Demucs vocal separation for analysis.
Automatic transcription, editable lyrics, and optional manual timing corrections.
A caption preview on black, so you can hear and check the timing before rendering the background video.
An image playlist with independent crops, ordering, transitions, and metadata selection.
Even, silence-based, or manual image switching across the audio duration.
Transitions and global effects, including morphs, ghost zoom, glitches, light leaks, and combined black pulses with flashes.
An alternative video playlist for adding captions to footage you already have.
Animated lyric captions and MP4 export with the original soundtrack.

The image path builds a video from held images and visual effects. Character movement and lip sync must already exist in footage supplied through the video path. No diffusion or video-generation checkpoint is needed for this workflow.

Groups share settings through Eclipse Nova's Set/Get nodes, keeping the canvas organized while preserving a common size, frame rate, and audio clock.

Required Nodes and Model Setup

Custom Node Packages

ComfyUI Eclipse Nova supplies the loaders, transcription, lyric rendering, image timeline, effects, routing, and saving. Use a build containing Load Image Timeline, Image Timeline Video, and their current blending controls.
Audio Separation Nodes for ComfyUI supplies the AudioSeparation node used for Demucs vocals. Install it to load the supplied workflow without missing nodes; using the vocal-separation branch is optional.

Install each package's requirements into the Python environment used by ComfyUI, then restart ComfyUI and refresh the browser. Smart Model Loader Nova and SmartLLM Nova are not required by this workflow.

Whisper Large-v3 — Required for Automatic Transcription

Eclipse uses the CTranslate2 version of Whisper large-v3. Download the five runtime files from the supported faster-whisper-large-v3 revision:

model.bin
config.json
tokenizer.json
vocabulary.json
preprocessor_config.json

Place them in ComfyUI/models/whisper/large-v3/.

Whisper requires this manual model setup. Eclipse verifies the files, so use the linked revision. A different conversion or a Turbo checkpoint is not interchangeable with these files.

Qwen3 Word Timing — Optional

The supplied Transcribe Audio node currently selects Qwen3 · compare only. This runs an additional timing comparison while retaining Whisper's timestamps. Select Whisper for a run without Qwen.

To use Qwen, install the optional requirements from the Eclipse Nova repository with ComfyUI's Python:

python -m pip install -r requirements-qwen.txt

Restart ComfyUI after installation. On the first Qwen run, Eclipse downloads missing files for Qwen3-ForcedAligner-0.6B-hf, approximately 1.85 GB for a complete installation, into:

ComfyUI/models/qwen3_forced_aligner/Qwen3-ForcedAligner-0.6B-hf/

Later runs reuse the verified local files. This downloads model files, not Python dependencies. Qwen estimates timings for supplied words; it does not correct the transcript or require an instruction prompt.

Wav2Vec2 — Optional Image-Switch Analysis

The optional activity encoder uses wav2vec2-chinese-base_fp16.safetensors in ComfyUI/models/audio_encoders/.

It helps find quiet points for Silence image switching. It does not transcribe lyrics or control caption timing. To run without this model, disconnect audio_encoder_output on Timeline inputs. Keep analysis_audio connected for Silence mode to analyze the vocals or original audio directly.

For Even or Manual image switching, disconnect both analysis inputs on Timeline inputs to avoid unnecessary analysis. One enabled image skips these branches automatically.

Fonts and Media

Caption fonts come from ComfyUI/models/fonts/. Choose an installed font that supports the language of your lyrics. Eclipse supplies default fonts when initializing an empty font folder.

Before your first run: replace the example audio, image, and video selections with your own files. The workflow stores file references; it does not include those media files. Clear or mute any example manual lyrics and timing before changing songs.

Quick Start — Make a Lyric Video from Images

Set the canvas. In Video Settings, choose the size and FPS in Smart Folder. The supplied workflow uses 864 × 1152 at 25 FPS. These shared values also reach the loaders and caption renderers.
Load the soundtrack. In Audio Settings, select or upload an audio file, or a video containing the audio you want. Use Load Audio's start and duration to select an excerpt. Duration 0 uses the remainder of the source.
Prepare transcription. Clear or mute the manual correction fields. Choose the language in Transcribe Audio, or use Auto. Choose Whisper if you have not installed the optional Qwen backend.
Add your images. Keep Load Image(s) active and Load video (s) muted. Use + Image in Load Image Timeline, or drop multiple files onto its list. Enable the desired slots and arrange their order.
Check the crops. Set each image's fit and crop position, then click Update previews to retrieve the shared dimensions. Choose the image-switch mode and transitions in Image Timeline Video.
Review the lyrics. Use Play on the black caption preview to run through that preview. Listen, correct words or timing if needed, and rebuild the preview.
Review the image video. Run the normal queue. With unchanged reviewed inputs, execution continues to the image timeline preview, where the second Stop gate lets you inspect crops, image changes, and effects.
Render and save. Queue unchanged again to continue through final captions and Save Video. Configure the final caption appearance in Render Lyrics, and the destination and filename through Video Settings and Save Video.

Play and Stop serve different purposes: the node's Play button executes upstream work through that node, without downstream saving. Stop controls a normal full-workflow queue: changed results pause for review, and a subsequent unchanged queue continues. The media player's play button simply plays the existing preview.

How the Workflow Fits Together

Audio → transcript and timing review → image timeline or loaded video → final lyric overlay → Save Video

Three things remain independent:

The soundtrack clock: Load Audio defines the excerpt used by the rest of the workflow.
Image-switch timing: Even, Silence, or Manual determines when the background image changes.
Caption timing: transcription, alignment, and your corrections determine when words appear.

Changing a blend, crop, or global style does not move the soundtrack or the planned image-switch times.

The small IO nodes pack inputs into a PIPE and extract results. Keep the existing PIPE connections when customizing the layout: they carry per-image options and the reviewed lyric data between stages.

Group-by-Group Reference

1. Video Settings

Smart Folder supplies the shared width, height, FPS, and output folder. The supplied folder configuration saves beneath ComfyUI's output directory in a dated video folder.

Change the shared size here instead of changing each downstream panel separately. Connected size inputs take priority over local values. After changing dimensions, refresh the image crop previews before rendering.

For this native-video flow, Save Video preserves the incoming video's frame rate. Its local FPS field does not override the video already built upstream.

2. Audio Settings

Load Audio selects the recording and the excerpt. The resulting audio is the original soundtrack used by the image timeline and lyric renderers.

The optional Demucs branch separates vocals for transcription and activity analysis. Its vocals do not replace the music in the exported lyric video. Disable the Demucs branch with its toggle to use the original mix for analysis instead.

Keep Clip-relative start in the transcript group at 0 for this workflow: Load Audio has already selected the excerpt. Applying the same start offset again would trim it twice. After changing the recording or excerpt, rebuild transcription and review any manual corrections against the new audio.

3. Transcript & Corrections + Caption Timing Preview

Transcribe Audio returns recognized lyrics and timing. Its Play button lets you inspect this result before rendering either background path.

To correct words without editing timestamps:

Copy the complete transcript into Manual lyrics.
Correct the wrong words, remove unwanted text, or supply the missing sung lines.
Enable Manual Lyrics + Timing and leave Manual timing JSON empty.
Rebuild the black caption preview and listen again.

The renderer keeps the original timing as a reference and aligns changed wording within the corresponding phrases. Unchanged lyrics can reuse their existing timing. This does not guarantee recognition of every missing sung phrase, so check the preview and report.

To correct timestamps explicitly: copy the current timing JSON into Manual timing JSON, then edit the relevant line or word times. The JSON must match the selected lyrics, including line structure and word text. Populated manual timing takes priority over automatic alignment. For wording-only edits, leave this field empty; the current PIPE flow needs no {} placeholder.

Muting the manual inputs restores the original transcript and timing.

Word timing choices:

Whisper: use the established Whisper timing.
Qwen3 · compare only: retain those times and include alternative estimates and disagreements in the report.
Qwen3: apply valid complete phrase estimates; invalid or incomplete estimates retain their existing timing.

Singing, sustained notes, and repeated phrases can challenge both models. A different estimate is not automatically more accurate. For a Qwen comparison after editing lyrics, select it in the black preview renderer's Alignment & timing section; the initial Transcribe Audio comparison only covers its original recognized text.

The black preview plays the original soundtrack with captions. Its Stop gate pauses a normal queue before background rendering. The three SRT, timing JSON, and report viewers are for inspection; they are not review gates or required links in the execution path.

4. Load Image(s) → Selected Images → Video

Despite the group title, this version uses Load Image Timeline directly. There is no separate Image Selector confirmation step.

Add, Arrange, and Crop

Click + Image, then browse or upload a file.
Drop several images onto the left-hand list to append them as slots.
Dropping onto the preview replaces the selected image with the first file and adds any remaining files as new slots.
Reorder using the drag handles or arrows. Disable a slot to keep it for later; removing a slot does not delete its file.
Set Crop, Pad, or Stretch per image. New slots default to Crop / Center / Center. Horizontal and vertical crop positions let you preserve a subject near an edge.

Update previews refreshes crops using connected width/height values. With shared size inputs connected, it runs only the work needed to resolve those inputs and load the images; it does not render the timeline or save a video. Without those connections, local-size previews update without queueing.

Choose When Images Change

One image: hold it for the complete audio duration. Global styles can still animate visual effects over it.
Even: divide the duration among all enabled images.
Silence: search near evenly spaced targets for a quiet gap. The silence controls choose the search range, minimum gap, and whether the switch follows the gap start or sustained activity resume. If no suitable gap is found, that target keeps its even spacing.
Manual: enter increasing switch times in seconds, one per image after the first. For three images, 10, 22.5 switches to image two at 10 seconds and image three at 22.5 seconds.

Check the Timeline report for the actual cuts and any fallback. Activity detection measures quietness or changes in activity; it is not a musical beat detector or a guarantee of vocal silence.

Choose How Images Change

Each slot's Blend to next and Blend seconds apply to the next enabled image. The final image has no outgoing transition.

Choose Use timeline setting to inherit the default from Image Timeline Video's Blending section.
Choose a specific slot transition to override that default, including its duration.
Use Cut for an immediate switch.

Available effects include smooth crossfades, filmic and perceptual blends, dissolve, wipes, and:

Morph: optical-flow warping between images. Similar framing works best; very different images can stretch or ghost.
Ghost zoom: enlarged, offset translucent copies around the transition.
Flash / light leak: soft, warm-white flashes over the blend.
Block glitch: displaced blocks, scanlines, and split-color edges.
Zoom blur: a zoom-and-blur pulse across the change.
Dip to black: fade out, then reveal the next image.

Blend windows are centered on the planned cuts. Short image intervals reduce the available blend length. Transitions do not extend the video or shift the audio.

Give the Whole Video a Style

Global style affects every image interval and transition, including a single-image video.

Cool desaturated applies a continuous color grade.
Ghost zoom, Light leak, Block glitch, Zoom blur, and Black pulses repeat their visual effect across the sequence.
Black pulse flash combines two soft warm-white flashes with a black dip in each cycle.
Ghost edit combines grading, ghost zoom, and flashes.
Glitch edit combines grading, blocks, zoom blur, flashes, and black dips.

Style strength controls intensity. Style period sets the cycle length in seconds. The effect clock continues across image changes and is not automatically synchronized to musical beats. Use Off for no global styling; per-image transitions still apply.

Metadata and Review

Image metadata can read the selected image's saved generation settings. Enable Use metadata for one image and choose which sampler, prompt, and seed groups enter the image PIPE. This does not change metadata at every image transition.

The supplied Save group builds its own size/FPS/filename metadata. Selected-image prompt or seed data is not currently connected to that Save metadata path.

Keep Load Image Timeline → PIPE → Timeline inputs connected. Leave the IO's separate images input empty: the PIPE already contains the images and their crop/blend settings.

Image Timeline Video streams the rendered frames to a temporary video file. The group's Preview Video is the second Stop gate. Review its result and queue unchanged to continue.

5. Load Video (s) — Alternative Background

This group is muted in the supplied workflow. To use footage:

Mute the entire Load Image(s) group.
Enable Load video (s).
Add one or more clips with + Video or drop them onto the playlist.
Set their order, frame trims, fit, and optional transitions.

Load Video supports per-clip Blend to next settings, a default Blending section, and the same Global style effects. Build a joined preview to inspect the result. Picture transitions preserve the existing clip durations and audio cuts; they do not crossfade the audio.

Shared width, height, and FPS feed the loader. The supplied workflow uses fixed timing. The final lyric renderer uses the loaded audio as its soundtrack and ignores the background video's audio. A shorter background holds its last frame through the remaining caption duration; footage beyond that duration is omitted.

The background selector prefers the image timeline when both sources are active. Mute the unused source group to avoid rendering an unused branch.

6. Render Lyrics

This renderer places the reviewed lyrics and timing over the selected background. It can reuse the resolved timing instead of starting transcription again merely because the background or appearance changed.

Choose Whole line, Active word, Floating words/lines, or Rotating words/lines. Then adjust the font, size, colors, outline, placement, animation, and optional glow. The seed controls repeatable floating placement.

Appearance belongs to each renderer. The black review and final overlay can have different fonts, sizes, and animations. Timing adjustment is also local: if you use a visual timing offset in the review renderer, apply the intended offset in the final renderer too. Positive values delay captions; they do not shift the audio.

7. Save Video

Save Video with Generation Data exports the selected result as MP4/H.264. Adjust CRF and the encoding preset here; lower CRF generally means higher quality and larger files.

The save path comes from Smart Folder through the filename controls. The workflow currently enables embedded workflow and generation metadata; a separate workflow JSON sidecar is optional.

The source selector prefers the captioned video. Mute the final lyric renderer to save the selected background video without captions. Use Mute for this fallback rather than Bypass.

Soundtrack behavior: native VIDEO retains its embedded audio. Save's separate AUDIO socket does not replace it. The image timeline already contains the original soundtrack. If you instead save a loaded-video fallback with its clips muted, that fallback is silent—even though the Save node has an AUDIO connection.

Common Questions

Why Did the Queue Stop Before Saving?

The black caption preview and image timeline preview both have Stop enabled. Changed results require review. Queue again without changing those inputs to continue, or disable the relevant Stop control for uninterrupted full queues.

Why Does Changing the Timeline Blend Do Nothing?

A specific Blend to next selection on an image overrides the renderer's default. Set that slot to Use timeline setting to use the shared blend and duration. Global style is a separate whole-video effect.

Why Is the Crop Preview Still Using the Old Size?

Click Update previews after changing shared dimensions. Connected width/height values take priority over local fields, and crop edits use the most recently retrieved input size.

Why Are Some Lyrics Missing or Early?

First check the selected audio excerpt and any manual text left over from a different song. Then inspect the black preview and report. Unresolved lyrics may be omitted, and recordings cut before the final phrase cannot contain that phrase. Use wording corrections for recognition mistakes, explicit timing JSON for local timestamp fixes, and Timing adjustment only for a consistent overall offset.

Can I Use Just One Cover Image?

Yes. Enable one image and it fills the full audio duration. You can keep it still, add a repeating global style, and render any of the caption modes over it.

Does a Long Video Have to Fit Entirely in RAM?

The image timeline, lyric renderer, and native-video save path process video frames incrementally through files. Audio and loaded source images still occupy memory, and temporary videos need disk space. Keeping the existing VIDEO/PIPE route avoids expanding the full output into an IMAGE batch.

Start with the soundtrack, review the words and timing, then shape the images and captions into your video. 🌒

Files with SHA256: 60f685d9de2c73a0f2079599497a00acca3ede7f8c5c72c4dc94cce384cf91b5