This ComfyUI Workflow allows Audio Reactivity for AI animation in an EASY way š
DOCS, WF, EXPLANATIONS : https://github.com/yvann-ba/ComfyUI_Yvann-Nodes
Please Give a ā on GitHub
it really helps pushing forward the pack and share workflows
-
This Workflow lets you sync multiple image inputs with your audio, making your animations come alive by switching between images based on beats (like bass, drums, vocals...) with smooooth transitions (or sharp transitions if you're a techno guy)
It uses an audio implementation of IPAdapter to smoothly blend styles based on your audio-reactive images and includes ControlNet to help shape your animation based on your cool input video :Nice:
The WF is based on Stable Diffusion 1.5 and HyperSD (8 steps), itās designed to create high-quality animations efficiently, even on low VRAM/GPU setups (like 6gb VRAM )
Description
FAQ
Comments (20)
It's an error related to your gpu, not this pack of nodes, your gpu just can't handle the numbers of images that you feed to IPAdapter Batch, try to reduce the batch size and to add a prep img for clip vision after the images batch multi
@yvann_mp4Ā Thanks, the clip vision after batch helped.
Im running into this error while trying to run the workflow. Tried connecting the load audio directly as well as switching the files but no success. Please Help
Error occurred when executing Audio Reactive | Yvann: argument of type 'function' is not iterable File "C:\Users\...\Documents\ComfyUI\ComfyUI\execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\...\Documents\ComfyUI\ComfyUI\execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\...\Documents\ComfyUI\ComfyUI\execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\...\Documents\ComfyUI\ComfyUI\custom_nodes\ComfyUI_Yvann-Nodes\nodes\audio\Audio_Reactive_Yvann.py", line 108, in process_audio if audio is None or 'waveform' not in audio or 'sample_rate' not in audio: ^^^^^^^^^^^^^^^^^^^^^^^
I could fix the issue myself by updating comfyui and all its components.
I get the following error
Downloading umxl model...
Using cache found in C:\Users\User/.cache\torch\hub\sigsep_open-unmix-pytorch_master
Error in model processing: unexpected EOF, expected 3499992 more bytes. The file might be corrupted.
!!! Exception during processing !!! 'NoneType' object is not subscriptable
Traceback (most recent call last):
File "C:\Users\User\ComfyUI_windows_portable\ComfyUI\execution.py", line 323, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\User\ComfyUI_windows_portable\ComfyUI\execution.py", line 198, in get_output_data
return_values = mapnode_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\User\ComfyUI_windows_portable\ComfyUI\execution.py", line 169, in mapnode_over_list
process_inputs(input_dict, i)
File "C:\Users\User\ComfyUI_windows_portable\ComfyUI\execution.py", line 158, in process_inputs
results.append(getattr(obj, func)(**inputs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\User\ComfyUI_windows_portable\ComfyUI\comfy_extras\nodes_audio.py", line 155, in save_audio
for (batch_number, waveform) in enumerate(audio["waveform"].cpu()):
~~~~~^^^^^^^^^^^^
TypeError: 'NoneType' object is not subscriptable
great job and it worked at the very first time. but unfortunately I didn't understood how to get more than 2 seconds video
frame_load_cap in the load video section?
@BonescuadĀ yes !! that's it. Thank you for the suggestion.
increase video frame per second based on input video, simply calculate with chatgpt
I'm getting error with the node "IPAdapater Unified Loader; the error pops up as "ClipVision model not found". I can't find the source of this issue and when i try to manually add the clipvision model by adding nodes it doesn't do anything either? :(
if you models in another folder or drive connect them with your yaml
Edit 12/4: Up and running with the new version, thanks Yvann!
I can't get this to work. It wants the following nodes which I can't find in Manager or on Github:
IPAdapter Audio Reactive | Yvann
Audio Reactive | Yvann
I do have:
ComfyUI_Yvann-Nodes
installed and I have updated it.
Weird because I got the img2vid running fine, I'd think it would use the same resources. Hope someone can help, I think this would be useful.
Ive got the same problem...
@TedrerĀ Hmm well that's not good. I know a month ago is a decade in Internet time but still, you see it posted, you think it's going to work. Do your nodes disappear when you try to rebuild them? I'd suggest posting a top level comment, the author should be informed of that but not necessarily a reply to me. Good luck!
Thanks for the issue, I've just fixed that, it's because we have released the V2, so the wf is different and the nodes too, i just forget to update the wf on civitai, have fun !
@Tedrer it's fixed you can redownload the wf on civitai to fix it
@yvann_mp4Ā Up and running, thanks Yvann! Did this used to be the "Life in her hands" wf?
i am an extreme noob at this but i am getting an error message Cannot execute because a node is missing the class_type property.: Node ID '#495'. However I am struggling to identify the node id 495? any tips?
Go to Manager Menu > Install Missing Custom Nodes. After that download all missing models which name indicate on the red outline highlighted nodes.