This workflow demonstrates the newest addition to my node suite: video as a target.
This workflow will show you in particular how to use audio to modulate the direction of video playback.
There will soon be other video-level targets to manipulate.
DONT FORGET: this works NOT ONLY with audio, but with ANY other feature as well (time, MIDI, motion, proximity, etc....etc....)
Give me a star on github and a sub on youtube if you think this is cool. Thanks!
Ryan
Description
FAQ
Comments (10)
That is what I am talking about!
I have the following error
DownloadOpenUnmixModel
unexpected EOF, expected 3499992 more bytes. The file might be corrupted.
Downloading umxl model...
Exception in callback ProactorBasePipeTransport.call_connection_lost(None)
handle: <Handle ProactorBasePipeTransport.call_connection_lost(None)>
Traceback (most recent call last):
File "asyncio\events.py", line 84, in _run
File "asyncio\proactor_events.py", line 165, in callconnection_lost
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
Using cache found in C:\Users\User/.cache\torch\hub\sigsep_open-unmix-pytorch_master
!!! Exception during processing !!! unexpected EOF, expected 3499992 more bytes. The file might be corrupted.
Traceback (most recent call last):
File "C:\Users\User\ComfyUI_windows_portable\ComfyUI\execution.py", line 323, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\User\ComfyUI_windows_portable\ComfyUI\execution.py", line 198, in get_output_data
return_values = mapnode_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\User\ComfyUI_windows_portable\ComfyUI\execution.py", line 169, in mapnode_over_list
process_inputs(input_dict, i)
File "C:\Users\User\ComfyUI_windows_portable\ComfyUI\execution.py", line 158, in process_inputs
results.append(getattr(obj, func)(**inputs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\User\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_RyanOnTheInside\nodes\audio\audio_nodes.py", line 46, in download_and_load_model
separator = torch.hub.load('sigsep/open-unmix-pytorch', model_name, device='cpu')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\User\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\hub.py", line 570, in load
model = loadlocal(repo_or_dir, model, args, *kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\User\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\hub.py", line 599, in loadlocal
model = entry(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\User/.cache\torch\hub\sigsep_open-unmix-pytorch_master\openunmix\__init__.py", line 342, in umxl
target_models = umxl_spec(targets=targets, device=device, pretrained=pretrained)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\User/.cache\torch\hub\sigsep_open-unmix-pytorch_master\openunmix\__init__.py", line 293, in umxl_spec
state_dict = torch.hub.load_state_dict_from_url(target_urls[target], map_location=device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\User\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\hub.py", line 769, in load_state_dict_from_url
return torch.load(cached_file, map_location=map_location, weights_only=weights_only)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\User\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\serialization.py", line 1114, in load
return legacyload(
^^^^^^^^^^^^^
File "C:\Users\User\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\serialization.py", line 1357, in legacyload
typed_storage._untyped_storage._set_from_file(
RuntimeError: unexpected EOF, expected 3499992 more bytes. The file might be corrupted.
try deleting the openunmix folder under models and downloading again
@ryanontheinside The folder is empty, what is the name of the file? EDIT: I added the files from this repository but still doesn't work
https://github.com/sigsep/open-unmix-pytorch/tree/master/openunmix
@gabibmx you can delete what you added from there, grab the umxl and umxhq models from the following page
https://sigsep.github.io/open-unmix/#%F0%9F%8F%81-getting-started
So openunmix folder should look like:
bass-8d85a5bd.pth
drums-9619578f.pth
other-b52fbbf7.pth
vocals-b62c91ce.pth
training-json-logs.zip (unzipped)
UMXHQ-MUSDB18.zip (unzipped)
?
@gabibmx
-openunmix
---umxhq.pth
---umxl.pth
@ryanontheinside I am not able to find those files... one of them supposed to be in the zip file of this link right? https://zenodo.org/records/5069601
I am getting really nice results with this workflow. I'm kind of addicted to Ryan's node suite :D But I have noticed that my audio+video outputs are out of sync and some video frames run without sound in the end. Anyone know how to fix this?
Did you find a solution?