This workflow demonstrates the audio reactive IPAdapter weights made possible by this node pack. The nodes are highly modular.
In addition to audio, you can do the same with motion, MIDI, depth, color, and more.
In addition to IPAdapters, you can also directly modulate masks, and images. More soon.
Check out the github project for more information
Description
FAQ
Comments (11)
First, Thank you for what appears to be some really interesting and useful nodes and workflows! I was about to try this one, and am curious if it requires video, or if an audio file could be used on its own. I see on one of your other workflows for audio reactive output that the video input can alternately be swapped for an audio one. I was hoping to create visuals for some of my music projects. Thanks again! :)
you can give it audio directly, but each of them require "video" input to get the total number of frames and the size of the images. You can give it a stack of blank images and it should do fine. Unless its youre using motion as the feature, in which case it really does require video . Thanks for trying them out!
@ryanontheinside Thanks for the reply! I guess I could just drop each track into an editor and make a blank video to give the frame count, then prompt for visuals. Haven't tried yet, but I think I'll get a chance today. Much appreciated! :)
@SkyDood Hell yea - you can also use EmptyImage node!
@ryanontheinside finally getting around to trying this, but encountering issues. I should preface with saying that I couldn't find the detail lora you'd had applied in the initial settings, so I opted for another 1.5 based one which I got from https://civitai.com/models/82098/add-more-details-detail-enhancer-tweaker-lora
My first issues, I'm assuming were due to either the fact that I still had apps open which were eating resources (a video editor and an audio mixing app) and on the initial attempts I tried with a 720p 25fps video which contained no visuals and only the song file. On subsequent attempt (after closing those apps) I tried with a lower resolution version 480p 25fps, and this time, the process got beyond loading the video file without the memory allocation issue that was first stopping progress, but now I'm getting a different error (here is the first line):
!!! Exception during processing!!! Unable to allocate 6.25 MiB for an array with shape (480, 854, 4) and data type float32
I'm curious if I should be using a video with a specific resolution, or if it is related to something else entirely. Any suggestions you might have to offer would be appreciated. Oh, I'm doing this on a ryzen 5 machine with 32gb sys ram and a 4070 super TI with 16gb vram, and have left everything at default aside from the detail lora, the video file, and the image file, which I pointed to a 1024x1024 image of my own.
Thanks again! :)
Edit: I'm wondering if it is because I am using a video without any visuals that I am getting this error because it seems to be getting hung up during or just after the remove bg step. Another possibility I'd considered is that the video I'm using is in a landscape aspect ratio as opposed to portrait, which your original setup seems geared for.
@SkyDood this sounds like an out of memory issue - so you were on the right track with trying a smaller resolution, which will help tremendously and is often times recommended anyways as SD does better at certain things with 512x512.
Try doing the video in chunks, say 100 frames at a time. I'd actually start with 16 frames and make sure everything works, and tweak settings to your liking, and then go from there.
@ryanontheinside That's probably it. I actually hadn't even noticed that you'd included a video file, which is higher resolution than what I'd used in my last failed attempt, so it is looking like it may have been frame count, as I fed it a video that was for a full song that was over 4 minutes long. LOL! Maybe I got a little ahead of myself :) I'll try with a shorter clip a bit later. I appreciate you getting back to me. Hopefully I'll get it figured out, and this conversation will aid others in doing the same, so they don't have to bug you too. LOL! Will report back once I've had a chance to take another crack at it. I was not really considering the number of frames being the issue, and had assumed that it was in background removal on an empty video throwing a monkeywrench (like a divide by zero type thing Haha).
@SkyDood its no bother at lmk
Question: When extracting audio features, if I feed the IPAweights strategy with 10 images, for how long will each image influence the generation?. Will it cycle between the images or will he calculate the duration of influence of each image based on the length of the loaded audio?
Does it defaults to (1.0,0.0) weights? Do I need to add a FeatureMod Rebase and set up the weight thresholds manually? if I want to use, lets say, (1.5,0.0) weight interpolation?
I'm not sure about your first question, but if there is missing or unexpected behavior let me know .
For your second question, yes. Use mixer to boost gain and others if needed
@ryanontheinside On the traditional IPAWeights, you manaully set the number of frames that each image will influence the generation. For example: 10 images at 60 frames each with ease_in_out timing. How do I know for how many frames will the RyanWeightsStrategy node use each image to influence the weights of the IPA for generation?