@pxlpshr thx, problem is when I add contronlet model to the mix this workflow crashes at the fina l step "Requested to load AutoencoderKL". So contorlnet + AutoencoderKL at the same time cannot fit within my 8 GB Vram GPU :/ is there anyway to 'bake' the controlnet steps in advance or somehow defer the AutoencoderKL till after controlnet is applied?

@pxlpshr thx i played with that it can work a bit and also reducing resolution helps too
keep up the good work

@pxlpshr Where to add the controlnet?

@pxlpshr Just wanted to ask, I can have good results by lowering denoise at 0.1 for instance, adding a controlnet lineart, but then I almost have no motion anymore, do you have any suggestion?

I got it to work now nvm but now I keep getting pixelated outputs. It runs all good but then the output is pixelated? :(. am I doing something wrong, someone?

This is my workflow complete image causing it: https://filetransfer.io/data-package/dqdgcQVb#link

Install the newest version of xformers, probably from the torch website's repo for AMD if they have one, otherwise you'll have to build it... AMD wrote a HIP version of memory effecient attention that's in the current version. They only tested MI250X so I don't know if it'll help but it's in there.

@GnomeExplorer  pip install this? xformers-0.0.25-cp311-cp311-manylinux2014_x86_64.whl Im on linux

@GnomeExplorer Yeah xformers isnt for amd :(

@cocknito The current git repo has HIP code for AMD in it but you'll probably have to build your own wheel, I'm guessing manylinux is either cuda or their CPU-only algorithms.

@cocknito PS it builds relatively easily on Windows despite some weirdness in the scripts that I needed to patch last year when it wasn't supported officially with flash-attention-2 so linux build from github shouldn't hit any problems for you, I think they've included stuff in the build script to try to scale back the job count to fit in system memory and still get everything so you shouldn't have to do anything weird to run it.

@GnomeExplorer Bought a Nvidea card purely for this, waiting for it to arrive now. They better don't release sora next week sigh

@cocknito You made the right choice if this is what you want to do with it, AMD just isn't there. The situation is even worse on Windows because measuring free VRAM on the DirectML backend for torch is doable but a huge hack that requires knowing the total VRAM for the card in advance and reading in percentages from an array returned by an undocumented API it exposes which returns different values depending on the arbitrary block size you pass it.

I did the same last year (although it was mainly for rendering) after realizing AMD was mostly going the empty gesture route with GPU render support just like they have with the ML stuff on consumer cards. Like I mentioned their xformers checkin was MI250X and up. They never got the renderers up to par with the cost of the 7900XTX vs. RTX4090 either, you usually get more than double the performance for a card that's only ~1.5x MSRP for reference versions and maybe $100 more than that for the 2 slot water cooled version when it's not being scalped. The water cooling isn't really needed (the 3 slot cooler on the 7900XTX keeps it cool enough to run indefinitely at almost 3GHz like the 4090. In a workstation having 2-slot cards is important if you want to populate at least half of the PCIe lanes available on something like a threadripper pro since riser cables aren't in PCIe 4.0 spec and won't work right. If they hit MSRP again I plan on buying another 4090 this year and selling the 7900 on ebay, if prices stay high I'll probably just wait for the inevitable price crash when scalpers buy a bunch of 5090s before the holidays and gamers start claiming they're no faster than 3090s (they might not be for the ancient-engine games most people play just like the 4090 wasn't a great upgrade for that for most people) and unpurchased stock shows up at or below retail for a while.

AMD ProRender allows you to distribute rendering across the two brands of cards but it's so much slower on the 7900 that after the overhead of kicking it off it only contributes maybe a 20% decrease in render time at the cost of 100% power draw increase. The problem with ProRender is that they seem to have stopped updating the plugins for everything, so I had to install a weird outdated Blender version to even test it last year. I use Houdini and there's no build for v20 which came out ~5 months ago so that's out. Renderman has the same issue but Pixar is at least on a fairly regular release schedule and the public releases support GPU+CPU render now (everybody is calling that XPU for some reason) so I'll be interested to compare that to Karma XPU.

Now if I was only playing games it would be a different story, it had no problem running everything I tried at 60fps / 4k HDR without any dumb upscaling tricks, even the single day 0 AAA release I tried that supposedly had all kinds of performance issues (Forspoken).

After I posted I checked their timing tool and noticed a bunch of discreptancies on the matrix schedulers on RDNA3... They're actually slower for bf16 and fp16 than manually lowering the matmul to GPU code and running it as a series of packed fp16 instructions... they have the same throughput as the fp32 versions. Because the scheduling hardware distributes the fma instructions needed for wmma I don't think xformers will actually increase performance even on that, and 5700XT didn't have matrix schedulers so it probably won't do much to benefit things except maybe overall memory use. Anyway it doesn't matter now, you'll be able to do a lot with nearly any NVidia card. I tested the beta 0.9 release of the one-step SDXS model the other night with a batch size of 64x512x512 and the generation speed averaged out to 5ms per image on a 4090... they weren't fantastic quality images but I didn't have any instructions for using it properly so a second step might have been enough to fix it and still remain within I'd guess any of the faster 3000 & 4000 series cards will hit speeds well above what's needed to do realtime text/video2video with it once that's worked out. If it's already at 200fps there's a lot of leeway.

I had this exact issue yesterday! What I did was recreate the nodes myself; in my case I had to just add the IPAdapter Embeds node and use that in place of the red node that I had. You may have to replace other nodes too in order to get things to work.

Good luck, and I hope this was helpful!!

SyntaxError: Unexpected non-whitespace character after JSON at position 4 (line 1 column 5) i have this error and seem to be the same as you.

@mpr9348378 This helped me a little so I'll help a little more :)

@emiliosoiv00784 and anyone else with this issue...

First delete and replace the IPAdapter Encoder with a new node of the same name, it will have different options.

For this node...

ipadpter input = IPAdapter Model Loader node's IPadapter output.

image = Input image node's image output

clip_vision input = Load CLIP Vision node's CLIP_VISION output

Now add the IPAdapter Embeds node with these inputs...
model input = AnimateDiff Loader node's model output

ipadapter input = IPAdapter Model Loader node's IPAdapter output

pos_embed input = The IPAdapter Encoder node you just created's pos_embed output

clip_vision input = Load CLIP Vision node's CLIP_VISION output

Model output goes into your KSampler node's model input

Delete the red node and you should be good to go!

@ckarmor425 I am grateful that I no longer have a compile time error but now I've got a runtime one 😅

I set up ComfyUI for the first time to try to execute this workflow. I imported the workflow JSON, used the ComfyUI Manager to download the missing custom nodes, and then executed the steps you listed. After queueing a prompt, I got this error:

Error occurred when executing ADE_AnimateDiffLoRALoader: expected str, bytes or os.PathLike object, not NoneType

line 36, in load_motion_lora

if not Path(lora_path).is_file():

Seems I'm still missing something fundamental. Anyway, you've helped this much, maybe you're willing to help a little more 😛

@pxlpshr Thank you OP for fixing this for us who are too dumb to figure it out ourselves 😓

I'm able to run the workflow now! It took several hours and sadly all it spat out was a video of noise :(. Sorry to keep nagging you when you've done so much, but any idea what I might still be missing?

dont you need cuda to run stable diffusion in general? (Nvidia only)

+1

+1

+1

+1

I found a solution: you can use INT Constant node (add node - KJnodes - constants) in quantity of 2 to set the width and height. Connect these nodes to the Upscale Image node in width and height items respectively.

The original node with the error (SeargeIntegerPair) can be deleted in this case.

@str_bboy903 Then I have "Error occurred when executing VAEEncode: division by zero"

Just delete it and "convert input to widget" for Height and Width.

it is a motion module

I've looked around and can't find it anywhere...

Me too

@pxlpshr Thanks for your reply. I'm not sure what you mean by 'v3.' What is the name of this node?

@pxlpshr still same issue and i'm using v3

@pxlpshr all good! thanks for your help

You should create folder named ipadapter in comfyui/models/ folder and put it there

hi how are u? tnx for this module one question?

when i run this model i got this error

When it reaches" node hires "it gives this error

got prompt Failed to validate prompt for output 18: * HighRes-Fix Script 14: - Value not in list: preprocessor: 'CannyEdgePreprocessor' not in ['_'] - Value not in list: control_net_name: 'Control nets\DensePose.safetensors' not in [] - Value not in list: use_controlnet: 'False' not in ['_'] - Value -1 smaller than min of 0: seed - Value not in list: pixel_upscaler: 'ESRGAN\1x-AnimeUndeint-Compact.pth' not in ['ESRGAN_4x.pth'] Output will be ignored Failed to validate prompt for output 10: Output will be ignored Prompt executed in 0.03 seconds

Use shatterAnimatediff_v10.safetensors for motion lora

You can ignore Crystools and use "Integer Constant" instead of missing node.

@richif I am kind of new to this, do I find it in the manager nodes? Btw thank you so much for a respose!

@pxlpshr Thanks a lot for your help!

It worked for me as well!

thanks so much pxlpshr

@pxlpshr Hi! Where do I find 'toggle use_controlnet to true' ?

@debelllg In your High res fix script node, at the bottom, there should be "use_controlnet false." Click on the dot next to false. It will turn blue and say "true".

i'm still having the same problem, creating a new high-res fix script mode didnt fix it for me. there is no controlnet toggle

for highresfix : duplicate the node then on the old one use fix node recreate. then untoggle the seed node and put 0 in stead of -1 then toggle back. put same values as duplicated node. worked for me. thanks for this great workflow

It's the seed. Random seed = "-1". I fixed it by typing in the seed from the Ksampler

where do all the different links save exactly? pls help :')

hi how are u? tnx for this module one question?

when i run this model i got this error

When it reaches "node hires" it gives this error

got prompt Failed to validate prompt for output 18: * HighRes-Fix Script 14: - Value not in list: preprocessor: 'CannyEdgePreprocessor' not in ['_'] - Value not in list: control_net_name: 'Control nets\DensePose.safetensors' not in [] - Value not in list: use_controlnet: 'False' not in ['_'] - Value -1 smaller than min of 0: seed - Value not in list: pixel_upscaler: 'ESRGAN\1x-AnimeUndeint-Compact.pth' not in ['ESRGAN_4x.pth'] Output will be ignored Failed to validate prompt for output 10: Output will be ignored Prompt executed in 0.03 seconds