THIS DOES NOT ANIMATE IMAGES. I'm using Wan Steady Dancer to make the videos.
Tools exist to do this, so why bother?
If you've worked with existing pose extraction tools, you've probably seen the glitches—limbs vanishing off-screen, skeletons disappearing, or joints drifting into impossible positions. That got me wondering: could a model like Klein infer body positioning even when parts are out of frame?
I took stills of many videos and ran them through the normal skeleton pose extractor tools then train this lora with the results.
The outcome surprised me—it works better than expected, though it's far from perfect. V3 is my third iteration, and honestly more of a proof of concept than a finished product. I have some ideas for improvements, but can't commit to how much time I'll invest. Hopefully someone smarter runs with it.
When I used this with video, I scaled down the frames by half. I seems to go a little faster.
Examples are made with Wan SteadyDancer using the workflow from this guy. Just change the pose input to be the output of my Lora model.
I won't be uploading V1 & V2.
Notes:
V3:
Bone colors can fluctuate in video output, though testing suggests this doesn't affect usability all that much.
Front-facing only. I included some rear-facing shots in training, but clearly not enough—if your subject turns around, the skeleton will break
Description
FAQ
Comments (6)
fantastic lora!
this is a brilliant idea. I've been struggling with openpose lately. I'll definitely give this a go.
my main pipeline is qwen edit based but maybe I could patch this in
I'm going to set your expectations low and say this lora can be janky. I trained in on mostly waist up photos, which in hindsight, was not the best idea. I'm prepping a new dataset with more pose variety, including full bodies, partial bodies, more motion blurred images, etc. I might have an update within the next couple weeks.
great idea, will try
Good idea, but no matter what image with character I give for processing (standing, sitting, lying), this lora always generates a skeleton in the same front view standing pose,
It's pretty limited on what it can do.