Based on Latent Vision's "Animation with weight scheduling and IPAdapter" tutorial, updated and optimized for SDXL with HotShotXL. You should watch that first.

The QRMonster model is too large, by replacing it with canny and depth small it can have similar results while saving vram. Also using the f16 version of HotShotXL saves some vram. The saved vram can help push renders to higher resolutions. With 16gb I can generate 1280x720 and then hi-res fix it (2x then .65) up to 1664x936 for the 2nd pass. Without IPadapters I have been able to get even higher resolution passes rendered. If you have more vram you can likely push this even further.

There is a second set of Animatediff nodes in case you want to change the settings or add FreeInt to the 2nd pass. You could adjust the context overlap to increase some speed by sacrificing some consistency (make sure both have the same settings). You should be able to bypass the control nets and just use images if wanted. You could also pretty easily replace the text images with a video for vid2vid. Experiment with samplers, ancestral normal seem to work best for me with most checkpoints.

Note: I often get an out of memory error when moving to the 2nd pass, you can usually just queue it again to continue.