change to a different upscale method witch helped with speed and useing a ref to image model.

recommend for low vram users to use comfy ck-attention with lowvram selected for speed that wat working for me and my 5060ti