Seems like I overcooked it a bit, but this is low-effort lora, so just reduce weights and you'll be fine ¯\_(ツ)_/¯