Try playing with lora weight to get best possible results. Also, keep in mind, base model plays a very important part.