SDXL 1.0 isn't exactly the most reliable when it comes to making LoRAs. It's just too old and doesn't have the advancements other newer models have. Try making this on something like Pony or Illustrious; the results will look much cleaner, as long as you tag your images cleanly, correctly, and in-depth.