Prodigy vs. Adafactor for monochrome/manga data?

When I first converted my Mana model (the original Pony version of which I made a year ago today!) to Illustrious, I instantly noticed the generations looked... wrong. The colors and lines were weird no matter what I did (note the gallery has the best generations): Prompt, sampler, checkpoint, VAE, pairing with a style LoRA and more didn't really fix the results. These issues weren't present in the original Pony versions and I couldn't figure it out, and moved on. Then I converted some more models I made to Illustrious with a high degree of B&W data and saw the same issues. I got really suspicious when I made some Pony V2s with the same updated data and they worked fine. I got curious if it was really a difference between what Pony and Illustrious could handle as training data, or the training settings. I had read a few lines in training guides about optimizers, and the change from Prodigy to Adafactor as the default optimizer that seemed the biggest difference between this site's default training settings for the two bases, so I tried retraining one but with Prodigy as the optimizer choice and it just worked like I'd expect.

Does anyone who knows more about Optimizers or training settings know why this is? I found the disparity in results very strange, especially since most of what little I could find about Optimizers treated Prodigy as obsolete.