At lower strengths adding "blue shirt" helps enforce the proper color scheme.

Also you put enough strength on a color, you can influence the color of the outfit.