Experimental mixed training in 32bit Unet and 8bit TE (Kohya)

Experimental

I would not recommend this for anyone but advanced users.

In the train_network.py file find and change the unet and text encoder if desired

unet_weight_dtype = torch.float32

You can see in my training I am running the 32bit optimizer.

I am running the TE in FP8 this may cause problems but it is an experiment.

Save in Float

NOTE:

The training is taking longer then XL and it is on SD 1.5 (768x768 Latent Size)

Update:

I was fully expecting to see the failed training noise images however to my suprise I had good results on epoch 1. I might set this up for an overnight training as it is est 8 hours vs 30 minutes for BF16