diffusion_model_pretraining_merge

Original repo with instructions here: https://github.com/marduk191/diffusion_model_pretraining_merge

# 🔀 SafeTensor Model Merger: Advanced Model Blending Tool

Merge multiple AI models with precision using SMA, EMA, and WMA averaging methods!

Are you tired of choosing between different fine-tuned models? Want to create the perfect blend that combines the best aspects of your favorite checkpoints? This powerful Python tool lets you merge SafeTensor models using advanced mathematical averaging techniques, giving you complete control over the final result.

## 🚀 What Makes This Special?

Unlike simple model mixing tools, this merger offers three scientifically-backed averaging methods that give you precise control over how models are combined:

- 🎯 Memory Efficient: Handle large models without running out of RAM

- ⚡ GPU Accelerated: CUDA support for lightning-fast processing

- 🧮 Advanced Math: SMA, EMA, and WMA algorithms for professional results

- 🎛️ Full Control: Custom weights and parameters for perfect blends

- 📊 Smart Metadata: Track merge settings and source models

## 📥 Quick Setup

```bash

pip install torch tqdm

```

That's it! Download the script and you're ready to start merging.

## 🎨 Three Powerful Merging Methods

### 1. 📊 Simple Moving Average (SMA) - "Democracy Mode"

Perfect for: Balanced blends where all models matter equally

```bash

python safetensor_merger.py ./models ./balanced_blend.safetensors --method sma

```

Every model gets equal vote in the final result. Great for creating stable, well-rounded models.

### 2. 📈 Exponential Moving Average (EMA) - "Recency Bias"

Perfect for: When newer/later models should dominate

```bash

# Favor recent models heavily

python safetensor_merger.py ./models ./modern_blend.safetensors --method ema --alpha 0.8

# Slight preference for recent models

python safetensor_merger.py ./models ./subtle_blend.safetensors --method ema --alpha 0.6

```

Alpha Cheat Sheet:

- 0.1-0.3: Early models dominate

- 0.4-0.6: Balanced with slight recency bias

- 0.7-0.9: Recent models take control

### 3. ⚖️ Weighted Moving Average (WMA) - "Custom Control"

Perfect for: When you know exactly which models should contribute most

```bash

# Make first model 50% of the blend

python safetensor_merger.py ./models ./custom_blend.safetensors --method wma --weights "0.5,0.3,0.2"

# Ignore middle model completely

python safetensor_merger.py ./models ./binary_blend.safetensors --method wma --weights "0.5,0,0.5"

```

## 🎯 Real-World Usage Examples

### Creating a Photorealistic Portrait Master

```bash

# Blend 3 portrait models with emphasis on the most detailed one

python safetensor_merger.py ./portrait_models ./portrait_master.safetensors \

--method wma --weights "0.6,0.3,0.1" --device cuda

```

### Evolution-Based Merging

```bash

# Use EMA to create a model that evolves toward your latest training

python safetensor_merger.py ./training_checkpoints ./evolved_model.safetensors \

--method ema --alpha 0.7 --device cuda

```

### Style Transfer Blend

```bash

# Equal parts anime, realistic, and artistic styles

python safetensor_merger.py ./style_models ./triple_style.safetensors \

--method sma --device cuda

```

## ⚙️ Command Reference

Basic Syntax:

```bash

python safetensor_merger.py <input_directory> <output_file> [OPTIONS]

```

Key Options:

- --method sma|ema|wma - Choose your blending algorithm

- --alpha 0.1-0.9 - EMA smoothing factor (higher = more recent bias)

- --weights "0.4,0.3,0.3" - Custom WMA weights (comma-separated)

- --device cuda|cpu - Use GPU acceleration when available

## 💡 Pro Tips for Better Merges

### 🎯 Method Selection Guide:

- Use SMA when all models are equally good and you want balance

- Use EMA when you have a progression of models (training epochs, refinements)

- Use WMA when you have specific models you trust more than others

### ⚡ Performance Optimization:

- Always use --device cuda if you have a NVIDIA GPU

- For models over 6GB, consider using CPU to avoid memory issues

- Process multiple merges in batch for efficiency

### 🧪 Experimental Combinations:

Try these interesting weight patterns:

```bash

# "Bookend Blend" - Ignore middle models

--weights "0.4,0,0,0.6"

# "Pyramid Fade" - Gradual decline

--weights "0.5,0.3,0.15,0.05"

# "Focus Burst" - Emphasize one model heavily

--weights "0.1,0.8,0.1"

```

## 🔍 What You'll See During Merging

The tool provides detailed feedback:

```

🔍 Discovered 3 models:

→ realistic_model_v2.safetensors

→ anime_style_xl.safetensors

→ artistic_blend.safetensors

🧮 Found 247 common tensors

⚙️ Using EMA with α=0.6 on CUDA

🚀 Processing: 100%|██████████| 247/247 [00:15<00:00, 16.2it/s]

✅ Merge complete!

📁 Saved: ./my_perfect_blend.safetensors

```

## 🔬 Technical Details

Supported Tensor Types: All standard formats including F32, F16, BF16, INT8, and scalar tensors

Memory Management: Intelligent chunking prevents OOM errors even with huge models

Metadata Preservation: Every merged model stores complete information about the merge process

## 🚨 Troubleshooting

"No models found" → Make sure your directory contains .safetensors files

"Weight mismatch" → Count your models and provide the same number of weights

"CUDA unavailable" → Install CUDA PyTorch or use --device cpu

## 🎉 Ready to Create Your Perfect Model?

Whether you're blending checkpoint variations, combining different training stages, or experimenting with style transfers, this tool gives you the mathematical precision to create exactly the model you envision.

Download the script, choose your method, and start creating model magic!

---

Have you created an amazing blend? Share your results and method combinations in the comments below! 🎨

## 📋 Full Command Examples

```bash

# Quick balanced merge

python safetensor_merger.py ./models ./output.safetensors

# GPU-accelerated EMA blend

python safetensor_merger.py ./models ./ema_blend.safetensors --method ema --alpha 0.7 --device cuda

# Custom weighted merge

python safetensor_merger.py ./models ./custom_blend.safetensors --method wma --weights "0.6,0.25,0.15" --device cuda

# Conservative merge favoring early models

python safetensor_merger.py ./models ./conservative.safetensors --method ema --alpha 0.2

```

Happy merging! 🚀