Update, v1.2, June 1: moved metadata to external files, allowing a consistent sha256 hash every time a file is converted. See the version notes for all changes. Tweaked the title from 'files' to 'embeddings' to reflect the tool's limitations.I wanted an easy way to convert .pt (PyTorch/PickleTensors) and .bin files for Textual Inversions and VAEs to the Safetensors format. DiffusionDalmation <a target="_blank" rel="ugc" href="https://github.com/DiffusionDalmation">on GitHub</a> has a Jupyter/Colab notebook (MIT license) that handled .pt files but not .bin files because of missing data in the .bin files that I had. Hugging Face has a function in the <a target="_blank" rel="ugc" href="https://github.com/huggingface/safetensors">Safetensors repo</a> (Apache license) that handles .bin, and probably .pt but I liked the training metadata details from the notebook version.WARNING: code within files will be executed when the models are loaded - any malicious code will be executed, too. Do not run this on your own machine with untrusted/unscanned files containing pickle imports.I started with pieces of both scripts and rewrote it into a script that will try to convert both types as individual files or a directory of files. The .safetensors gets a new file hash since it is a new file, but they are functionally identical from my testing. I have only tested on PyTorch 2 with SD1.5 TIs and VAEs though. It works on Windows with CPU or CUDA, but has theoretical support for the MacOS Metal backend and will fall back to using CPU. Buy me a Mac and I'll test it there. ;PAssuming that you're in a trusted environment converting models you trust, you can activate an existing venv and run it from there, or set up a new venv with requirements.txt - the convert.bat script should handle this for you on Windows.Using convert.bat:<pre><code>V:\sd-info\safetensors-converter\convert.bat O:\embeddings</code></pre>Or reusing an existing venv from the automatic1111 web UI and running the script directly:<pre><code>V:\stable-diffusion-webui\venv\Scripts\activate
python V:\sd-info\safetensors-converter\bin-pt_to_safetensors.py O:\embeddings
deactivate</code></pre><ul><li>By default, the displayed metadata about the original file will be stored in a "&lt;modelname&gt;.metadata.txt" file alongside the .safetensors file. You can add --json and/or --html to the command to save it in those formats instead (may be helpful for things like the sd-model-preview-xd extension for automatic1111). You can also pass --skip-meta if you don't want the metadata saved at all.</li><li>You can pass '.' in as the &lt;convert_path&gt; value to convert anything in the current directory, or provide the full path to a file or directory. It now recurses through subdirectories and convert what it finds.</li><li>If you get an error on a specific file, it may just have the wrong extension, e.g. try renaming .bin to .pt or the other way around (my .pt VAEs needed to be named .bin - VAEs are barely tested on this, so YMMV).</li><li>Safetensors metadata will be added detailing the original format and shape of the tensor (vectors, dimensions).</li><li>The .pt conversion will display and save metadata about the training model, hash, and steps, when available.</li><li>The .bin conversion does not provide those extra details, the format seems to lack that data.</li><li>Post-conversion, the script will check file sizes and compare that the output tensors match the original. It will throw an error if the file has changed too much or there's a mismatch.</li><li>The script should halt whenever there's an error, and will overwrite any existing .safetensors files with the same base name as the original file.</li></ul>Before/after file size and hashes for a few example TIs:<pre><code>.:
total 0
drwxr-xr-x 1 user 0 Jun 2 13:29 negatives/
drwxr-xr-x 1 user 0 Jun 2 13:29 nobodies/
drwxr-xr-x 1 user 0 Jun 2 13:29 sliders/

./negatives:
total 457
-rw-r--r-- 1 user 231339 Apr 30 19:59 ng_deepnegative_v1_75t.pt
-rw-r--r-- 1 user 230488 Jun 2 13:29 ng_deepnegative_v1_75t.safetensors

./nobodies:
total 9
-rw-r--r-- 1 user 3931 May 16 06:21 LulaCipher.bin
-rw-r--r-- 1 user 3152 Jun 2 13:29 LulaCipher.safetensors

./sliders:
total 105
-rw-r--r-- 1 user 50036 May 20 10:35 AS-MidAged.pt
-rw-r--r-- 1 user 49232 Jun 2 13:29 AS-MidAged.safetensors

$ for i in */*.*; do sha256sum ${i}; done
54e7e4826d53949a3d0dde40aea023b1e456a618c608a7630e3999fd38f93245 *negatives/ng_deepnegative_v1_75t.pt
4fff59d544381804f989fa1db606dce90e1a31070fb8b74ee7238508ddc88bbb *negatives/ng_deepnegative_v1_75t.safetensors

433c565251ac13398000595032c436eb361634e80e581497d116f224083eb468 *nobodies/LulaCipher.bin
79850379fbb29ece0c3c2fef0e5e9a2dee02bd65827f7a0c6743c848560fb6ad *nobodies/LulaCipher.safetensors

d9a9546a597ad34497d4a5a24624478df056b5a9426a1934efdbfd65177b120d *sliders/AS-MidAged.pt
cd5bfdc84fe2e3730162b360fbf242167a37ec2a7876a61b3ab94906de7e79e4 *sliders/AS-MidAged.safetensors</code></pre>

Convert .bin/.pt embeddings to Safetensors

Quality of life improvements:<ul><li>Added an optional Windows script &amp; requirements.txt to create its own venv in the script directory with the minimum packages</li><li>Default to using CPU (4x faster than CUDA in my testing?) but add a --accel flag if you want to use CUDA or Metal (MacOS) anyway</li><li>Added recursive directory parsing so it can handle files in subdirectories</li><li>Added a very basic count of how long conversion takes, for benchmarking</li><li>Code cleanup - merged the .bin &amp; .pt functions, only import the necessary functions, misc rewrites and fixes</li><li>Try to store vectors/dimensions for .bin files - may only have dimensions available but it'll use what it finds</li></ul>I'm sure there's more that could be improved but I'm happier with it now. Except for one...Known Issue: the .safetensors file's checksum changes every conversion - my understanding is that the safetensors package code to serialize the data in '<a target="_blank" rel="ugc" href="https://github.com/huggingface/safetensors/blob/main/safetensors/src/tensor.rs">tensors.rs</a>' uses Rust HashMaps, <a rel="ugc" href="https://doc.rust-lang.org/std/collections/struct.HashMap.html">which use a random seed</a> so the order will always be different in the actual file. The rest of the file with tensor content is identical. Short of not including the metadata, I'm not sure I can work around that.

Files with SHA256: 3a757098cf66a4bd21242488447137f37665945ac2b7c43e8f03a952d4905f8e