Embeddings Scanner - Work in Progress
Just wanted to share a brief update to a file I've been working on today (2/27/2025)... I really meant to get started on submitting a file for "cs-cr1stal-v2.safetensors" but it instead ended up being a coding challenge.
Cover:
This is output from a file which runs a program which will scan given folders, and pull metadata from it, attempting to figure out text used in the embedding. To do this, it uses a "cosine_similarity" test, which measures how close two vectors (words, phrases, or documents) are in meaning based on their angular distance in a high-dimensional space.
Cosine Similarity:
cos(θ)=A⋅B / ∥A∥∥B∥
A and B are vectors,
A⋅B is their dot product,
∥A∥ and ∥B∥ are their magnitudes.
This returns a similarity score between -1 and 1:
1.0 → Exactly the same
0.0 → Completely unrelated
-1.0 → Opposite meaning (rare in text embeddings)
It compares the embedding against every token in token_embeddings_np.
This results in an array of similarity scores, where each score represents how close embedding is to a given word in token_embeddings_np.
The highest score usually indicates the most likely match.
However, this requires that your tensor data is matched (768, 768) not mismatched, like the data files I have on hand. So I had to create a custom solution to proceed. I cannot vouch for the accuracy of said output with a custom cosine test or using the regular one. But generating some words which may be close to the original embedding is better than nothing. Plus it was a lot of fun and headaches getting the program created.
Below are a few outputs for the files I have as embeddings:
Token Extraction Report
File: cs-cr1stal-v2.safetensors
Recovered Tokens (clip_g):
artillery, hota, ise, shu, replic, scam, rious, dublin
Token Categories (clip_g):
complete_words: ['artillery</w>', 'hota</w>', 'ise</w>', 'shu</w>', 'scam</w>', 'rious</w>', 'dublin</w>']
subwords: ['replic']
special_tokens: []
Recovered Tokens (clip_l):
kili, kani, buxton, oku, atee, filmed, moss, tama
Token Categories (clip_l):
complete_words: ['buxton</w>', 'atee</w>', 'filmed</w>', 'moss</w>', 'tama</w>']
subwords: ['kili', 'kani', 'oku']
special_tokens: []
Embedding Analysis Report
File: cs-cr1stal-v2.safetensors
Extracted Tensors:
['clip_g', 'clip_l']
Metadata:
modelspec.date: 2024-03-01T06:03:49
modelspec.implementation: https://github.com/Stability-AI/generative-models
modelspec.title: cs-cr1stal
modelspec.architecture: stable-diffusion-xl-v1-base/textual-inversion
modelspec.prediction_type: epsilon
modelspec.resolution: 1024x1024
modelspec.sai_model_spec: 1.0.0
Token Influence Scores (clip_g):
0: 41.106262, 1: 44.973324, 2: 41.182266, 3: 45.735714, 4: 49.725018, 5: 41.504845, 6: 38.598206, 7: 41.405197
Token Influence Scores (clip_l):
0: 25.239185, 1: 26.280500, 2: 24.082230, 3: 22.488239, 4: 22.652016, 5: 27.779140, 6: 25.013981, 7: 29.288673
Vector Magnitudes (clip_g):
1.446441, 1.560272, 1.462756, 1.599032, 1.750444, 1.476632, 1.371009, 1.466156
Vector Magnitudes (clip_l):
1.141096, 1.187049, 1.122770, 1.029753, 1.036277, 1.264530, 1.122089, 1.326557
Tensor: clip_g
Shape: (8, 1280)
Mean: -0.000356, Std Dev: 0.042500
Max: 0.172852, Min: -0.200195
Tensor: clip_l
Shape: (8, 768)
Mean: 0.000260, Std Dev: 0.041778
Max: 0.141602, Min: -0.160156
The safetensor embeddings I can create heatmaps and other graphs and the analysis metadata is more complete than .pt metadata so far.
While the .json output for .pt files seems to be more robust, I am having difficulty pulling any useable data from them. For example, below is the first 25 lines from bad-hands-5, but the recovered tokens so far is only "than, kung".
"metadata": {
"name": "bad-hands-5",
"step": 1364,
"sd_checkpoint": "7ab762a7",
"sd_checkpoint_name": "blossom-extract"
},
"string_to_token": {
"*": 265
},
"tensors": {
"*": [
[
0.03990235924720764,
-0.24732241034507751,
0.12515558302402496,
0.04334777966141701,
-0.11283779889345169,
-0.10596635937690735,
0.05618717148900032,
0.15741495788097382,
0.017960386350750923,
0.24233952164649963,
0.03615247830748558,
0.08795509487390518,Negative_hand-neg was even more cryptic with the same word/token used multiple times:
Recovered Tokens (*):
platz</w>, platz</w>, platz</w>, platz</w>, platz</w>, platz</w>, platz</w>, platz</w>
"metadata": {
"name": "negative_hand",
"step": 8000,
"sd_checkpoint": "08d6be750b",
"sd_checkpoint_name": "NegTIModel"
},
"string_to_token": {
"*": 265
},
"tensors": {
"*": [
[
-0.005260467529296875,
0.005260467529296875,
-9.363889694213867e-05,
0.0009665489196777344,
0.00736236572265625,
0.0016450881958007812,
0.01461029052734375,
-0.0013437271118164062,
0.05426025390625,
-0.006092071533203125,
-0.0009241104125976562,
0.00470733642578125,Perhaps some .pt files are better than others at extracting real or imagined words, such as those found for KimberlyNobodySD15:
Recovered Tokens (*):
jur, ãĥĦ</w>, repar, barra, ballin, recession</w>, sleek</w>, mmon</w>, sovereign</w>, tcm</w>, cheek</w>, master, brief</w>, lose</w>, assassin</w>, pedestrian</w>, nigan</w>, insignia</w>, host</w>, nebu, bhafc</w>, poking</w>, any, mming</w>, ðĿĻ, stra</w>, gigi, uses</w>, osborne</w>, recruits</w>, diaper</w>, ckey</w>, meat</w>, wannabe</w>, territories</w>, âĺİï¸ı</w>, scones</w>, imac</w>, blamed</w>
{
"metadata": {
"name": "KimberlyNobodySD15"
},
"string_to_token": {
"*": 265
},
"tensors": {
"*": [
[
0.0002763632801361382,
-0.014794615097343922,
0.003927452024072409,
-0.02636842057108879,
0.013607680797576904,
-0.011484980583190918,
-0.01567922532558441,
0.011889073066413403,
0.003515473334118724,
-0.01937977597117424,
-0.022474249824881554,
-0.01452008169144392,
0.002272453159093857,
-0.016848713159561157,
0.032789696007966995,So that's it basically. I've hit the glass ceiling on work for today but hopefully when all is said and done you can try to get sensible data when or if needed.