@duskfallcrew I understand now thank you 😂

@duskfallcrew Without claim of warranty or guarantee of correctness or understandability, I believe this is how it works: Every model has 2 (or 3 counting VAE) neural nets: text encoder and UNET. Each is made of a metric F-ton of tensors, which are more or less just lists of numbers, i.e., weights. The text encoder is like a look-up table (LUT) where it takes the tokens of the words you put in your prompt and converts them to vectors. So, if you type the word bird as your prompt, the CLIP (or OpenCLIP for SD2), the language module which does that conversion, takes the word, breaks it up into tokens (and there should be only one token, bird), and finds it in the LUT, then returns the vectors, which might be something like "182, 221, 28". Then, those numbers are passed to the UNET. After the UNET waves a dead chicken over the latent space, an image of pure noise is refined into one of a bird. Also, there is an a1111 extension to take a prompt, break it into tokens, and then show you the vectors that correspond to each token.

If you think of each neural net as being like a fishnet, and you stretch it out in a plane (but not too tight or this illustration won't work), and then you put a baseball on it, wherever the baseball is, the coordinates on that net will have a weight corresponding to the weight the baseball is exerting on the net and causing it to indent. If you put a baseball on the net, it's going to have more weight than something like an empty can, and so gravity will pull the net down farther where the baseball is than where the empty can is. With me so far? Can you see how if you arrange a bunch of objects on the net, you can recreate a picture of those objects by recording how much the net is pulled down by gravity in each location? (If you have difficulty understanding that part, go play this number game and you'll get it.)

Now, back to the text encoder and UNET. Everything up to this point is sort of how I conceptualize it in my brain based on what I've read about the architecture of SD, but this is just a reiteration of something that I read once: An embedding will adjust the weights of the text encoder only, which is why you can't use a TI/embedding to teach new data to a model. So if bird normally returns (182 221 28), and you use an embedding for mockingbirds, it's going to return vectors that are a little or a lot closer to the vectors for the prompt mockingbird. Hypernets (remember them? lol) do the same thing, only for the UNET. And LoRAs (and their derivatives, viz., LoHA, LoCon, LECO) alter the weights in both the TEnc and the UNET.

In essence, what the embedding merge extension does is... if you have a prompt, (red:0.8) fox eating (fried:1.2) tofu, and it gets converted (I'm using symbols here because I don't want to look or make up numbers) (A:0.8) B CD (E:1.2) F, you can put all that together by using math. If the vector A is something like (26 88 3) and the vector B is (12 7 35), then you can combine those two vectors—which are just numerical representations of tokens—by adding them after you adjust the weighting. Since A is at weight 0.8, we'd need to multiply each number in the vector by 0.8, to get (20.8 70.4 2.4). So we add that to the value for B, and we get (32.8 77.4 37.4). We can also add on all the other vectors too, after adjusting for the weight of E. Now, instead of having to go through and enter (red:0.8) fox eats (fried:1.2) tofu, we can simply use the name of the new embedding merge we created.

This also saves a bit of time during creation of the image, since all the math is done ahead of time instead of going through and looking up all those vectors and then following them one after another like the directions of a treasure map ("First go north 5 steps. Then go east 3 steps. Then go south 8 steps. Then go west 7 steps."), we can go straight to the location of the "treasure" ("Go southwest 5 steps.").

If any of that doesn't make sense, I apologize, but that's the best way I can explain it. As I said, there's no warranty or guarantee of correctness or understandability. 😜 Maybe someone else can explain it in a way that is easier to understand.

@duskfallcrew what tool do you use as when I search for embedmerge the results are for supermerger

@dita thank you for the explanation :)