Joy Caption Standalone
NF4 Model works extremely fast even on low VRAM Cards
Easy to use standalone GUI and model loading (No HuggingFace)
Word replacement can replace vagina with pussy or labia minora with inner pussy lips for PONY models (Optional)
Banned words can be omitted (Optional)
Automatically creates sized png (No latent errors from bad jpg) just select the training size eg: 512, 1024, 1536 or any other size. (Aspect Ratio is kept)
Updated bat file to create a VENV (You can also use Comfy UI or Forge VENV)
Tip in the user defined caption "Write a long detailed description for this image with some WD14 tags." or "Write a long detailed description for this image with some Booru-like tags."
METADATA UPDATE
This is an update of the base script to parse existing metadata from images. Joy caption can intelligently use this information.
Logic is in the script to append the user prompt with the correct phrase to use metadata. DO NOT write anything in the user prompt, the script handles it.
It is set to ignore 10 characters or less and crop to 200 if the length is longer then that to avoid truncation. This should not be an issue if the use 77 words or less is added to the user prompt.
Names of animals and people
Locations
Film Exposure
Etc
Getty Images is an excellent source of accurate metadata.
Description
FAQ
Comments (38)
I was giving me a "ModuleNotFoundError: No module named piexif" error.
I just activated the venv through cmd and installed piexif manually with pip, now it runs fine. In case anyone else has a similar issue.
you can just write in requirement: piexif
juxotexu123 Yeah, I thought of that after the fact. It was an easy fix either way.
Thank you both ♥♥♥♥
I've spent about four hours trying to get this to work off line. It keeps trying to go to Huggyface. I've tried a number of work arounds but no luck. Any guidance?
I did not update the requirements to include piexif when I added metadata but HF is not part of the script. It loads from ./model
This works great! Thanks for making it and sharing it with us!
Can any one explain the steps, where & how to use it
can someone please explain why this app resizes images for the max instead of the min dimension? isn't it the other way around for training for bucket sizes? for 1024 based models, you dont want less than 1024, so why does this app do the opposite?
thanks ahead of time. also, is there a way to skip the image resizing in this app?
So it does two functions one is to ensure that the size works with joy, the other is to ensure the png is compatible with training. It is not setup to skip but you can set any size you wish and keep the aspect ratio.
If you put 2048 and have a 2048x1536 image it stays at that size without change
@Felldude thanks for getting back to me fell! and how about the min vs max dimension? most models seems to want to be trained at min 1024, so setting the max dimension doesn't seem useful to me, but maybe ive been doing this wrong all along
@olivereads38255 If the size is 768x1280 you would set the max dimension to 1280 and it would stay 768x1280 it will always do the math on the larger dimension, but keep the aspect ratio
@Felldude yes, but wouldn't you rather want to set the minimum to 1024 instead? that way for training you wouldn't have to keep checking the smaller dimension for each image meeting the min requirement
@olivereads38255 I prefer to know what the maximum dimension of all the images will be. If I have a 4k by 512 image (I would not train on that) but I would rather have it go to 128x1024 then 1024x8k
@Felldude i think i understand now, so you'd rather risk some quality loss in the dataset rather than overwhelm the training on massive outliers, is that correct?
@olivereads38255 I still don't think your following, if the image is 2048x2048 and I set the max to 1536px then the image will be 1536x1536 - If all your images are 1024 and the size is set to 1024 they will just be converted to png and set to RGB-8 no alpha
@Felldude i understand it batches by resizing by the larger dimension, but im curious what your reasoning is to 'rather have it go to 128x1024 then 1024x8k'.
is that pretty much to speed up joy caption processing, specifically? because for the rest of the lora training, id assume youd want the higher resolutions whenever possible
@olivereads38255 You have three separate things at work, 1: the vision model this center crops the image to 384x384. This is done no matter what the image size is and does not effect the output 2: The output size of the caption software, which is a safety step to ensure the max dimension not the minimum and it keeps the ratio so this is moot. 3: The latent training size, very very few models benefit from 2048x2048 training. The tensor size is simply not large enough.
@Felldude ooh! it knew something wasnt adding up, thank you for explaining this! so for your personal datasets, what maximum dimensions do you usually set for training flux and wan loras (and i assume images are different than videos)?
@olivereads38255 It is dependent on the dataset, and tensor size, batch and how variant the data is. When first training a model they usually iterate up from 128x128 to 1024x1024 while decreasing batch size. Imagine trying to chisel in skin pore data into a cube, you might want to get the base shape first
@Felldude damn, i guess overestimated how large images should be for datasets. i assumed they wanted 1024 as a minimum for even the smaller dimension of images in the dataset, ever since sdxl. how do i find what tensor size im working with?
@olivereads38255 I would look it up, when dealing with video models the tensor is 5D including the time in the latent
Great job, i was fiddling with the idea for a while, i am glad i stumbled upon this.
Thanks
Fast & easy to use
Thanks for sharing ♥♥♥
Thanks, I am hopping to do a mistral version that doesn't require Linux, but it will likely require flash attention or VLLM to fit in 16GB
This tool is not GPU agnostic. Attempting to run this on AMD-family ROCm cards will throw errors relating to CUDA dependencies built into the tool.
Likely the bitsandbytes nf4 - if you have an AMD card (24GB) that can use the full version, ask gpt to point to fancyfeast/llama-joycaption-beta-one-hf-llava and remove any cuda dependency
I was giving me an "AttributeError: 'LlavaForConditionalGeneration' object has no attribute 'vision_tower'" error, how can I fix this?
Thats the new transformers version not exposing layers - downgrade to 4.36.0
I get this error: ModuleNotFoundError: No module named 'piexif'. Any clues please?
pip install piexif or add piexif to the requirements .txt
Thanks. Did that. Now this: AttributeError: 'LlavaForConditionalGeneration' object has no attribute 'vision_tower'
@stbeerystudios662 That is the new transformers causing the issue, I need to set the requirement file to be 4.6.0 not 5
In case anyone is still struggling with this, open you requirements.txt file and replace all the text with this: --extra-index-url https://download.pytorch.org/whl/cu128
torch
torchvision
torchaudio
transformers==4.49.0
accelerate
bitsandbytes
piexif
Pillow
good