Clean up your Image Datasets with this Tool

Do you have duplicate directories of image datasets that are slightly different, and now you have to do some clean up work? It can be a hassle to compare images from both directories side by side and then open up all of the caption files to compare them too. This tool, Dataset Dedupe,  helps you compare the images and captions from both directories side by side based on the file base name. There is no need to upload your files to a server. It operates on your local file system and it's open source.

You can delete duplicate images, copy them over to the other directory. Update your captions, copy those captions too, and paste those captions to multiple images if needed.

It also integrates with the VLM Caption Server project so you can have AI caption the images and select the prompt to be detailed, short, or tags.