Introduction

In the appendix of my previous article, I discussed techniques for multi-level image description. Recently, I implemented this technique using Gemini 2.0 and applied it to batch-generate image descriptions for training models and LoRAs. This paper presents the findings and shares the approach.

Why Multi-Level Descriptions Are Necessary ?

Before delving into the details, it is pertinent to explain the rationale behind adopting multi-level image descriptions. The quality of image generation in Diffusion models is primarily influenced by three factors:

1. Model architecture

2. Dataset used for training

3. Input descriptions provided during image generation

First, the impact of model architecture is evident. Comparing outputs from models like FLUX and SDXL reveals that architectural differences and the scale of parameter weights significantly affect image quality. The most direct approach to enhancing quality involves modifying the architecture—such as integrating plugins into specific layers or employing auxiliary components like embeddings or LoRAs. However, this method poses a high technical barrier; while training LoRAs has become more accessible, fine-tuning specific layers remains programmatically challenging.

Second, the dataset’s role is equally clear: a richer prior knowledge base, derived from a larger and more diverse dataset, yields more varied outputs.

Third, the input description merits particular attention. In both image generation and model training, descriptions must align closely with the images, as this correspondence directly governs generation performance. For anime models, descriptions are typically derived from Danbooru tagger models, which rely on a finite set of Danbooru tokens. This limitation results in prompts that fail to capture all image details. A solution is to combine natural language with Danbooru tags, creating multi-level descriptions that leverage natural language to address concepts beyond the scope of Danbooru tags. Testing further revealed that images generated solely with Danbooru tags appear "simple", whereas those using multi-level descriptions exhibit more sophisticated composition and content representation. The following example illustrates this effect:

In conclusion, I posit that training models with multi-level image descriptions could enhance generation performance. Accordingly, I have explored the application of multi-level descriptions to images.

Implementation

The multi-level image description method proposed in this article is implemented using Gemini 2.0. Although Grok 3 was recently released, testing revealed its safety restrictions prevent the input of NSFW images. In contrast, Gemini 2.0 allows safety filters to be disabled and supports structured JSON output, making it more suitable. Consequently, Gemini 2.0 was selected for generating natural language descriptions and integrating Danbooru tags to produce multi-level image descriptions.

The batch generation of multi-level descriptions is achieved by invoking the Gemini API, with details provided in the Gemini API documentation. The process begins by specifying the image storage directory, retrieving paths to images and corresponding text files containing Danbooru tags, and configuring the Gemini LLM model. An API key must be obtained and supplied as follows:

Next, a structured output format must be defined to facilitate the batch processing of key textual content:

For natural language captions, the LLM generates output in the following JSON format: {'caption': 'caption of image'}. For the final tag output, the LLM produces: {'tags': ['tag1', 'tag2', 'tag3', ...]}. This structure streamlines image description handling across each processing step.

The generation of multi-level image descriptions follows three steps:

1. Obtain natural language descriptions for the images.

2. Integrate natural language descriptions with Danbooru tags.

3. Write the final multi-level tags to text files corresponding to the images.

These steps are implemented with the following code:

After processing with the above code, the multi-level descriptions for each image are saved to text files, facilitating their use in subsequent model training.

Conclusion

This article outlines a method for batch-generating multi-level image descriptions using Gemini. The code files are available in the Attachments and may prove useful to readers. The alignment between the generated multi-level descriptions and their corresponding images has not yet been evaluated, though this analysis may be conducted in the future. A current limitation is the prolonged runtime, a common issue with LLM models. However, this approach executes relatively quickly compared to local LLM inference, and performance could be further improved using PyTorch’s DataLoader.