<h1 id="image-to-prompt-and-smart-detection-workflow-guide">Image-to-Prompt &amp; Smart Detection — Workflow Guide</h1>38 nodes · 7 groups · 16 unique node types — 92.1% Eclipse nodes<ul><li>35 Eclipse nodes at the top-level</li><li>Built with <a target="_blank" rel="ugc" href="https://github.com/r-vage/ComfyUI_Eclipse">ComfyUI_Eclipse</a> custom nodes</li></ul><hr /><h2 id="what-is-this">What Is This?</h2>This workflow is a comprehensive, modular playground for Image-to-Prompt (VLM) generation and Smart Detection built entirely with the ComfyUI_Eclipse custom node suite.It provides an interactive interface for:<ul><li>Loading single images or batching directories of files (with video frame extraction support).</li><li>Interactively selecting frames/images on the fly in the frontend using the <code>Image Selector</code>.</li><li>Running object detection, segmentation, and bounding-box labels using the <code>Smart Detection</code> node.</li><li>Querying multiple vision-language models (VLMs) and text-only LLMs concurrently using the unified <code>Smart LM Loader</code> (Smart LML).</li><li>Generating prompt variations, timeline structures (for video generators like Wan2.1 and LTX), and song lyrics.</li><li>Direct interactive chat with custom system prompts and dynamic routing.</li></ul>The workspace is designed for modularity and wireless routing: groups are self-contained and communicate using <code>Set/Get</code> named channels, meaning you can toggle groups on and off via muting without breaking any visual wiring.<hr /><h2 id="unified-smart-loaders-and-backends">🌒 Unified Smart Loaders &amp; Backends</h2>A key strength of the ComfyUI_Eclipse suite is consolidating fragmented model wrappers and APIs into two primary smart loaders. Instead of installing and configuring a dozen different node packs for Hugging Face, GGUF, Ollama, vLLM, and Docker servers, everything is unified under a standard input interface.<h3 id="1.-smart-language-model-loader-(smart-lml)">1. Smart Language Model Loader (Smart LML)</h3><code>Smart LM Loader [Eclipse]</code> serves as the single entrypoint for LLMs, VLMs, and ONNX taggers. It natively handles prompt templates, system instructions, and advanced sampling (temperature, top_p, min_p, Mirostat, repetition penalties).It supports 8 distinct backends selectable directly from the node:<ul><li>Transformers (Native local Hugging Face execution): Standard local VLM runs (Florence-2, Qwen2-VL, Pixtral) and text-only models. Supports FP16, BF16, and INT8/INT4 quantization.</li><li>GGUF (Local quantized weights via <code>llama-cpp-python</code>): Running large models locally on lower-VRAM consumer GPUs with high quant efficiency (e.g. Q8_0, Q4_K_M).</li><li>Ollama (API interface / Local Docker container): Offloading computation to a background Ollama daemon or remote Ollama server.</li><li>llama.cpp (Local Docker-based server engine): Standardized containerized CPU/GPU inference.</li><li>vLLM (Docker) (High-throughput Docker container): Offloading inference to a vLLM server container for maximum speed and structured batch requests.</li><li>vLLM (Native) (Native local Python library): Fast local inference on Linux systems with native vLLM installed.</li><li>SGLang (Docker) (High-performance container engine): Rapid structured text generation and blazing-fast decoding.</li><li>WD14 Tagger (Local ONNX-based taggers): Running aesthetic classifier models (like SwinV2 or ConvNeXt) to auto-tag anime/general images locally.</li></ul><h3 id="2.-smart-detection">2. Smart Detection</h3><code>Smart Detection [Eclipse]</code> is a unified object detection, text grounding, and segmentation node. It acts as an adapter that unifies two major model families under a single pipeline:<ul><li>Vision-Language Models (VLMs): Uses Florence-2 or Qwen-VL to run text-grounding queries (e.g., "detect face, hair, blue dress") and converts textual coordinate outputs into binary masks, cropped bounding boxes, and ComfyUI Impact Pack-compatible SEGS.</li><li>YOLO Models: Loads YOLO v8, v9, v10, and v11 models for high-speed object detection and precise facial/segment segmentation.</li></ul><h3 id="3.-smart-model-loader-(main)">3. Smart Model Loader (Main)</h3>Used in standard rendering workflows (like iGEN ONE) but registered under the same suite. It loads standard Checkpoints, UNets, CLIP text encoders, VAEs, and LoRAs. It contains a template dictionary matching filenames to expected SHA256 hashes and CivitAI AIRs. If a file is missing, it displays an interactive Download button to automatically fetch it directly into your ComfyUI models directory.<h3 id="adding-custom-models-to-the-registry">Adding Custom Models to the Registry</h3>To add your own custom models to the Smart LM Loader dropdowns, you should edit the local user registry file:<ul><li>File Path: [registry/user_models.json](file:///mnt/data/AI/custom_nodes/comfyui_eclipse/registry/user_models.json)</li></ul>Insert your model details under the appropriate backend key (e.g. <code>"transformers"</code>, <code>"gguf"</code>, <code>"ollama"</code>):<pre><code>{
 "transformers": {
 "My-Custom-VLM-Name": {
 "repo_id": "username/repo-name",
 "family": "VLM",
 "has_vision": true
 }
 },
 "gguf": {
 "My-Custom-GGUF-Name": {
 "repo_id": "username/repo-name-GGUF",
 "family": "LLM_TEXT",
 "has_vision": false,
 "quantizations": ["Q4_K_M", "Q8_0"],
 "file_pattern": "model-name-{quant}.gguf"
 }
 }
}
</code></pre>IMPORTANT: Model names starting with an underscore (e.g. <code>_example_Phi-4</code>) are treated as examples and are ignored. Ensure your custom entry keys do not have a leading underscore.<hr /><h2 id="how-it-works-the-basics">How It Works — The Basics</h2><h3 id="wireless-routing-(setget)">Wireless Routing (Set/Get)</h3>Rather than cluttering the canvas with messy visual noodles, the workflow uses Set/Get nodes:<ul><li>SetNode: Publishes the loaded image as a named stream <code>REF_IMAGE</code>.</li><li>GetNode: Subscribers inside the Detection, Image to Prompt, and Timeline Prompts groups retrieve this reference image wirelessly.</li></ul><h3 id="mode-bridges-(single-upload-vs.-directory-batching)">Mode Bridges (Single Upload vs. Directory Batching)</h3>The workflow includes a smart toggling mechanism for image sourcing:<ul><li>Load Image (Metadata Pipe): For drag-and-dropping individual files from your browser/computer.</li><li>Load Batch From Folder: For batching directories of images or decoding frames directly from video files.</li><li>Mode Bridge Set &amp; Get: Wireless switches that communicate which loading path is active. When you toggle the active path, the <code>Any Multi-Switch</code> routes the correct image stream to <code>Set_REF_IMAGE</code> automatically.</li></ul><hr /><h2 id="group-by-group-reference">Group-by-Group Reference</h2><h3 id="1.-load-image">1. Load Image</h3>The entry point of the workflow. Exposes two loading methods:<ul><li>Single Image: [Load Image (Metadata Pipe) [Eclipse]](file:///mnt/data/AI/custom_nodes/comfyui_eclipse/py/RvImage_LoadImage_Pipe.py).</li><li>Batch / Video: [Load Batch From Folder [Eclipse]](file:///mnt/data/AI/custom_nodes/comfyui_eclipse/py/RvImage_LoadBatchFromFolder.py). Supports listing folders or decoding MP4/MKV video files frame-by-frame.</li><li>Interactive Filtering: [Image Selector [Eclipse]](file:///mnt/data/AI/custom_nodes/comfyui_eclipse/py/RvImage_Selector.py). When batch loading a folder or decoding video, this node pauses the ComfyUI run and renders an interactive grid of thumbnails. You click the frames you want (supports shift-click range and text filtering), click "Confirm", and it resumes to output only that selected subset.</li><li>Routing: Uses two <code>Mode Bridge Get</code> and <code>Mode Bridge Set</code> nodes to govern whether single or batch mode is selected. The active image goes to <code>Set_REF_IMAGE</code>.</li></ul><h3 id="2.-detection">2. Detection</h3>Performs visual segmentation and object locating:<ul><li>Subscribes to <code>REF_IMAGE</code> via a <code>GetNode</code>.</li><li>Smart Detection: Uses a model (defaulting to Florence-2) to detect coordinates or segment structures.</li><li>Outputs: Renders the isolated region to <code>Preview Image (DOM)</code>, shows the binary segmentation mask in <code>Preview Mask</code>, and dumps textual results (like bounding box text or OCR outputs) to <code>Show Text</code>.</li></ul><h3 id="3.-image-to-prompt">3. Image to Prompt</h3>Generates detailed textual descriptions from your image:<ul><li>Uses two parallel <code>Smart LM Loader</code> nodes loaded with vision-language models (e.g. Qwen2-VL-7B-Instruct or Florence-2).NOTE: This parallel setup with two separate nodes is for showcase/demonstration purposes only. In a typical production workflow, multiple sequential operations like captioning and tag generation can be executed inside a single loader node using its built-in multi-task system.</li><li>Executes tasks like creating detailed descriptive prompts or extracting comma-separated tagging keys.</li><li>Outputs prompts to two separate <code>Show Text</code> boxes.</li></ul><h3 id="4.-timeline-prompts-(wan-ltx)">4. Timeline Prompts (Wan / LTX)</h3>A dedicated VLM generator for text-to-video prompt engineering:<ul><li>Uses <code>Smart LM Loader</code> (VLM mode) to analyze <code>REF_IMAGE</code>.</li><li>Generates structured timeline descriptions (e.g. "0s: subject starts sitting; 2s: subject turns head") tailored for video models like Wan2.1 and LTX-Video.</li><li>Dumps the structured timelines to a <code>Show Text</code> box.</li></ul><h3 id="5.-prompt-variations">5. Prompt Variations</h3>An LLM generator to brainstorm prompts:<ul><li>Uses <code>Smart LM Loader</code> (text-only mode) to take a simple seed prompt (e.g. "a futuristic cyberpunk street") and expand it into 3 creative prompt variations.</li><li>Dumps the results to a <code>Show Text</code> box.</li></ul><h3 id="6.-song-lyrics">6. Song Lyrics</h3>A creative text generation playground:<ul><li>Uses <code>Smart LM Loader</code> (text-only mode) configured to compose structured song lyrics or poetry based on user-supplied themes.</li><li>Dumps the lyrics to a <code>Show Text</code> box.</li></ul><h3 id="7.-direct-chat">7. Direct Chat</h3>A general playground to communicate directly with your loaded LLM or VLM:<ul><li>Exposes three pre-configured system prompts using [String Multiline [Eclipse]](file:///mnt/data/AI/custom_nodes/comfyui_eclipse/py/RvText_Multiline.py) nodes.</li><li>Any Multi-Switch: Routes one of the three system prompts to the loader based on your selection (V1, V2, or V3).</li><li>Smart LM Loader: Executes the conversation.</li><li>Fast Mode Toggle: Lets you customize model options (device, compile, temperature) via simple frontend buttons.</li><li>Show Text: Displays the direct chat responses.</li></ul><hr /><h2 id="quick-start-guide">Quick Start Guide</h2>TIP: Ensure your Hugging Face, GGUF, or Ollama server credentials are configured in <code>config.json</code> if using Docker-based backends. Local GGUF and Transformers backends run out-of-the-box using models placed in <code>models/llms/</code> or <code>models/text_encoders/</code>.<h3 id="how-to-get-captions-from-an-uploaded-image">How to Get Captions from an Uploaded Image</h3><ol><li>Drag and drop your image into the Load Image node in the Load Image group.</li><li>In the Image to Prompt group, select your preferred VLM from the dropdown on the <code>Smart LM Loader</code> (e.g. <code>Florence-2-large</code>).</li><li>Click Queue Prompt.</li><li>The generated prompt and tags will appear in the <code>Show Text</code> nodes inside the group.</li></ol><h3 id="how-to-detect-and-mask-objects-(e.g.-facehair)">How to Detect and Mask Objects (e.g., Face/Hair)</h3><ol><li>Ensure your image is loaded in Load Image.</li><li>Locate the Detection group and enter your search query in the <code>detection_prompt</code> widget of the <code>Smart Detection</code> node (e.g. "face, hair").</li><li>Click Queue Prompt.</li><li>The node will locate the facial/hair boundaries, isolate them, and display the result in the <code>Preview Image (DOM)</code> and the binary mask in the <code>Preview Mask</code> node.</li></ol><h3 id="how-to-run-batch-image-tagging">How to Run Batch Image Tagging</h3><ol><li>In the Load Image group, set the bridge settings to Folder Mode.</li><li>Enter the path to your folder in the <code>directory</code> widget of <code>Load Batch From Folder [Eclipse]</code>.</li><li>Set the <code>Image Selector</code> to bypass if you want to automatically process all images in sequence, or leave it active to manually select files when execution pauses.</li><li>Run the queue. The workflow will process each file sequentially through the active groups.</li></ol><hr /><h2 id="custom-node-packages-used">Custom Node Packages Used</h2><ul><li><a target="_blank" rel="ugc" href="https://github.com/r-vage/ComfyUI_Eclipse">ComfyUI_Eclipse</a> — Handles unified smart loaders (<code>Smart LM Loader</code>, <code>Smart Detection</code>), directory list-loading (<code>Load Batch From Folder</code>), visual filtering (<code>Image Selector</code>), wireless routing (<code>Set/Get</code>), and text previews (<code>Show Text</code>, <code>Preview Image (DOM)</code>, <code>Preview Mask</code>).</li></ul><hr />Explore, test, and build beautiful language-guided pipelines. 🌒

Detection / Image to Prompt

Updated for the changes in Eclipse 4.1.15 because there is a breaking change in the smart lm loader

Files with SHA256: 1bd6f36a3e9e25bcf2bc582caa5706bbe19a1fc37a65f06c33ae4b445a8d59b8