Update the plugin to latest version to avoid incorrect placement during large buffer and close to edge mask
Comfyui-In-Context-Lora-Utils (20241214)
v3.4 update:
Use kontext to improve workflow
v3.3g update:
a simple workflow for dev ic lora using new nodes.
v3.3f update:
Fix missing connection on crop mask to original result
Usage:
Input two images
mask the target area from image 1 as reference
mask the target area from image 2 as editing area
Parameter:
https://civarchive.com/articles/9292
中文视频讲解：
https://www.bilibili.com/video/BV19MiZYJE82/?vd_source=1f3b87e4654ce717c5f87bb51a78ffc0
Sponsors me for more open source project:
Buy me a coffee:
v3.3 update:
Add auto mask before final output.
Move parameter above load image section for more space.
Adjust fit image logic, update logic to avoid cropping image.
Adjust plugin calculation to avoid incorrect replacement.
v3.2 update:
add prompt and redux switch.
add one step refine.
Update the latest plugin, 2024-12-05, for v3.2: Comfyui-In-Context-Lora-Utils
v3.1 update:
Add AutoPatch Node to avoid patch mode and patch type selection.
Fix incorrect width height in edge case which lead to incorrect resize and incorrect replace in final result
v3.0 update: To adjust lossy detail issue, I develop new nodes for this context window editing workflow.
Download the latest plugin for v3.0: Comfyui-In-Context-Lora-Utils
v3.0 workflow enable any context window editing with provided mask and image.
It could create very small context window from source image and replace it back to the original image.
v2.4 update: Based on another person created workflow, improve image process logic with my Comfyui-In-Context-Lora-Utils
PS: demo image is created by 书墨旅人 from a wechat group.
v2.3 update: Two pass sampler to increase the consistency
v2.2 update: Fix missing flux guidance issue
v2.1 update: Remove Lora from the workflow
v2.0 update: Add Target Image And Accept Mask Input
PS: Need the latest Comfyui and newly release inpainting flux model
PS2: You might need to adjust seed a few time to get the best result
Download Inpainting Flux Model (Fill) Here:
https://huggingface.co/black-forest-labs/FLUX.1-Fill-dev/tree/main
V2.0 extention:
https://github.com/lrzjason/Comfyui-In-Context-Lora-Utils
## How to install
Download the zip file.
Extract to ..\ComfyUI\custom_nodes.
Restart ComfyUIFor try on lora, you could download from below:
https://civarchive.com/models/950111/flux-simple-try-on-in-context-lora
v1.3 update: added new plugin for simplify in context lora workflow
v1.2 update add more description and tips
add Ratio Based on 1024 for high resolution generation
The remove background node need the ckpt from below url
https://github.com/chflame163/ComfyUI_LayerStyle?tab=readme-ov-file#TransparentBackgroundUltra
As newly discoved in context lora usage of Flux.
https://ali-vilab.github.io/In-Context-LoRA-Page/
We could use image as condition to generate similar output.
I introduce this workflow to simply remove the ic-lighting and inpainting models for product background editing.
Please use with this lora:
https://civarchive.com/models/933026/flux-product-design-in-context-lora
Modify the Main Description to descript the input image.
Modify the Extra Description to descript the wanted image. You could refer [IMAGE1] in Extra Description.
For example:
Main Description: This is a picture of a gold pendant. The design of the pendant is very unique, shaped like a pair of spread wings with a heart-shaped hollow in the middle. The whole pendant presents a delicate and elegant feeling, suitable for use as a necklace or bracelet ornament.
Extra Description: A beatiful woman wearing [IMAGE1] walking on street
Feel free to dm me if you have any query.
Contact: [email protected]
QQ Group: 866612947
Description
FAQ
Comments (19)
It seems that your workflow did not use In-Context lora anymore from > 2 version. But I want to say redux like ipadapter, but In-Context lora has greater possibility in many case. Redux is just copy. So hope you can create workflow using INC lora.
Yes, you are right. A dedicated IC lora could do a great job on its task. It could migrate an abstract object into another scene after training. For example, reference is a cup cake with IC Lora, it could transform into a dress with cup cake element. It is quite hard for redux. But currently, it is leak of tools to train fill model and IC lora trained on dev model is not working very well on fill model.
Using IC lora, you could just use the latest workflow and modify a bit by adding lora loader and change the model to dev. Then, change the guidance type to prompt. You could use the IC Lora. (you might want to add emptyImage and its related parameters.
@xiaozhijason Can you please provide a workflow using the latest version but also running IC lora? I want to be able to change the material of furniture in a photo without losing the shape of the furniture, using a provided reference material image. Right now it completely remakes the furniture without keeping the original details
@xiaozhijason Can you explain in a bit more detail how to apply IC lora to the latest version? I would like to use it to adjust furniture in a room by giving an example image
@tggbot1155 v3.3g
It's great, but after adding sam, it's not so flexible to use
You could disable sam section and use manual mask which is no difference than old version
lol, for me its the opposite, sam working fine, and masking not. I'm using for adding some public seating bench to another image where there is nothing, just pavement.
Hey I got this error and I cant seem to see which version is the issue:
Error in image processing: Error in batch processing: Could not run 'torchvision::deform_conv2d' with arguments from the 'CUDA' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'torchvision::deform_conv2d' is only available for these backends: [CPU, Meta, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradHIP, AutogradXLA, AutogradMPS, AutogradIPU, AutogradXPU, AutogradHPU, AutogradVE, AutogradLazy, AutogradMTIA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, AutogradMeta, AutogradNestedTensor, Tracer, AutocastCPU, AutocastXPU, AutocastCUDA, FuncTorchBatched, BatchedNestedTensor, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PreDispatch, PythonDispatcher]. CPU: registered at C:\actions-runner\_work\vision\vision\pytorch\vision\torchvision\csrc\ops\cpu\deform_conv2d_kernel.cpp:1162 [kernel] Meta: registered at /dev/null:19 [kernel] BackendSelect: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\BackendSelectFallbackKernel.cpp:3 [backend fallback] Python: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\PythonFallbackKernel.cpp:153 [backend fallback] FuncTorchDynamicLayerBackMode: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\functorch\DynamicLayer.cpp:497 [backend fallback] Functionalize: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\FunctionalizeFallbackKernel.cpp:349 [backend fallback] Named: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\NamedRegistrations.cpp:7 [backend fallback] Conjugate: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\ConjugateFallback.cpp:17 [backend fallback] Negative: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\NegateFallback.cpp:18 [backend fallback] ZeroTensor: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\ZeroTensorFallback.cpp:86 [backend fallback] ADInplaceOrView: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\VariableFallbackKernel.cpp:86 [backend fallback] AutogradOther: registered at C:\actions-runner\_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autograd\deform_conv2d_kernel.cpp:256 [autograd kernel] AutogradCPU: registered at C:\actions-runner\_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autograd\deform_conv2d_kernel.cpp:256 [autograd kernel] AutogradCUDA: registered at C:\actions-runner\_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autograd\deform_conv2d_kernel.cpp:256 [autograd kernel] AutogradHIP: registered at C:\actions-runner\_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autograd\deform_conv2d_kernel.cpp:256 [autograd kernel] AutogradXLA: registered at C:\actions-runner\_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autograd\deform_conv2d_kernel.cpp:256 [autograd kernel] AutogradMPS: registered at C:\actions-runner\_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autograd\deform_conv2d_kernel.cpp:256 [autograd kernel] AutogradIPU: registered at C:\actions-runner\_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autograd\deform_conv2d_kernel.cpp:256 [autograd kernel] AutogradXPU: registered at C:\actions-runner\_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autograd\deform_conv2d_kernel.cpp:256 [autograd kernel] AutogradHPU: registered at C:\actions-runner\_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autograd\deform_conv2d_kernel.cpp:256 [autograd kernel] AutogradVE: registered at C:\actions-runner\_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autograd\deform_conv2d_kernel.cpp:256 [autograd kernel] AutogradLazy: registered at C:\actions-runner\_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autograd\deform_conv2d_kernel.cpp:256 [autograd kernel] AutogradMTIA: registered at C:\actions-runner\_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autograd\deform_conv2d_kernel.cpp:256 [autograd kernel] AutogradPrivateUse1: registered at C:\actions-runner\_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autograd\deform_conv2d_kernel.cpp:256 [autograd kernel] AutogradPrivateUse2: registered at C:\actions-runner\_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autograd\deform_conv2d_kernel.cpp:256 [autograd kernel] AutogradPrivateUse3: registered at C:\actions-runner\_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autograd\deform_conv2d_kernel.cpp:256 [autograd kernel] AutogradMeta: registered at C:\actions-runner\_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autograd\deform_conv2d_kernel.cpp:256 [autograd kernel] AutogradNestedTensor: registered at C:\actions-runner\_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autograd\deform_conv2d_kernel.cpp:256 [autograd kernel] Tracer: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\autograd\TraceTypeManual.cpp:297 [backend fallback] AutocastCPU: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\autocast_mode.cpp:209 [backend fallback] AutocastXPU: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\autocast_mode.cpp:351 [backend fallback] AutocastCUDA: registered at C:\actions-runner\_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autocast\deform_conv2d_kernel.cpp:48 [kernel] FuncTorchBatched: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\functorch\LegacyBatchingRegistrations.cpp:731 [backend fallback] BatchedNestedTensor: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\functorch\LegacyBatchingRegistrations.cpp:758 [backend fallback] FuncTorchVmapMode: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\functorch\VmapModeRegistrations.cpp:27 [backend fallback] Batched: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\LegacyBatchingRegistrations.cpp:1075 [backend fallback] VmapMode: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\VmapModeRegistrations.cpp:33 [backend fallback] FuncTorchGradWrapper: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\functorch\TensorWrapper.cpp:207 [backend fallback] PythonTLSSnapshot: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\PythonFallbackKernel.cpp:161 [backend fallback] FuncTorchDynamicLayerFrontMode: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\functorch\DynamicLayer.cpp:493 [backend fallback] PreDispatch: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\PythonFallbackKernel.cpp:165 [backend fallback] PythonDispatcher: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\PythonFallbackKernel.cpp:157 [backend fallback]
You might try to reinstall torch and related library
Great workflow! However, I've noticed that your generated images capture intricate details from the source clothing image with remarkable accuracy. In contrast, my generated images seem to lack these details. Could you recommend any specific settings or adjustments to help me achieve similar results with the same level of quality?
Could you upload your output which lack details? I don't really know which part is making wrong result of your refence and target. Also, could you upload your reference and target seperately for me to test?
hello I would like to know why the end image is always different from the entry image if it has any settings please tell me
for example when I put on a t-shirt the workflow also changes the pants except I just enter a t-shirt image
It depands on many things. First, if you are using SAM mask, you should change the prompt to t-shirt which could select the t-shirt as a mask. Then, you could adjust the final image selection to 2 which use the original mask. It could replace the original masked area rather than an auto selected mask using remove background.
thank you !
@xiaozhijason I tried and it works thank you very much and if I want to change the background or a logo it's the same procedure
Is there any way we can also replicate the texture of fabric weave in image generation?
just put the texture image in the reference image