My attempt at a straightforward workflow centered on the following custom nodes:
Segmentation results can be manually corrected if automatic masking result leaves more to be desired.
Crop & Stitch allows for inpainted area to be gracefully upscaled to SDXL optimal resolutions for sampling which is then blended back into the original image.
Note that because only the masked region undergoes denoising, its color will not always exactly match the full original image. Passing the image through another sampler on low denoise or using post-processing custom nodes/external image editing may help. For extreme cases image harmonization techniques could be used, though I have personally never needed to.
I'd recommend using ComfyUI-Manager for installing and updating any other custom nodes used within the workflow.