Controlnets for image refinement

[fxlab]

Heya all, I'm trying out Krita Ai diffusion to refine storyboards into colorscript. My process currently is to clean up the storyboard frame so there is no hatching as I found that works better with Scribble controlnet for the first iteration of the process.
So setup is to have a Scribble control net pointing at the image (1) layer with midway weight.
Next I usually need to paint over any issues and halucinations and do general image editing to get a nice base for the final image. You can see the result in image (2).
Now I'm stuck on the last phase which sometimes works but also sometimes produces the result as in the images below. The setup is to have the same prompt as for phase 1 but use Canny Edge or Soft edge with image (2).
If anyone has any suggestions please let me know.
1.
2.
3.

[kiwaygo]

First, to troubleshoot your process:

The setup is to have the same prompt as for phase 1 but use Canny Edge or Soft edge with image (2).

Did you directly point the ControlNet to image (2), or did you extract a canny or soft edge layer from image (2) and point your control net to that layer? The latter is what needs to happen since your image (2) on its own does not resemble a canny or soft edge bitmap.

Now that's out of the way... Was the problem that step 3 did not retain the color scheme from the result of step 2? It is sometimes tricky to control color at 100% strength generation. You may try to add color information (blue sky, white sail, sunny, etc.) into your prompt but your mileage may vary. Plus, there really isn't a true "color" or "hue" ControlNet out there.

A couple of possible solutions I'd suggest:

1. Try adding an extra "unblur" ControlNet on a gaussian-blurred image (2). Unblur ControlNet tries to force the generated outcome to be a "sharper" version of the source image, so it should ensure color scheme stability. Do this but do also keeping your existing soft edge or canny ControlNet setup in order to retain edges from image (2). Note that SD1.5 unblur ControlNet currently suffers from color-shift issues (documented here), so try SDXL unblur ControlNet if you can afford to run it.

2. If approach 1 doesn't work, reduce generate strength. Gradually "refine" your step 2 result in multiple steps into your final image, and hand fix the image between steps to guide the AI. This is more of taking a safe bet attitude to generation, telling AI to change the image at a lower rate so that you can manually filter out what you don't like.

3. If approach 2 feels too laborious, maybe try using reference ControlNet and point it (directly, no additional extraction step needed) to image (2). Reference ControlNet uses "IP-Adapter" that in a way appends to your text-prompt what can be seen in image (2), automatically. It should see the color scheme in image (2). The problem is that it may possibly see too much or too little from image (2) than your expectation. You have very little control over it.

LMK if this helps.

[fxlab]

Thank you for the in-depth suggestions, I'll try them all out and report back. Cheers!