Color coding in ControlNet Segmentation not behaving as expected

[alexknows]

Using ControlNet Segmentation I'm would like to replace a small part of a wall with the flooring. I use the map to make those parts the same color as the flooring, however, when using img2img and regenerating the image it's not replacing it as I would expect.

See image and controls:

The red circles indicate the areas I want to replace with the floor.

[huchenlei]

You can try ControlNet++ seg model. It has better adherance to the control input.
https://github.com/Mikubill/sd-webui-controlnet/wiki/Model-download#controlnet-models

[huchenlei]

Also please attach full console logs. The control image is not attached to the output, which indicates that there is some problem hooking ControlNet.

[huchenlei]

SD15 is trained on 512x512 images. So by default it does not perform well on larger width/height. There are some research to make SD15 generate in high resolution, such as HiDiffusion, but I don't think they are available to A1111 rn.

[alexknows]

@huchenlei here are the logs

logs.txt

[huchenlei]

Seems like you have corrupted model file?

2024-05-25T01:51:19.820574519Z *** Error running process: /workspace/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py
2024-05-25T01:51:19.822449665Z     Traceback (most recent call last):
2024-05-25T01:51:19.822463046Z       File "/workspace/stable-diffusion-webui/modules/scripts.py", line 619, in process
2024-05-25T01:51:19.822469196Z         script.process(p, *script_args)
2024-05-25T01:51:19.822473836Z       File "/workspace/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1222, in process
2024-05-25T01:51:19.822478276Z         self.controlnet_hack(p)
2024-05-25T01:51:19.822483227Z       File "/workspace/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1207, in controlnet_hack
2024-05-25T01:51:19.822487287Z         self.controlnet_main_entry(p)
2024-05-25T01:51:19.822490657Z       File "/workspace/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 919, in controlnet_main_entry
2024-05-25T01:51:19.822493747Z         model_net, control_model_type = Script.load_control_model(p, unet, unit.model)
2024-05-25T01:51:19.822496827Z       File "/workspace/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 436, in load_control_model
2024-05-25T01:51:19.822499957Z         control_model = Script.build_control_model(p, unet, model)
2024-05-25T01:51:19.822503187Z       File "/workspace/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 464, in build_control_model
2024-05-25T01:51:19.822523098Z         state_dict = load_state_dict(model_path)
2024-05-25T01:51:19.822526688Z       File "/workspace/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/utils.py", line 21, in load_state_dict
2024-05-25T01:51:19.822529728Z         state_dict = safetensors.torch.load_file(ckpt_path, device=location)
2024-05-25T01:51:19.822532878Z       File "/workspace/venv/lib/python3.10/site-packages/safetensors/torch.py", line 311, in load_file
2024-05-25T01:51:19.822535988Z         with safe_open(filename, framework="pt", device=device) as f:
2024-05-25T01:51:19.822539048Z     safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge

[alexknows]

@huchenlei yup, I redownloaded the model using wget -0 and it's working now.

Question: Do you know what's the prompt "short_code" we have to include to tell the model to change a particular colored area using color_coding_semantic_segmentation_classes

Thank you for all your help!