IP-Adapter support for StableDiffusion3ControlNetPipeline

[guiyrt]

What does this PR do?

Inherit from SD3IPAdapterMixin to allow image prompting.

Fixes #10129

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@hlky
@yiyixuxu

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[SahilCarterr]

Can you show some examples images? @guiyrt

[guiyrt]

Here are a few examples using stabilityai/stable-diffusion-3.5-large-controlnet-canny and InstantX/SD3.5-Large-IP-Adapter:

Inference code

import torch
from PIL import Image

from diffusers.models import SD3ControlNetModel
from diffusers.image_processor import VaeImageProcessor
from diffusers import StableDiffusion3ControlNetPipeline
from transformers import SiglipVisionModel, SiglipImageProcessor


class SD3CannyImageProcessor(VaeImageProcessor):
    def __init__(self):
        super().__init__(do_normalize=False)
    def preprocess(self, image, **kwargs):
        image = super().preprocess(image, **kwargs)
        image = image * 255 * 0.5 + 0.5
        return image
    def postprocess(self, image, do_denormalize=True, **kwargs):
        do_denormalize = [True] * image.shape[0]
        image = super().postprocess(image, **kwargs, do_denormalize=do_denormalize)
        return image

model_id = "stabilityai/stable-diffusion-3.5-large"
image_encoder_id = "google/siglip-so400m-patch14-384"
ip_adapter_id = "InstantX/SD3.5-Large-IP-Adapter"
controlnet_id= "stabilityai/stable-diffusion-3.5-large-controlnet-canny"

controlnet = SD3ControlNetModel.from_pretrained(
    controlnet_id, torch_dtype=torch.float16
)

feature_extractor = SiglipImageProcessor.from_pretrained(
    image_encoder_id, torch_dtype=torch.float16
)

image_encoder = SiglipVisionModel.from_pretrained(
    image_encoder_id, torch_dtype=torch.float16
)

pipe = StableDiffusion3ControlNetPipeline.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    feature_extractor=feature_extractor,
    image_encoder=image_encoder,
    controlnet=controlnet
)
pipe.image_processor = SD3CannyImageProcessor()

# Load IP Adapter
pipe.load_ip_adapter(ip_adapter_id, revision="f1f54ca369ae759f9278ae9c87d46def9f133c78")
pipe.set_ip_adapter_scale(0.5)
pipe._exclude_from_cpu_offload.append("image_encoder")
pipe.enable_sequential_cpu_offload()

# Input
controlnet_image = Image.open("canny.jpg").convert('RGB')
ip_adapter_img = Image.open("image.jpg").convert('RGB')

# please note that SD3.5 Large is sensitive to highres generation like 1536x1536
image = pipe(
    width=1024,
    height=1024,
    prompt="a fox with trees in the background",
    negative_prompt="lowres, low quality, worst quality",
    num_images_per_prompt=4,
    generator=torch.manual_seed(42),
    ip_adapter_image=ip_adapter_img,
    control_image=controlnet_image,
    controlnet_conditioning_scale=1.0,
    guidance_scale=3.5,
    num_inference_steps=60,
).images[0]

image.save(f"result.jpg")

Here I used the original image as input for the IP-Adapter:

These results look awesome, and using the IP-Adapter helps a lot, check some outputs without image prompt:

Here I tried to use different image prompts to change the background:

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[hlky]

Thanks @guiyrt! The examples are great 🤗