stablediffusion (sdxl) ip-adapter support

[Suprhimp]

Feature request

I just wonder is it possible to use ip-adpater with optiumum-neuron environment.

Because I found ip-adapter related parameter in pipeline

But When I load ip-adpater with this code

from optimum.neuron import NeuronStableDiffusionXLPipeline
from transformers.models.clip.modeling_clip import CLIPVisionModelWithProjection
import torch
# model id you want to compile
vanilla_model_id = "stabilityai/stable-diffusion-xl-base-1.0"

# configs for compiling model
compiler_args = {"auto_cast": "matmul", 
                 "auto_cast_type": "bf16",
                 "output_hidden_states": True
                 }
input_shapes = {
  "height": 1024, # width of the image
  "width": 1024, # height of the image
  "num_images_per_prompt":1, # number of images to generate per prompt
  "batch_size": 1 # batch size for the model
  }
controlnet_id = "diffusers/controlnet-canny-sdxl-1.0"

image_encoder = CLIPVisionModelWithProjection.from_pretrained(
        "h94/IP-Adapter",
        subfolder="models/image_encoder",
        torch_dtype=torch.float16,
    )

sd = NeuronStableDiffusionXLPipeline.from_pretrained(vanilla_model_id,controlnet_ids=controlnet_id,image_encoder = image_encoder,
 export=True, **input_shapes, **compiler_args)

print('IP adapter load start!!!!!!!!!!!!!!')
sd.load_ip_adapter(
        "h94/IP-Adapter",
        subfolder="sdxl_models",
        weight_name="ip-adapter-plus_sdxl_vit-h.safetensors",
    )
print('IP adapter load End!!!!!!!!!!!!!')

# Save locally or upload to the HuggingFace Hub
save_directory = "compile_with_ip_adpater"
sd.save_pretrained(save_directory)

It gives me error

Traceback (most recent call last):
  File "compile.py", line 30, in <module>
    sd.load_ip_adapter(
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/diffusers/loaders/ip_adapter.py", line 234, in load_ip_adapter
    unet._load_ip_adapter_weights(state_dicts, low_cpu_mem_usage=low_cpu_mem_usage)
AttributeError: 'NeuronModelUnet' object has no attribute '_load_ip_adapter_weights'

Motivation

If optimum-neuron cover ip-adapter, almost all stablediffusion function will can be replaced with npu.

If I can use ip-adapter in optimum neuron, there is no reason to use stable diffusion in gpu for our product.

Your contribution

I can be a tester..! or contributor if possible..

[dacorvo]

@Suprhimp you inadvertently submitted one of your personal token in the issue. Please revoke or rotate it as soon as possible and if possible let us know once it is done.

[Suprhimp]

Thanks for the checks.. I remove the key and I revoked it. @dacorvo

[Suprhimp]

Can I get any information of this? Does it impossible to use ip-adapter in optimum?

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[JingyaHuang]

Will add the support.