Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flux failures using from_pipe #10399

Open
vladmandic opened this issue Dec 27, 2024 · 1 comment · May be fixed by #10400
Open

Flux failures using from_pipe #10399

vladmandic opened this issue Dec 27, 2024 · 1 comment · May be fixed by #10400
Labels
bug Something isn't working

Comments

@vladmandic
Copy link
Contributor

vladmandic commented Dec 27, 2024 •
edited
Loading

Describe the bug

loading a flux model as usual works fine. pipeline can then be switched to img2img or inpaint without issues.
but once its img2img or inpaint, it cannot be switched back to txt2img since some of the pipeline modules are mandatory to be registered (even if as none) in txt2img and not present in img2img

Reproduction

pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell")
pipe = AutoPipelineForInpainting.from_pipe(pipe) # this is ok
pipe = AutoPipelineForText2Image.from_pipe(pipe) # this fails

issue seems to be due to fact that flux txt2img pipeline defines:

        self.register_modules(
            vae=vae,
            text_encoder=text_encoder,
            text_encoder_2=text_encoder_2,
            tokenizer=tokenizer,
            tokenizer_2=tokenizer_2,
            transformer=transformer,
            scheduler=scheduler,
            image_encoder=image_encoder,
            feature_extractor=feature_extractor,
        )

while img2img and inpaint pipelines only define:

        self.register_modules(
            vae=vae,
            text_encoder=text_encoder,
            text_encoder_2=text_encoder_2,
            tokenizer=tokenizer,
            tokenizer_2=tokenizer_2,
            transformer=transformer,
            scheduler=scheduler,
        )

Logs

has been incorrectly initialized or  
<class 'diffusers.pipelines.flux.pipeline_flux_img2img.FluxImg2ImgPipeline'> is incorrectly implemented.
Expected {'text_encoder_2', 'scheduler', 'tokenizer', 'transformer', 'text_encoder', 'tokenizer_2', 'vae'} 
to be defined, but 
dict_keys(['vae', 'text_encoder', 'text_encoder_2', 'tokenizer', 'tokenizer_2', 'transformer', scheduler', 'image_encoder', 'feature_extractor']) are defined.

System Info

diffusers==0.33.0.main

Who can help?

@yiyixuxu @sayakpaul @DN6 @asomoza
@vladmandic vladmandic added the bug Something isn't working label Dec 27, 2024
@hlky hlky linked a pull request Dec 27, 2024 that will close this issue
Open
@sayakpaul
Copy link
Member

Cc: @yiyixuxu

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
2 participants
@sayakpaul @vladmandic