Diffusers development slowing down

[bghira]

@yiyixuxu can you elaborate on the causes and plan to address the outstanding 179 pull requests that often contain really important fixes for the project? it seems honestly (no offence intended) like this project has become a bit of a side quest. is anyone seriously working on it still?

[yiyixuxu]

hi @bghira

We removed the stale bot's ability to close PRs (upon your request) - so the number of outstanding PR isn't a fair measurement of the development.

HuggingFace has had a company-wide off-site for the last week, but we are getting back to work now.

[bghira]

the list i linked was nonspecific, not intended as a list of "these are what we need" but to show that the outstanding work goes back further than just one week.

in reality all of the pull requests should be monitored and closed more regularly by an actual human who decides earlier that they're either not needed or should be merged.

Clem has pinned to the top of his X profile a tweet about how Hugging Face is profitable these days, so maybe more than four people can be assigned to work on what has become a somewhat important library.

but I can't take the statement back about how it feels like a hobby project, because nothing has made it stop feeling like that, yet. i've been told that at some point this project will shift from "startup code quality" to production code quality and then the # Copied from ... stuff goes away. but it's all still there. so this is still in "startup code quality" phase.

Hugging Face is profitable, but unprofitable companies I've worked at have more people assigned to similar projects. i would love to walk my statement back once the project seems to have a solid vision that aligns with community needs rather than being a loosely organised folder of examples that we need to extend and maintain ourselves.

[asomoza]

I find that your comment is backwards though. Almost all the projects I worked (no just ML) when they turn more professional they slow down a lot because you have to code more tests, put more guardrails, ensure your library almost never breaks, etc. Its the hobbyist projects that have really fast developments cycles because they don't need to care about that.

There's some other uis and libraries that break a lot and people don't seem to complain but when diffusers breaks people get mad and open a lot of issues, that shows that people care more about stability than speed here.. Also you pointed to another project that has more open issues and half the PRs and probably it's used a lot less and it's newer (don't really know because I don't use it) which seems kind of biased.

I just ask that maybe you could be more understanding that we also have to take breaks and we try to do our best, the skip layer guidance PR you opened was when we had this break and also there was some misunderstanding with it.

As I told you in discord, if you give me the PRs you think are important and haven't been taken care of, I can help with them and close the ones that are not going to be merged and review/ask the team to merge them.

[bghira]

1. "Almost all the projects I worked (no just ML) when they turn more professional they slow down a lot because you have to code more tests, put more guardrails, ensure your library almost never breaks, etc. It’s the hobbyist projects that have really fast development cycles because they don't need to care about that."

my comment is not backwards, it's highlighting that this project is positioning itself as critical infrastructure that is widely used within this community while it is being managed with a hybrid approach that does not align with this level of growing importance.

the argument about slowing down to maintain professional code standards misses the point. the frustration isn't about the pure speed of development. it is about the lack of communication, transparency, and shared vision with other stakeholders.

a community can tolerate slowness when it done with intent and purpose and this does not currently seem or feel to be the case.

2. "There’s some other UIs and libraries that break a lot and people don’t seem to complain, but when diffusers break people get mad and open a lot of issues, that shows that people care more about stability than speed here."

well those UIs and libraries are not the one that is currently really being discussed, it is this project, Diffusers, and it is a widely used piece of kit, not a small project that has few users to complain.

in fact, people caring more about stability than speed underscores my point - since stability matters a lot to the community, actively triaging pull requests and issues is even more important, right? these pull requests are not, like, 99.9% untested features and experimental overhauls.

3. "You pointed to another project that has more open issues and half the PRs and probably it’s used a lot less and it’s newer (don’t really know because I don’t use it) which seems kind of biased."

this was in the SLG pull request, where contextually, Candle was moving faster and might be more responsive to pipeline concerns for SD3.5. in some production codebases, the pipeline backend isn't really important - just the ability to host and call a microservice container endpoint is all that is needed.

but bringing it up here and now feels like it's again missing the point. that is merely a suggestion i made in one place, not an over-arching idea that now everybody should switch to Candle.

furthermore, pointing out the relative newness and inefficiencies in the Candle project does nothing to address the issues bring brought up in this discussion about the Diffusers project.

4. "I just ask that maybe you could be more understanding that we also have to take breaks and we try to do our best."

this is confusing sympathy with accountability, no one has questioned the need to take breaks, it is a request for better communication, transparency, and accountability.

it's a deflection from the core issue which relates to process. in fact, this resonates with my point already, where i've mentioned the abundance of resources available to Hugging Face to address any shortages or avoiding overworking team members.

furthermore, it brings up the idea that the break itself was done without any announcement to the community or really indication on the issue tracker that critical issues may go unaddressed for a period of time while team members are away. in Sweden there is a nation-wide vacation that takes place near the season of autumn and companies make this announcement in advance. communities are understanding.

team members must feel empowered to continue the project and not allow it to grind to a halt while some are away.

5. "The skip layer guidance PR you opened was when we had this break and also there was some misunderstanding with it."

it's the kind of reasoning here that underscores the request for greater transparency and communication. if there was some misunderstanding, why wasn't it communicated earlier? it could have avoided frustration.

this is not a problem with a single pull request, it is an over-arching pattern.

6. "As I told you in Discord, if you give me the PRs you think are important and haven’t been taken care of, I can help with them and close the ones that are not going to be merged and review/ask the team to merge them."

while the offer for you to help me maintain the diffusers project issue tracker is appreciated, i can't help but feel like it is offloading the responsibility of maintenance to unpaid community members. in contrast, it is the paid members who should be actively managing the project and monitoring incoming issues and pull requests for relevance and triaging by importance.

this request on its own underscores my point about a request for effective triaging and communication. it is a subtle acknowledgement of the current system's failure to efficiently maintain the project, where unrelated third party contributors are now being asked to chase down the project's core members to ask for continued follow-ups on enhancements/bugfixes.

[bghira]

cc @vladmandic who was a part of the original SLG pull request and asked what can be done about the problems, perhaps you have some ideas about community involvement.

[asomoza]

I understand your points but sadly I can't answer all of them, I do agree with the communication part and try to fix it in the future.

Personally I try to address and if I can't I tell the other team members about them, but I don't see any outstanding PRs like you mention aside from the SLG and the "schedulers catch up" that could prevent people from using the current models.

I don't see as offloading to unpaid members if the only thing I'm doing is just asking people who complain to be more specific about their problem, I'm not asking you to fix the issues or merge the PRs, just to give a list of the these problems you're mentioning which seems to be really important to you or the community, sorry if it sounded like I was trying you to make all the work,

Still, I'll try to search for them but probably I don't have the same vision as you because I care and know a lot more about inference than training.

Thank you for your detailed feedback, I really appreciate that you're taking the time to answer with so much detail.

[vladmandic]

thanks for looping me in - and i really do welcome this kind of conversation, i think its overdue.
while i agree with pretty much everything that was said, i do want to highlight few items:

the argument about slowing down to maintain professional code standards misses the point. the frustration isn't about the pure speed of > development. it is about the lack of communication, transparency, and shared vision with other stakeholders.
a community can tolerate slowness when it done with intent and purpose and this does not currently seem or feel to be the case.

this was and still is a big issue. its impossible to plan what's coming up without any insight into plans.
right now we're basically left clicking refresh to see if a new model or feature has been implemented or not.

same with release schedules - i've written a custom installer that relies on diffusers main branch because i cannot rely/wait on unknown release schedule.

if you give me the PRs you think are important and haven’t been taken care of

i really try to avoid this unless its highly critical. but the fact that its needed more and more often is a problem.
there should be some kind of prioritization - while some prs may not be urgent or even really ready, some others should have much quicker turnaround.

... projects I worked (no just ML) when they turn more professional they slow down a lot because you have to code more tests, put more guardrails, ensure your library almost never breaks, etc.

while this is true (and positive), it leaves out other side of the equation - it should also be balanced with actual process guidelines.
right now, diffusers is somewhere in neverland.

how many times did i need to do a ping-back to prevent issue being closed by a bot? that is not just silly, that is unprofessional.
and if we're talking about professional, there should be a triage: for issues and for prs. and triage should not only be initial thing.
and for example, if pr is assigned for review and it received no timely review of any kind, it should be flagged by a process.

one more thing that comes from professional vs community driven apparent conflict: docs and test.
i've seen too many quality prs come from community just to get stalled and given-up on because the lack of automated tests and/or docs.
i agree that tests and docs are important, but if diffusers are going to be welcoming of community contributions, there has to be a process to help out with items that hobby-developer is not familiar with and has no interest in.

for example, see #9041 or #9017 - good and simple prs (not urgent ones, just using here as an example)
stalled for over 3 months simply because of lack of automated tests. and there is no process to highlight that.
quality/tests/docs are all very positive things, but current process is really demotivating any kind of community contributions.

[yiyixuxu]

@vladmandic thanks for your feedback! overall I think they are constructive and helpful

this was and still is a big issue. its impossible to plan what's coming up without any insight into plans.
right now we're basically left clicking refresh to see if a new model or feature has been implemented or not.

same with release schedules - i've written a custom installer that relies on diffusers main branch because i cannot rely/wait on unknown release schedule.

For this, we cannot do anything about it because such information was often confidential - e.g. we were not allowed to talk about timelines of SD3.5 release beforehand. It is not that we failed to communicate it with you ahead of time - we actually very carefully and purposely try to keep them as secret because we have to. A lot of our releases are planned around with these model releases so the same limitation applies to our releases

for most of our regular features, we work in the open so you can track the progress with the PR so I'm assuming this particular comment is not about these regular features

and regards your reply to this comment ( in response to "if you give me the PRs you think are important and haven’t been taken care of"). We are asking you help provide feedbacks on what's important - I don't see why it is a problem. We maintainers have some sense of understanding which features are important and which ones are not, but fundamentally, we are not the users, and our insights are often incorrec. Having direct feedback from our power users will be very valuable and help us prioritize our time.

i really try to avoid this unless its highly critical. but the fact that its needed more and more often is a problem.
there should be some kind of prioritization - while some prs may not be urgent or even really ready, some others should have much quicker turnaround.

I agree with the comment about our CI and testing process, I do also notice the same issues that a lot of PRs are left stuck at the make style step - we will look into ways to improve them.

[vladmandic]

I understand the confidentiity required when working on unreleased models.
The planning I'm referring to is more about actual model functionality beyond day-zero support.

For example, day zero may be txt2img.
But what are followup plans for img2img/inpaint, lora, ipadapter, controlnet? I'd list those 4 as an absolute most for any high impact model - and this is where we stumble with unknown plans, sometimes for months at the time.

[yiyixuxu]

But what are followup plans for img2img/inpaint, lora, ipadapter, controlnet? I'd list those 4 as an absolute most for any high impact model - and this is where we stumble with unknown plans, sometimes for months at the time.

Thanks for clearing it up!
For img2img/inpaint, it is more straightforward - we can for sure do a better job communicating a timeline with you and try to add the support either on day-0 or shortly after day0 release, either by ourselves or from the community

For contronet/lora/ip-adapter it is usually more case-by-case, e.g. it depends on whether there would be checkpoints from the community and the quality of them - we will want to wait a bit to add support once we are confident the models are used a lot. so again, it would be helpful if you can share your insights/feedback when you think a controlnet/ip-adapter checkpoints need to be supported, we would take them into account.

[vladmandic]

thanks!

and on community contributions, my rule-of-thumb is that pr review should not be more than 3 interactions - after that, it's a) good to merge, b) close-enough so take it over finishing line yourself, c) switch back to draft.

If pr takes 10+ interactions due to make style, lack of tests or docs or any other minor and non-functional item, it's no longer a teaching moment - only thing contributor is learning at that time is "oh, I'm NOT doing this again".

[bghira]

patrick would often come along and finish my PRs up the way he saw fit, and for a short while sayak was doing the same. this is fine with me to do.

[bghira]

i wanted to extend an apology for my tone during the initial discussion and any perceived or real hostile comments, and also thank you all for the constructive response and the follow-through that's resulted in much improved triage and review. it seems like a lot of good things will keep happening as we all work to keep that spirit up together.

[yiyixuxu]

thanks! @bghira

[DN6]

Hi @vladmandic @bghira. The feedback about transparency and communication is valid and we would like to do better on that front. We've put together an initial roadmap board for Diffusers to show what is being worked on/prioritized for the next release.
https://github.com/orgs/huggingface/projects/61/views/1

It's a bit rough at the moment since a lot of stuff has already been merged, but I think post v0.32.0 it will be a really good way to track everything being included in an upcoming release.

[vladmandic]

thanks for sharing this - its very informative and i hope the project view stays current in the future.

one concern is that ask from hf team to us was to highlight what would be considered as high priority.
i highlighted several items, but i don't see ANY of them in the project board (and for the first 3, there is no progress/update since then):

• SD35 IPAdapter: Add support for SD 3.5 IP-Adapters #9966
• SD35 LoRA: Improve SD35 LoRA support to cover most popular LoRA formats #9950
• Flux IPAdapter: Support IPAdapters for FLUX pipelines #9825
• Flux NF4: Flux.1 cannot load standard transformer in nf4 #9996

[sayakpaul]

We're still experimenting with the idea of "high-priority" issues. But usually when there are updates on an issue (in terms of solutions), we can consider it's being actively considered.

In this case, all except for Flux IP Adapter and SD35 LoRA (this is on my mind) are being worked upon which is evident from the threads themselves:

• [WIP] SD3.5 IP-Adapter Pipeline Integration #9987
• Flux.1 cannot load standard transformer in nf4 #9996 (comment)

SD35 LoRA should have made its way sooner but other things have come up sadly. But know that it will be in soon. I have not left its thought :)

[vladmandic]

Thanks Paul

[sayakpaul]

Update: SD3.5 LoRA is up: #10167

[sayakpaul]

For Flux IP-Adapter, I think @asomoza can provide an update

[asomoza]

I'll review and test the IP Adapter today too