does it mean that the optimizer actually does not take any parameters to update? So the model parameters won't be updated in the training loop?

I guess.

@weixiong-ur have you had the same result as me ?

a few things:

• I'm confused by this repr. You only have a single GPU no? (per accelerate env). We don't support non-multi-GPU FSDP.
• And if FSDP truly were broken in such a way, it'd be a much larger problem which we know it's not.

Can you give a non jupyter notebook-based repr, and if you are sharing a jupyter notebook, for security reasons, please share the notebook in a gist with outputs not as a PDF as I'm hesitant to open these due to security reasons, and it's difficult to copy/paste from them etc

Not sure if it's related, but users reported an error in PEFT that points in a similar direction. Note that the error is not caused by PEFT, as I could reproduce it without PEFT. From the error message, it seems like the params passed to the optimizer are not consistent with the model parameters. I could resolve the error by downgrading to:

• trl==0.11.0
• tokenizers>=0.19,<0.20
• transformers==4.44.2
• accelerate==0.33.0

@eljandoubi Maybe you could check if these versions resolve your issue.

@muellerzr
You can use my branch of the transformers repository [here] to train a model like Donut with FSDP wrapping based on layer size. It will print the number of parameters before and after applying FSDP wrapping.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

Can you check if huggingface/transformers#35212 has solved the issue? If not, could you try if additionally switching off flash attention helps.