Xet Storage Integration

[hanouticelina]

This issue tracks the integration of xet storage support into huggingface_hub.

The current plan is to make hf_xet available as an optional dependency via pip install huggingface_hub[hf_xet], similar to what we already have with hf_transfer.
Mid-term, the plan is to integrate hf_xet as a required dependency.

Implementation Plan

The integration will be completed in phases on a dedicated xet branch. As a first step, we need to have the staging environment (https://hub-ci.huggingface.co/) ready for xet testing. cc @coyotte508 maybe.

On huggingface_hub side, we will split the work into parallel tasks for the upload and download workflows, each handled through separate PRs:

• Integrate changes from the xet poc. A full diff between the poc and the main huggingface_hub can be found here (private link).
• Add tests and provide documentation for each part.

cc @julien-c @LysandreJik @bpronan @Pierrci