storage-limits (#1515)

* storage-limits

* mini tweak

* Additional links + details (#1516)

* add some details about billing of payg storage

* Apply suggestions from code review

Co-authored-by: Julien Chaumond <[email protected]>

---------

Co-authored-by: Julien Chaumond <[email protected]>

* up - grant private repo

* Update docs/hub/storage-limits.md

Co-authored-by: Pierric Cistac <[email protected]>

* Update docs/hub/storage-limits.md

* Update docs/hub/billing.md

Co-authored-by: vb <[email protected]>

* Update docs/hub/storage-limits.md

* up - clarify best-effort.

---------

Co-authored-by: Simon Brandeis <[email protected]>
Co-authored-by: Vaibhav Srivastav <[email protected]>
Co-authored-by: Pierric Cistac <[email protected]>

docs/hub/_redirects.yml

@@ -17,3 +17,4 @@ searching-the-hub: /docs/huggingface_hub/searching-the-hub
 api-webhook: webhooks
 adapter-transformers: adapters
 security-two-fa: security-2fa
+repositories-recommendations: storage-limits

docs/hub/_toctree.yml

@@ -25,8 +25,8 @@
         title: "How-to: Create automatic metadata quality reports"
   - local: notebooks
     title: Notebooks
-  - local: repositories-recommendations
-    title: Repository size recommendations
+  - local: storage-limits
+    title: Storage Limits
   - local: repositories-next-steps
     title: Next Steps
   - local: repositories-licenses

docs/hub/billing.md

@@ -1,6 +1,6 @@
 # Billing
 
-At Hugging Face, we build a collaboration platform for the ML community (i.e., the Hub) and monetize by providing simple access to compute for AI.
+At Hugging Face, we build a collaboration platform for the ML community (i.e., the Hub) and monetize by providing advanced features and simple access to compute for AI.
 
 Any feedback or support request related to billing is welcome at [email protected]
 
@@ -61,13 +61,14 @@ You can view invoices and receipts for the last 3 months in your billing dashboa
 
 ## Enterprise Hub subscriptions
 
-We offer advanced security and compliance features for organizations through our Enterprise Hub subscription, including [Single Sign-On](./enterprise-sso.md), [Advanced Access Control](./enterprise-hub-resource-groups.md) for repositories, control over your data location, and more.
+We offer advanced security and compliance features for organizations through our Enterprise Hub subscription, including [Single Sign-On](./enterprise-sso.md), [Advanced Access Control](./enterprise-hub-resource-groups.md) for repositories, control over your data location, higher [storage capacity](./storage-limits.md) for private repositories, and more.
 
 The Enterprise Hub is billed like a typical subscription. It renews automatically, but you can choose to cancel it at any time in the organization's billing settings.
 
 You can pay for the Enterprise Hub subscription with a credit card or your AWS account.
 
 Upon renewal, the number of seats in your Enterprise Hub subscription will be updated to match the number of members of your organization.
+Private repository storage above the [included storage](./storage-limits.md) will be billed along with your subscription renewal.
 
 
 <div class="flex justify-center">
@@ -80,6 +81,7 @@ Upon renewal, the number of seats in your Enterprise Hub subscription will be up
 The PRO subscription unlocks additional features for users, including:
 
 - Higher free tier for the Serverless Inference API and when consuming ZeroGPU Spaces
+- Higher [storage capacity](./storage-limits.md) for private repositories
 - Ability to create ZeroGPU Spaces and use Dev Mode
 - Ability to write Social Posts and Community Blogs
 - Leverage the Dataset Viewer on private datasets
@@ -89,5 +91,10 @@ View the full list of benefits at https://huggingface.co/subscribe/pro
 Similarly to the Enterprise Hub subscription, PRO subscriptions are billed like a typical subscription. The subscription renews automatically for you. You can choose to cancel the subscription at anytime in your billing settings: https://huggingface.co/settings/billing
 
 You can only pay for the PRO subscription with a credit card. The subscription is billed separately from any pay-as-you-go compute usage.
+Private repository storage above the [included storage](./storage-limits.md) will be billed along with your subscription renewal.
 
 Note: PRO benefits are also included in the Enterprise Hub subscription.
+
+## Pay-as-you-go private storage 
+
+Above the included 1TB (or 1TB per seat) of private storage in PRO and Enterprise Hub, private storage is invoiced at **$25/TB/month**, in 1TB increments.

docs/hub/datasets-adding.md

@@ -111,4 +111,4 @@ The Hugging Face Hub supports large scale datasets, usually uploaded in Parquet
 
 You can upload large scale datasets at high speed using the `huggingface_hub` library.
 
-See [how to upload a folder by chunks](/docs/huggingface_hub/guides/upload#upload-a-folder-by-chunks), the [tips and tricks for large uploads](/docs/huggingface_hub/guides/upload#tips-and-tricks-for-large-uploads) and the [repository limitations and recommendations](./repositories-recommendations).
+See [how to upload a folder by chunks](/docs/huggingface_hub/guides/upload#upload-a-folder-by-chunks), the [tips and tricks for large uploads](/docs/huggingface_hub/guides/upload#tips-and-tricks-for-large-uploads) and the [repository storage limits and recommendations](./storage-limits).

docs/hub/enterprise-hub.md

@@ -23,3 +23,5 @@ In this section we will document the following Enterprise Hub features:
 - [Tokens Management](./enterprise-hub-tokens-management)
 - [Analytics](./enterprise-hub-analytics)
 - [Network Security](./enterprise-hub-network-security)
+
+Finally, Enterprise Hub includes 1TB of [private repository storage](./storage-limits) per seat in the subscription, i.e. if your organization has 40 members, then you have 40TB included storage for your private models and datasets.

docs/hub/other.md

@@ -6,6 +6,7 @@
   - [Managing Organizations](./organizations-managing)
   - [Organization Cards](./organizations-cards)
   - [Access control in organizations](./organizations-security)
+- [Enterprise Hub](./enterprise-hub)
 - [Moderation](./moderation)
 - [Billing](./billing)
 - [Digital Object Identifier (DOI)](./doi)

docs/hub/repositories.md

@@ -13,6 +13,6 @@ In these pages, you will go over the basics of getting started with Git and inte
 - [Webhooks](./webhooks)
 - [Notifications](./notifications)
 - [Collections](./collections)
-- [Repository size recommendations](./repositories-recommendations)
+- [Repository storage limits](./storage-limits)
 - [Next Steps](./repositories-next-steps)
 - [Licenses](./repositories-licenses)

docs/hub/repositories-recommendations.md → docs/hub/storage-limits.md

@@ -1,9 +1,33 @@
-# Repository limitations and recommendations
+# Storage limits
 
-There are some limitations to be aware of when dealing with a large amount of data in your repo. Given the time it takes to stream the data,
-getting an upload/push to fail at the end of the process or encountering a degraded experience, be it on hf.co or when working locally, can be very annoying.
+At Hugging Face our intent is to provide the AI community with **free storage space for public repositories**. We do bill for storage space for **private repositories**, above a free tier (see table below).
 
-## Recommendations
+We [optimize our infrastructure](https://huggingface.co/blog/xethub-joins-hf) continuously to [scale our storage](https://x.com/julien_c/status/1821540661973160339) for the coming years of growth in Machine learning.
+
+We do have mitigations in place to prevent abuse of free public storage, and in general we ask users and organizations to make sure any uploaded large model or dataset is **as useful to the community as possible** (as represented by numbers of likes or downloads, for instance).
+
+## Storage plans
+
+| Type of account  | Public storage | Private storage              |
+| ---------------- | -------------- | ---------------------------- |
+| Free user or org | Best-effort* 🙏| 100GB                        |
+| PRO              | Unlimited ✅   | 1TB + pay-as-you-go          |
+| Enterprise Hub   | Unlimited ✅   | 1TB per seat + pay-as-you-go |
+
+💡 Enterprise Hub includes 1TB of private storage per seat in the subscription: for example, if your organization has 40 members, then you have 40TB of included private storage.
+
+*We aim to continue providing the AI community with free storage space for public repositories, please don't abuse and upload dozens of TBs of generated anime. If possible, we still ask that you consider upgrading to PRO and/or Enterprise Hub whenever possible.
+
+### Pay-as-you-go price
+
+Above the included 1TB (or 1TB per seat) of private storage in PRO and Enterprise Hub, private storage is invoiced at **$25/TB/month**, in 1TB increments. See our [billing doc](./billing) for more details.
+
+## Repository limitations and recommendations
+
+In parallel to storage limits at the account (user or organization) level, there are some limitations to be aware of when dealing with a large amount of data in a specific repo. Given the time it takes to stream the data,
+getting an upload/push to fail at the end of the process or encountering a degraded experience, be it on hf.co or when working locally, can be very annoying. In the following section, we describe our recommendations on how to best structure your large repos.
+
+### Recommendations
 
 We gathered a list of tips and recommendations for structuring your repo. If you are looking for more practical tips, check out [this guide](https://huggingface.co/docs/huggingface_hub/main/en/guides/upload#tips-and-tricks-for-large-uploads) on how to upload large amount of data using the Python library.
 
@@ -21,7 +45,7 @@ _* Not relevant when using `git` CLI directly_
 
 Please read the next section to understand better those limits and how to deal with them.
 
-## Explanations
+### Explanations
 
 What are we talking about when we say "large uploads", and what are their associated limitations? Large uploads can be
 very diverse, from repositories with a few huge files (e.g. model weights) to repositories with thousands of small files
@@ -31,9 +55,9 @@ Under the hood, the Hub uses Git to version the data, which has structural impli
 If your repo is crossing some of the numbers mentioned in the previous section, **we strongly encourage you to check out [`git-sizer`](https://github.com/github/git-sizer)**,
 which has very detailed documentation about the different factors that will impact your experience. Here is a TL;DR of factors to consider:
 
-- **Repository size**: The total size of the data you're planning to upload. We generally support repositories up to 300GB. If you would like to upload more than 300 GBs (or even TBs) of data, you will need to ask us to grant more storage. To do that, please send an email with details of your project to [email protected].
+- **Repository size**: The total size of the data you're planning to upload. We generally support repositories up to 300GB. If you would like to upload more than 300 GBs (or even TBs) of data, you will need to ask us to grant more storage. To do that, please send an email with details of your project to [email protected] (for datasets) or [email protected] (for models).
 - **Number of files**:
-    - For optimal experience, we recommend keeping the total number of files under 100k. Try merging the data into fewer files if you have more.
+    - For optimal experience, we recommend keeping the total number of files under 100k, and ideally much less. Try merging the data into fewer files if you have more.
       For example, json files can be merged into a single jsonl file, or large datasets can be exported as Parquet files or in [WebDataset](https://github.com/webdataset/webdataset) format.
     - The maximum number of files per folder cannot exceed 10k files per folder. A simple solution is to
       create a repository structure that uses subdirectories. For example, a repo with 1k folders from `000/` to `999/`, each containing at most 1000 files, is already enough.
@@ -57,7 +81,7 @@ happen (in rare cases) that even if the timeout is raised client-side, the proce
 completed server-side. This can be checked manually by browsing the repo on the Hub. To prevent this timeout, we recommend
 adding around 50-100 files per commit.
 
-## Sharing large datasets on the Hub
+### Sharing large datasets on the Hub
 
 One key way Hugging Face supports the machine learning ecosystem is by hosting datasets on the Hub, including very large ones. However, if your dataset is bigger than 300GB, you will need to ask us to grant more storage.
 
@@ -78,3 +102,13 @@ For hosting large datasets on the Hub, we require the following for your dataset
 - Avoid the use of custom loading scripts when using datasets. In our experience, datasets that require custom code to use often end up with limited reuse.
 
 Please get in touch with us if any of these requirements are difficult for you to meet because of the type of data or domain you are working in.
+
+### Sharing large volumes of models on the Hub
+
+Similarly to datasets, if you host models bigger than 300GB or if you plan on uploading a large number of smaller sized models (for instance, hundreds of automated quants) totalling more than 1TB, you will need to ask us to grant more storage. 
+
+To do that, to ensure we can effectively support the open-source ecosystem, please send an email with details of your project to [email protected].
+
+### Grants for private repositories
+
+If you need more model/ dataset storage than your allocated private storage for academic/ research purposes, please reach out to us at [email protected] or [email protected] along with a proposal of how you will use the storage grant.