From 7466f78a53deea9cc88fca713541cb6ae57b8fb1 Mon Sep 17 00:00:00 2001 From: Wauplin Date: Thu, 12 Dec 2024 12:52:07 +0100 Subject: [PATCH 01/19] Start DDUF documentation --- docs/hub/_toctree.yml | 2 + docs/hub/dduf.md | 152 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 154 insertions(+) create mode 100644 docs/hub/dduf.md diff --git a/docs/hub/_toctree.yml b/docs/hub/_toctree.yml index dcee8ed13..72d0c88d3 100644 --- a/docs/hub/_toctree.yml +++ b/docs/hub/_toctree.yml @@ -137,6 +137,8 @@ title: Integrate a library with the Hub - local: models-tasks title: Tasks + - local: dduf + title: DDUF - local: gguf title: GGUF sections: diff --git a/docs/hub/dduf.md b/docs/hub/dduf.md new file mode 100644 index 000000000..1d765c3ee --- /dev/null +++ b/docs/hub/dduf.md @@ -0,0 +1,152 @@ +# DDUF: Diffusion ??? Universal Format + +## Overview + +DDUF (Diffusion ??? Universal Format) is a file format designed to make storing, distributing, and using diffusion models much easier. Built on the ZIP file format, DDUF offers a standardized, efficient, and flexible way to package all parts of a diffusion model into a single, easy-to-manage file. + +This work draws inspiration from the [GGUF](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) format. + + + +We welcome contributions with open arms! + +To create a widely adopted file format, we need early feedback from the community. Nothing is set in stone, and we value everyone's input. Is your use case not covered? Please let us know! Discussions about the DDUF format happen in the https://huggingface.co/DDUF organization. + + + +## Motivation + +Yet, another file format? Yes, but for good reasons! + +The primary goal of DDUF is to create a community-endorsed file format for diffusion models. Current model distribution methods often involve multiple separate files, different weight-saving formats, and managing files from various locations. DDUF aims to solve these challenges by packaging all model components into a single file, enforcing a consistent structure while being opinionated about saving formats. + +The DDUF format is also designed to be language-agnostic. While we currently provide tooling for the Python ecosystem, there's nothing stopping similar tools from being developed in JavaScript, Rust, C++, and other languages. Like GGUF or safetensors, DDUF is built to be parsable from a remote location without downloading the entire file, which will enable advanced support on the Hugging Face Hub. + +## Key Features + +1. **Single file** packaging. +2. Based on **ZIP file format** to leverage existing tooling. +3. No compression, ensuring **`mmap` compatibility** for fast loading and saving. +4. **HTTP-friendly**: metadata and file structure can be fetched remotely using HTTP Range requests. +5. **Flexible**: each model component is stored in its own directory, following the current `diffusers` structure. +6. **Safe**: uses `safetensors` as weights saving format and prohibits nested directories to prevent ZIP-bombs. + +## Technical specifications + +Technically, a `.dduf` file **is** a [`.zip` archive](https://en.wikipedia.org/wiki/ZIP_(file_format)). By building on a universally supported file format, we ensure robust tooling already exists. However, some constraints are enforced to meet diffusion models' requirements: +- Data must be stored uncompressed (flag `0`), allowing mmap-compatibility. +- Data must be stored using ZIP64 protocol, enabling saving files above 2GB. +- The archive can only contain `.json`, `.safetensors`, `.model` and `.txt` files. +- A `model_index.json` file must be present at the root of the archive. It must contain a key-value mapping with metadata about the model and its components. +- Each component must be stored in its own directory (e.g., `vae/`, `text_encoder/`). Nested files must use UNIX-style path separators (`/`). +- Each directory must correspond to a component in the `model_index.json` index. +- Each directory must contain a json config file (one of `config.json`, `tokenizer_config.json`, `image_processor.json`). +- Sub-directories are forbidden. + +Want to check if your file is valid? Check it out using this Space: https://huggingface.co/spaces/DDUF/dduf-check. + +## Usage + +The `huggingface_hub` provides tooling to handle DDUF files in Python. It includes built-in rules to validate file integrity and helpers to read and export DDUF files. The goal is to see this tooling adopted in the Python ecosystem, such as in the `diffusers` integration. Similar tooling can be developed for other languages (JavaScript, Rust, C++, etc.). + +### How to read a DDUF file? + +Reading a DDUF file is as simple as calling `read_dduf_file` and passing a path as argument. Only the metadata is read, meaning this is a lightweight call that will not explode your memory. In the example below, we consider that you've already downloaded the [`FLUX.1-dev.dduf`](https://huggingface.co/DDUF/FLUX.1-dev-DDUF/blob/main/FLUX.1-dev.dduf) file locally. + +```python +>>> from huggingface_hub import read_dduf_file + +# Read DDUF metadata +>>> dduf_entries = read_dduf_file("FLUX.1-dev.dduf") +``` + +This will return a mapping where each entry corresponds to a file in the DDUF archive. A file is represented by a `DDUFEntry` dataclass that contains the filename, offset and length of the entry in the original DDUF file. This information is useful to read its content without loading the whole file. In practice, you won't have to handle low-level reading but rely on helpers instead. + +For instance, here is how to load the `model_index.json` content: +```python +>>> import json +>>> json.loads(dduf_entries["model_index.json"].read_text()) +{'_class_name': 'FluxPipeline', '_diffusers_version': '0.32.0.dev0', '_name_or_path': 'black-forest-labs/FLUX.1-dev', ... +``` + +For binary files, you'll want to access the raw bytes using `as_mmap`. This returns bytes as a memory-mapping on the original file. The memory-mapping allows you to read only the bytes you need without loading everything in memory. For instance, here is how to load safetensors weights: + +```python +>>> import safetensors.torch +>>> with dduf_entries["vae/diffusion_pytorch_model.safetensors"].as_mmap() as mm: +... state_dict = safetensors.torch.load(mm) # `mm` is a bytes object +``` + +Note: `as_mmap` must be used in a context manager to benefit from the memory-mapping properties. + +### How to write a DDUF file? + +A DDUF file can be exported by passing to `export_folder_as_dduf` a folder path containing a diffusion model: + +```python +# Export a folder as a DDUF file +>>> from huggingface_hub import export_folder_as_dduf +>>> export_folder_as_dduf("FLUX.1-dev.dduf", folder_path="path/to/FLUX.1-dev") +``` + +This tool scans the folder, adds the relevant entries and ensures the exported file is valid. If anything goes wrong during the process, a `DDUFExportError` is raised. + +For more flexibility, you can use [`export_entries_as_dduf`] and pass a list of files to include in the final DDUF file: + +```python +# Export specific files from the local disk. +>>> from huggingface_hub import export_entries_as_dduf +>>> export_entries_as_dduf( +... dduf_path="stable-diffusion-v1-4-FP16.dduf", +... entries=[ # List entries to add to the DDUF file (here, only FP16 weights) +... ("model_index.json", "path/to/model_index.json"), +... ("vae/config.json", "path/to/vae/config.json"), +... ("vae/diffusion_pytorch_model.fp16.safetensors", "path/to/vae/diffusion_pytorch_model.fp16.safetensors"), +... ("text_encoder/config.json", "path/to/text_encoder/config.json"), +... ("text_encoder/model.fp16.safetensors", "path/to/text_encoder/model.fp16.safetensors"), +... # ... add more entries here +... ] +... ) +``` + +This works well if you've already saved your model on the disk. But what if you have a model loaded in memory and want to serialize it directly into a DDUF file? `export_entries_as_dduf` lets you do that by providing a Python `generator` that tells how to serialize the data iteratively: + +```python +(...) + +# Export state_dicts one by one from a loaded pipeline +>>> def as_entries(pipe: DiffusionPipeline) -> Generator[Tuple[str, bytes], None, None]: +... # Build a generator that yields the entries to add to the DDUF file. +... # The first element of the tuple is the filename in the DDUF archive. The second element is the content of the file. +... # Entries will be evaluated lazily when the DDUF file is created (only 1 entry is loaded in memory at a time) +... yield "vae/config.json", pipe.vae.to_json_string().encode() +... yield "vae/diffusion_pytorch_model.safetensors", safetensors.torch.save(pipe.vae.state_dict()) +... yield "text_encoder/config.json", pipe.text_encoder.config.to_json_string().encode() +... yield "text_encoder/model.safetensors", safetensors.torch.save(pipe.text_encoder.state_dict()) +... # ... add more entries here + +>>> export_entries_as_dduf(dduf_path="my-cool-diffusion-model.dduf", entries=as_entries(pipe)) +``` + +## F.A.Q. + +### Why build on top of ZIP? + +ZIP provides several advantages: +- Universally supported file format +- No additional dependencies for reading +- Built-in file indexing +- Wide language support + +Why not use a TAR with a table of contents at the beginning of the archive? See explanation [in this comment](https://github.com/huggingface/huggingface_hub/pull/2692#issuecomment-2519863726). + +### Why no compression? + +- Enables direct memory mapping of large files. +- Ensures consistent and predictable remote file access. +- Prevents CPU overhead during file reading. +- Maintains compatibility with safetensors. + +### Can I Modify a DDUF file? + +No. For now, DDUF files are designed to be immutable. To update a model, create a new DDUF file. From d29af473fffd482b5a2c1b239d0fdaa1fb24fe4b Mon Sep 17 00:00:00 2001 From: Lucain Date: Thu, 12 Dec 2024 14:29:04 +0100 Subject: [PATCH 02/19] Update docs/hub/dduf.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Célina --- docs/hub/dduf.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/dduf.md b/docs/hub/dduf.md index 1d765c3ee..3c3ab7093 100644 --- a/docs/hub/dduf.md +++ b/docs/hub/dduf.md @@ -35,7 +35,7 @@ The DDUF format is also designed to be language-agnostic. While we currently pro Technically, a `.dduf` file **is** a [`.zip` archive](https://en.wikipedia.org/wiki/ZIP_(file_format)). By building on a universally supported file format, we ensure robust tooling already exists. However, some constraints are enforced to meet diffusion models' requirements: - Data must be stored uncompressed (flag `0`), allowing mmap-compatibility. -- Data must be stored using ZIP64 protocol, enabling saving files above 2GB. +- Data must be stored using ZIP64 protocol, enabling saving files above 4GB. - The archive can only contain `.json`, `.safetensors`, `.model` and `.txt` files. - A `model_index.json` file must be present at the root of the archive. It must contain a key-value mapping with metadata about the model and its components. - Each component must be stored in its own directory (e.g., `vae/`, `text_encoder/`). Nested files must use UNIX-style path separators (`/`). From b8f842143de448ec4415dc1603ebc0ded2ac12d8 Mon Sep 17 00:00:00 2001 From: Wauplin Date: Thu, 12 Dec 2024 14:30:25 +0100 Subject: [PATCH 03/19] add discussion tab --- docs/hub/dduf.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/dduf.md b/docs/hub/dduf.md index 3c3ab7093..3f3650407 100644 --- a/docs/hub/dduf.md +++ b/docs/hub/dduf.md @@ -10,7 +10,7 @@ This work draws inspiration from the [GGUF](https://github.com/ggerganov/ggml/bl We welcome contributions with open arms! -To create a widely adopted file format, we need early feedback from the community. Nothing is set in stone, and we value everyone's input. Is your use case not covered? Please let us know! Discussions about the DDUF format happen in the https://huggingface.co/DDUF organization. +To create a widely adopted file format, we need early feedback from the community. Nothing is set in stone, and we value everyone's input. Is your use case not covered? Please let us know! Discussions about the DDUF format happen in the [DDUF organization](https://huggingface.co/spaces/DDUF/README/discussions/2). From 4876ba8eb1a0dfd68df84f1a437c79f81db9ef15 Mon Sep 17 00:00:00 2001 From: Wauplin Date: Thu, 12 Dec 2024 14:33:30 +0100 Subject: [PATCH 04/19] add toc --- docs/hub/models-advanced.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/hub/models-advanced.md b/docs/hub/models-advanced.md index 1d84dd305..2b31a6509 100644 --- a/docs/hub/models-advanced.md +++ b/docs/hub/models-advanced.md @@ -4,4 +4,5 @@ - [Integrate your library with the Hub](./models-adding-libraries) - [Adding new tasks to the Hub](./models-tasks) +- [DDUF format](./dduf) - [GGUF format](./gguf) \ No newline at end of file From 3c5b4b1343ae7d62a9d08d19675affc7bcc56e2d Mon Sep 17 00:00:00 2001 From: Wauplin Date: Thu, 12 Dec 2024 14:38:12 +0100 Subject: [PATCH 05/19] dduf < gguf ? --- docs/hub/_toctree.yml | 4 ++-- docs/hub/models-advanced.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/hub/_toctree.yml b/docs/hub/_toctree.yml index 72d0c88d3..ffdb7fda9 100644 --- a/docs/hub/_toctree.yml +++ b/docs/hub/_toctree.yml @@ -137,8 +137,6 @@ title: Integrate a library with the Hub - local: models-tasks title: Tasks - - local: dduf - title: DDUF - local: gguf title: GGUF sections: @@ -148,6 +146,8 @@ title: GGUF usage with GPT4All - local: ollama title: GGUF usage with Ollama + - local: dduf + title: DDUF - title: Datasets local: datasets isExpanded: true diff --git a/docs/hub/models-advanced.md b/docs/hub/models-advanced.md index 2b31a6509..ba5cd80f7 100644 --- a/docs/hub/models-advanced.md +++ b/docs/hub/models-advanced.md @@ -4,5 +4,5 @@ - [Integrate your library with the Hub](./models-adding-libraries) - [Adding new tasks to the Hub](./models-tasks) -- [DDUF format](./dduf) -- [GGUF format](./gguf) \ No newline at end of file +- [GGUF format](./gguf) +- [DDUF format](./dduf) \ No newline at end of file From fdafd487a5908166b5272fd325414224ed7b3b09 Mon Sep 17 00:00:00 2001 From: Lucain Date: Thu, 12 Dec 2024 14:38:59 +0100 Subject: [PATCH 06/19] Apply suggestions from code review Co-authored-by: Julien Chaumond --- docs/hub/dduf.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hub/dduf.md b/docs/hub/dduf.md index 3f3650407..f4572ac1c 100644 --- a/docs/hub/dduf.md +++ b/docs/hub/dduf.md @@ -18,7 +18,7 @@ To create a widely adopted file format, we need early feedback from the communit Yet, another file format? Yes, but for good reasons! -The primary goal of DDUF is to create a community-endorsed file format for diffusion models. Current model distribution methods often involve multiple separate files, different weight-saving formats, and managing files from various locations. DDUF aims to solve these challenges by packaging all model components into a single file, enforcing a consistent structure while being opinionated about saving formats. +The primary goal of DDUF is to create a community-endorsed single-file format for diffusion models. Current model distribution methods often involve multiple separate files, different weight-saving formats, and managing files from various locations. DDUF aims to solve these challenges by packaging all model components into a single file, enforcing a consistent structure while being opinionated about saving formats. The DDUF format is also designed to be language-agnostic. While we currently provide tooling for the Python ecosystem, there's nothing stopping similar tools from being developed in JavaScript, Rust, C++, and other languages. Like GGUF or safetensors, DDUF is built to be parsable from a remote location without downloading the entire file, which will enable advanced support on the Hugging Face Hub. @@ -51,7 +51,7 @@ The `huggingface_hub` provides tooling to handle DDUF files in Python. It includ ### How to read a DDUF file? -Reading a DDUF file is as simple as calling `read_dduf_file` and passing a path as argument. Only the metadata is read, meaning this is a lightweight call that will not explode your memory. In the example below, we consider that you've already downloaded the [`FLUX.1-dev.dduf`](https://huggingface.co/DDUF/FLUX.1-dev-DDUF/blob/main/FLUX.1-dev.dduf) file locally. +Reading a DDUF file is as simple as calling `read_dduf_file` and passing a path as argument. Only the metadata is read, meaning this is a lightweight call that will not make your memory explode. In the example below, we consider that you've already downloaded the [`FLUX.1-dev.dduf`](https://huggingface.co/DDUF/FLUX.1-dev-DDUF/blob/main/FLUX.1-dev.dduf) file locally. ```python >>> from huggingface_hub import read_dduf_file From 3a278ba554e2d3ad784606e288e5ef4cc81956c1 Mon Sep 17 00:00:00 2001 From: Wauplin Date: Thu, 12 Dec 2024 14:39:43 +0100 Subject: [PATCH 07/19] typo --- docs/hub/dduf.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/dduf.md b/docs/hub/dduf.md index f4572ac1c..65602637d 100644 --- a/docs/hub/dduf.md +++ b/docs/hub/dduf.md @@ -147,6 +147,6 @@ Why not use a TAR with a table of contents at the beginning of the archive? See - Prevents CPU overhead during file reading. - Maintains compatibility with safetensors. -### Can I Modify a DDUF file? +### Can I modify a DDUF file? No. For now, DDUF files are designed to be immutable. To update a model, create a new DDUF file. From 2e4ee8b612d7270d2aec6c8bea59aa3f87562229 Mon Sep 17 00:00:00 2001 From: Wauplin Date: Thu, 12 Dec 2024 14:42:55 +0100 Subject: [PATCH 08/19] naming --- docs/hub/dduf.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hub/dduf.md b/docs/hub/dduf.md index 65602637d..883b708cc 100644 --- a/docs/hub/dduf.md +++ b/docs/hub/dduf.md @@ -1,8 +1,8 @@ -# DDUF: Diffusion ??? Universal Format +# DDUF ## Overview -DDUF (Diffusion ??? Universal Format) is a file format designed to make storing, distributing, and using diffusion models much easier. Built on the ZIP file format, DDUF offers a standardized, efficient, and flexible way to package all parts of a diffusion model into a single, easy-to-manage file. +DDUF (**D**DUF’s **D**iffusion **U**nified **F**ormat) is a file format designed to make storing, distributing, and using diffusion models much easier. Built on the ZIP file format, DDUF offers a standardized, efficient, and flexible way to package all parts of a diffusion model into a single, easy-to-manage file. This work draws inspiration from the [GGUF](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) format. From 789a71478b2f119399d44b10012c8bebb35c95ca Mon Sep 17 00:00:00 2001 From: Lucain Date: Thu, 12 Dec 2024 15:06:52 +0100 Subject: [PATCH 09/19] Update docs/hub/dduf.md Co-authored-by: Julien Chaumond --- docs/hub/dduf.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/hub/dduf.md b/docs/hub/dduf.md index 883b708cc..f69ce173e 100644 --- a/docs/hub/dduf.md +++ b/docs/hub/dduf.md @@ -1,5 +1,11 @@ # DDUF +
+ + +
+ + ## Overview DDUF (**D**DUF’s **D**iffusion **U**nified **F**ormat) is a file format designed to make storing, distributing, and using diffusion models much easier. Built on the ZIP file format, DDUF offers a standardized, efficient, and flexible way to package all parts of a diffusion model into a single, easy-to-manage file. From d33d93d0c1c229492d4fe8966597d0d6d43aa8fd Mon Sep 17 00:00:00 2001 From: Lucain Date: Thu, 12 Dec 2024 15:10:58 +0100 Subject: [PATCH 10/19] Apply suggestions from code review Co-authored-by: vb --- docs/hub/dduf.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/hub/dduf.md b/docs/hub/dduf.md index f69ce173e..79a89217a 100644 --- a/docs/hub/dduf.md +++ b/docs/hub/dduf.md @@ -12,6 +12,7 @@ DDUF (**D**DUF’s **D**iffusion **U**nified **F**ormat) is a file format design This work draws inspiration from the [GGUF](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) format. +We've seeded some DDUFs of popular formats for the community to play with: https://huggingface.co/DDUF, check them out! We welcome contributions with open arms! @@ -156,3 +157,9 @@ Why not use a TAR with a table of contents at the beginning of the archive? See ### Can I modify a DDUF file? No. For now, DDUF files are designed to be immutable. To update a model, create a new DDUF file. + +### Which frameworks/apps support DDUFs? + +- [diffusers](https://github.com/huggingface/diffusers) + +We are continuously reaching out to other libraries and frameworks, if you are interested in adding support for your project, open a Discussion in the [DDUF org](https://huggingface.co/spaces/DDUF/README/discussions). From 7b13ee2412955a1202933817dd6655a8c78b5f55 Mon Sep 17 00:00:00 2001 From: Wauplin Date: Thu, 12 Dec 2024 16:10:03 +0100 Subject: [PATCH 11/19] add scheduler_config.json --- docs/hub/dduf.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/dduf.md b/docs/hub/dduf.md index 883b708cc..2271e03dd 100644 --- a/docs/hub/dduf.md +++ b/docs/hub/dduf.md @@ -40,7 +40,7 @@ Technically, a `.dduf` file **is** a [`.zip` archive](https://en.wikipedia.org/w - A `model_index.json` file must be present at the root of the archive. It must contain a key-value mapping with metadata about the model and its components. - Each component must be stored in its own directory (e.g., `vae/`, `text_encoder/`). Nested files must use UNIX-style path separators (`/`). - Each directory must correspond to a component in the `model_index.json` index. -- Each directory must contain a json config file (one of `config.json`, `tokenizer_config.json`, `image_processor.json`). +- Each directory must contain a json config file (one of `config.json`, `tokenizer_config.json`, `image_processor.json`, `scheduler_config.json`). - Sub-directories are forbidden. Want to check if your file is valid? Check it out using this Space: https://huggingface.co/spaces/DDUF/dduf-check. From 97d6df3b255a6f1942a2608bcfebc233c48a8dc4 Mon Sep 17 00:00:00 2001 From: Lucain Date: Thu, 12 Dec 2024 18:47:27 +0100 Subject: [PATCH 12/19] Update docs/hub/dduf.md Co-authored-by: vb --- docs/hub/dduf.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/hub/dduf.md b/docs/hub/dduf.md index 79a89217a..f0cecc7ad 100644 --- a/docs/hub/dduf.md +++ b/docs/hub/dduf.md @@ -13,6 +13,7 @@ DDUF (**D**DUF’s **D**iffusion **U**nified **F**ormat) is a file format design This work draws inspiration from the [GGUF](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) format. We've seeded some DDUFs of popular formats for the community to play with: https://huggingface.co/DDUF, check them out! + We welcome contributions with open arms! From 80991569f02987820a03ed68d82e8e3784af483e Mon Sep 17 00:00:00 2001 From: Wauplin Date: Fri, 13 Dec 2024 10:40:42 +0100 Subject: [PATCH 13/19] preprocessor_config.json --- docs/hub/dduf.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/dduf.md b/docs/hub/dduf.md index 0d48ab9ef..2e98fa22f 100644 --- a/docs/hub/dduf.md +++ b/docs/hub/dduf.md @@ -48,7 +48,7 @@ Technically, a `.dduf` file **is** a [`.zip` archive](https://en.wikipedia.org/w - A `model_index.json` file must be present at the root of the archive. It must contain a key-value mapping with metadata about the model and its components. - Each component must be stored in its own directory (e.g., `vae/`, `text_encoder/`). Nested files must use UNIX-style path separators (`/`). - Each directory must correspond to a component in the `model_index.json` index. -- Each directory must contain a json config file (one of `config.json`, `tokenizer_config.json`, `image_processor.json`, `scheduler_config.json`). +- Each directory must contain a json config file (one of `config.json`, `tokenizer_config.json`, `preprocessor_config.json`, `scheduler_config.json`). - Sub-directories are forbidden. Want to check if your file is valid? Check it out using this Space: https://huggingface.co/spaces/DDUF/dduf-check. From ab18cc807f3f55f37cf17d354c39ca72df17a499 Mon Sep 17 00:00:00 2001 From: Lucain Date: Fri, 13 Dec 2024 10:44:19 +0100 Subject: [PATCH 14/19] Apply suggestions from code review Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> --- docs/hub/dduf.md | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-) diff --git a/docs/hub/dduf.md b/docs/hub/dduf.md index 2e98fa22f..535a96bf3 100644 --- a/docs/hub/dduf.md +++ b/docs/hub/dduf.md @@ -12,13 +12,13 @@ DDUF (**D**DUF’s **D**iffusion **U**nified **F**ormat) is a file format design This work draws inspiration from the [GGUF](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) format. -We've seeded some DDUFs of popular formats for the community to play with: https://huggingface.co/DDUF, check them out! +Check out the [DDUF](https://huggingface.co/DDUF) org to start using some of the most popular diffusion models in DDUF. We welcome contributions with open arms! -To create a widely adopted file format, we need early feedback from the community. Nothing is set in stone, and we value everyone's input. Is your use case not covered? Please let us know! Discussions about the DDUF format happen in the [DDUF organization](https://huggingface.co/spaces/DDUF/README/discussions/2). +To create a widely adopted file format, we need early feedback from the community. Nothing is set in stone, and we value everyone's input. Is your use case not covered? Please let us know in the DDUF organization [discussions](https://huggingface.co/spaces/DDUF/README/discussions/2). @@ -30,14 +30,14 @@ The primary goal of DDUF is to create a community-endorsed single-file format fo The DDUF format is also designed to be language-agnostic. While we currently provide tooling for the Python ecosystem, there's nothing stopping similar tools from being developed in JavaScript, Rust, C++, and other languages. Like GGUF or safetensors, DDUF is built to be parsable from a remote location without downloading the entire file, which will enable advanced support on the Hugging Face Hub. -## Key Features +Its key features include the following. 1. **Single file** packaging. 2. Based on **ZIP file format** to leverage existing tooling. 3. No compression, ensuring **`mmap` compatibility** for fast loading and saving. 4. **HTTP-friendly**: metadata and file structure can be fetched remotely using HTTP Range requests. -5. **Flexible**: each model component is stored in its own directory, following the current `diffusers` structure. -6. **Safe**: uses `safetensors` as weights saving format and prohibits nested directories to prevent ZIP-bombs. +5. **Flexible**: each model component is stored in its own directory, following the current Diffusers structure. +6. **Safe**: uses [Safetensors](https://huggingface.co/docs/diffusers/using-diffusers/other-formats#safetensors) as a weight-saving format and prohibits nested directories to prevent ZIP bombs. ## Technical specifications @@ -59,7 +59,7 @@ The `huggingface_hub` provides tooling to handle DDUF files in Python. It includ ### How to read a DDUF file? -Reading a DDUF file is as simple as calling `read_dduf_file` and passing a path as argument. Only the metadata is read, meaning this is a lightweight call that will not make your memory explode. In the example below, we consider that you've already downloaded the [`FLUX.1-dev.dduf`](https://huggingface.co/DDUF/FLUX.1-dev-DDUF/blob/main/FLUX.1-dev.dduf) file locally. +Pass a path to `read_dduf_file` to read a DDUF file. Only the metadata is read, meaning this is a lightweight call that won't explode your memory. In the example below, we consider that you've already downloaded the [`FLUX.1-dev.dduf`](https://huggingface.co/DDUF/FLUX.1-dev-DDUF/blob/main/FLUX.1-dev.dduf) file locally. ```python >>> from huggingface_hub import read_dduf_file @@ -68,7 +68,7 @@ Reading a DDUF file is as simple as calling `read_dduf_file` and passing a path >>> dduf_entries = read_dduf_file("FLUX.1-dev.dduf") ``` -This will return a mapping where each entry corresponds to a file in the DDUF archive. A file is represented by a `DDUFEntry` dataclass that contains the filename, offset and length of the entry in the original DDUF file. This information is useful to read its content without loading the whole file. In practice, you won't have to handle low-level reading but rely on helpers instead. +`read_dduf_file` returns a mapping where each entry corresponds to a file in the DDUF archive. A file is represented by a `DDUFEntry` dataclass that contains the filename, offset, and length of the entry in the original DDUF file. This information is useful to read its content without loading the whole file. In practice, you won't have to handle low-level reading but rely on helpers instead. For instance, here is how to load the `model_index.json` content: ```python @@ -85,11 +85,12 @@ For binary files, you'll want to access the raw bytes using `as_mmap`. This retu ... state_dict = safetensors.torch.load(mm) # `mm` is a bytes object ``` -Note: `as_mmap` must be used in a context manager to benefit from the memory-mapping properties. +> [!TIP] +> `as_mmap` must be used in a context manager to benefit from the memory-mapping properties. ### How to write a DDUF file? -A DDUF file can be exported by passing to `export_folder_as_dduf` a folder path containing a diffusion model: +Pass a folder path to `export_folder_as_dduf` to export a DDUF file. ```python # Export a folder as a DDUF file @@ -99,7 +100,7 @@ A DDUF file can be exported by passing to `export_folder_as_dduf` a folder path This tool scans the folder, adds the relevant entries and ensures the exported file is valid. If anything goes wrong during the process, a `DDUFExportError` is raised. -For more flexibility, you can use [`export_entries_as_dduf`] and pass a list of files to include in the final DDUF file: +For more flexibility, use [`export_entries_as_dduf`] to explicitly specify a list of files to include in the final DDUF file: ```python # Export specific files from the local disk. @@ -117,7 +118,7 @@ For more flexibility, you can use [`export_entries_as_dduf`] and pass a list of ... ) ``` -This works well if you've already saved your model on the disk. But what if you have a model loaded in memory and want to serialize it directly into a DDUF file? `export_entries_as_dduf` lets you do that by providing a Python `generator` that tells how to serialize the data iteratively: +`export_entries_as_dduf` works well if you've already saved your model on the disk. But what if you have a model loaded in memory and want to serialize it directly into a DDUF file? `export_entries_as_dduf` lets you do that by providing a Python `generator` that tells how to serialize the data iteratively: ```python (...) @@ -146,14 +147,16 @@ ZIP provides several advantages: - Built-in file indexing - Wide language support -Why not use a TAR with a table of contents at the beginning of the archive? See explanation [in this comment](https://github.com/huggingface/huggingface_hub/pull/2692#issuecomment-2519863726). +### Why not use a TAR with a table of contents at the beginning of the archive? + +See the explanation in this [comment](https://github.com/huggingface/huggingface_hub/pull/2692#issuecomment-2519863726). ### Why no compression? -- Enables direct memory mapping of large files. -- Ensures consistent and predictable remote file access. -- Prevents CPU overhead during file reading. -- Maintains compatibility with safetensors. +- Enables direct memory mapping of large files +- Ensures consistent and predictable remote file access +- Prevents CPU overhead during file reading +- Maintains compatibility with safetensors ### Can I modify a DDUF file? @@ -161,6 +164,6 @@ No. For now, DDUF files are designed to be immutable. To update a model, create ### Which frameworks/apps support DDUFs? -- [diffusers](https://github.com/huggingface/diffusers) +- [Diffusers](https://github.com/huggingface/diffusers) -We are continuously reaching out to other libraries and frameworks, if you are interested in adding support for your project, open a Discussion in the [DDUF org](https://huggingface.co/spaces/DDUF/README/discussions). +We are constantly reaching out to other libraries and frameworks. If you are interested in adding support to your project, open a Discussion in the [DDUF org](https://huggingface.co/spaces/DDUF/README/discussions). From 1df74c68a3d1d5fa0bb3755bfd06dbeb671b6103 Mon Sep 17 00:00:00 2001 From: Wauplin Date: Fri, 13 Dec 2024 10:45:55 +0100 Subject: [PATCH 15/19] rmmap --- docs/hub/dduf.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/dduf.md b/docs/hub/dduf.md index 535a96bf3..ea8a366f8 100644 --- a/docs/hub/dduf.md +++ b/docs/hub/dduf.md @@ -42,7 +42,7 @@ Its key features include the following. ## Technical specifications Technically, a `.dduf` file **is** a [`.zip` archive](https://en.wikipedia.org/wiki/ZIP_(file_format)). By building on a universally supported file format, we ensure robust tooling already exists. However, some constraints are enforced to meet diffusion models' requirements: -- Data must be stored uncompressed (flag `0`), allowing mmap-compatibility. +- Data must be stored uncompressed (flag `0`), allowing lazy-loading using memory-mapping. - Data must be stored using ZIP64 protocol, enabling saving files above 4GB. - The archive can only contain `.json`, `.safetensors`, `.model` and `.txt` files. - A `model_index.json` file must be present at the root of the archive. It must contain a key-value mapping with metadata about the model and its components. From 2283b18d643c7961e06ce659c2ee4dee53b4f412 Mon Sep 17 00:00:00 2001 From: Wauplin Date: Fri, 13 Dec 2024 10:48:28 +0100 Subject: [PATCH 16/19] reduce overhead --- docs/hub/dduf.md | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/docs/hub/dduf.md b/docs/hub/dduf.md index ea8a366f8..d87be8c21 100644 --- a/docs/hub/dduf.md +++ b/docs/hub/dduf.md @@ -8,7 +8,7 @@ ## Overview -DDUF (**D**DUF’s **D**iffusion **U**nified **F**ormat) is a file format designed to make storing, distributing, and using diffusion models much easier. Built on the ZIP file format, DDUF offers a standardized, efficient, and flexible way to package all parts of a diffusion model into a single, easy-to-manage file. +DDUF (**D**DUF’s **D**iffusion **U**nified **F**ormat) is a single-file format for diffusion models that aims to unify the different model distribution methods and weight-saving formats by packaging all model components into a single file. It is language-agnostic and built to be parsable from a remote location without downloading the entire file. This work draws inspiration from the [GGUF](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) format. @@ -22,14 +22,6 @@ To create a widely adopted file format, we need early feedback from the communit -## Motivation - -Yet, another file format? Yes, but for good reasons! - -The primary goal of DDUF is to create a community-endorsed single-file format for diffusion models. Current model distribution methods often involve multiple separate files, different weight-saving formats, and managing files from various locations. DDUF aims to solve these challenges by packaging all model components into a single file, enforcing a consistent structure while being opinionated about saving formats. - -The DDUF format is also designed to be language-agnostic. While we currently provide tooling for the Python ecosystem, there's nothing stopping similar tools from being developed in JavaScript, Rust, C++, and other languages. Like GGUF or safetensors, DDUF is built to be parsable from a remote location without downloading the entire file, which will enable advanced support on the Hugging Face Hub. - Its key features include the following. 1. **Single file** packaging. From 87599e583b97b1cc7be4355677fe8ce63b8e82fa Mon Sep 17 00:00:00 2001 From: Wauplin Date: Fri, 13 Dec 2024 10:50:37 +0100 Subject: [PATCH 17/19] language --- docs/hub/dduf.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/hub/dduf.md b/docs/hub/dduf.md index d87be8c21..16d7e4405 100644 --- a/docs/hub/dduf.md +++ b/docs/hub/dduf.md @@ -27,9 +27,10 @@ Its key features include the following. 1. **Single file** packaging. 2. Based on **ZIP file format** to leverage existing tooling. 3. No compression, ensuring **`mmap` compatibility** for fast loading and saving. -4. **HTTP-friendly**: metadata and file structure can be fetched remotely using HTTP Range requests. -5. **Flexible**: each model component is stored in its own directory, following the current Diffusers structure. -6. **Safe**: uses [Safetensors](https://huggingface.co/docs/diffusers/using-diffusers/other-formats#safetensors) as a weight-saving format and prohibits nested directories to prevent ZIP bombs. +4. **Language-agnostic**: tooling can be implemented in Python, JavaScript, Rust, C++, etc. +5. **HTTP-friendly**: metadata and file structure can be fetched remotely using HTTP Range requests. +6. **Flexible**: each model component is stored in its own directory, following the current Diffusers structure. +7. **Safe**: uses [Safetensors](https://huggingface.co/docs/diffusers/using-diffusers/other-formats#safetensors) as a weight-saving format and prohibits nested directories to prevent ZIP bombs. ## Technical specifications From 8480983f4c0c001e0dd27b74b2c0280595bb7f45 Mon Sep 17 00:00:00 2001 From: Wauplin Date: Fri, 13 Dec 2024 10:58:54 +0100 Subject: [PATCH 18/19] diffusers example --- docs/hub/dduf.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/hub/dduf.md b/docs/hub/dduf.md index 16d7e4405..04fb8120f 100644 --- a/docs/hub/dduf.md +++ b/docs/hub/dduf.md @@ -130,6 +130,23 @@ For more flexibility, use [`export_entries_as_dduf`] to explicitly specify a lis >>> export_entries_as_dduf(dduf_path="my-cool-diffusion-model.dduf", entries=as_entries(pipe)) ``` +### Loading a pipeline with Diffusers + +Diffusers has a built-in integration for DDUF files. Here is an example on how to load a pipeline from a stored checkpoint on the Hub: + +```py +from diffusers import DiffusionPipeline +import torch + +pipe = DiffusionPipeline.from_pretrained( + "DDUF/FLUX.1-dev-DDUF", dduf_file="FLUX.1-dev.dduf", torch_dtype=torch.bfloat16 +).to("cuda") +image = pipe( + "photo a cat holding a sign that says Diffusers", num_inference_steps=50, guidance_scale=3.5 +).images[0] +image.save("cat.png") +``` + ## F.A.Q. ### Why build on top of ZIP? From 91248209d72a22cfb1ac2f886c2f11c1d732e43b Mon Sep 17 00:00:00 2001 From: Wauplin Date: Fri, 13 Dec 2024 10:59:25 +0100 Subject: [PATCH 19/19] titlte --- docs/hub/dduf.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/dduf.md b/docs/hub/dduf.md index 04fb8120f..bd887c984 100644 --- a/docs/hub/dduf.md +++ b/docs/hub/dduf.md @@ -130,7 +130,7 @@ For more flexibility, use [`export_entries_as_dduf`] to explicitly specify a lis >>> export_entries_as_dduf(dduf_path="my-cool-diffusion-model.dduf", entries=as_entries(pipe)) ``` -### Loading a pipeline with Diffusers +### Loading a DDUF file with Diffusers Diffusers has a built-in integration for DDUF files. Here is an example on how to load a pipeline from a stored checkpoint on the Hub: