[DA-253] storage api methods with asset service and local filesystem backends #199
Conversation
… LocalFileBackend. Some asset storage operations implemented.
…lement some create and load operations.
…get_video_asset for Asset Service backend isn't working.
|
You have successfully added a new SonarCloud configuration ``. As part of the setup process, we have scanned this repository and found no existing alerts. In the future, you will see all code scanning alerts on the repository Security tab. |
| proj = {"id": proj_id, | ||
| "title": title, | ||
| "file": {"id": proj_file_id}} | ||
| output_path = os.path.join(LocalFileBackend._projects_root, proj_id, proj_file_id) |
There was a problem hiding this comment.
Projects are currently stored in subfolders based on the project name which would be nice to preserve so people can browse their Google/local drive and find things easy
https://github.com/Stability-AI/stability-sdk/blob/anima_stable/src/stability_sdk/animation_ui.py#L424
| @staticmethod | ||
| def add_asset_metadata(project_id: str, asset_id: str, mime_type: str, filename: str, project_key: str = None) -> None: | ||
| # metadata_index = self.load_metadata_index() # I assume metadata is updated by each operation | ||
| if project_id not in Project._metadata_index: | ||
| Project._metadata_index[project_id] = {} | ||
| Project._metadata_index[project_id][asset_id] = { | ||
| "mime_type": mime_type | ||
| } | ||
| if filename is not None: | ||
| Project._metadata_index[project_id][asset_id]["filename"] = filename | ||
| if project_key is not None: | ||
| Project._metadata_index[project_id][project_key] = asset_id | ||
| Project.save_metadata_index() | ||
|
|
||
| @staticmethod | ||
| def delete_project_metadata(project_id: str) -> None: | ||
| Project._metadata_index.pop(project_id, None) | ||
|
|
||
| @classmethod | ||
| def save_metadata_index(cls, metadata_index: dict = None) -> None: | ||
| if metadata_index is None: | ||
| metadata_index = cls._metadata_index | ||
| index_file = f"metadata_index.json" | ||
| with open(index_file, "w") as f: | ||
| json.dump(metadata_index, f) | ||
|
|
||
| @classmethod | ||
| def load_metadata_index(cls) -> dict: | ||
| index_file = "metadata_index.json" | ||
| if os.path.exists(index_file): | ||
| with open(index_file, "r") as f: | ||
| metadata_index = json.load(f) | ||
| return metadata_index | ||
| return {} |
There was a problem hiding this comment.
Is the metadata index necessary? Could it be implicit based on what's in the folders?
There was a problem hiding this comment.
My thinking on this was basically that for future purposes, I have an interest in representing this information in application memory. This metadata JSON saved to disk is an easy dump for bootstrapping reasons since we don't have a UI to look at what the in-memory representation.
I agree that it should currently be considered a cache at best, since someone could potentially create/delete/etc. projects+assets by other means (, and the metadata isn't saying much else interesting aside from what can be found in projects+assets).. and so the sync/refresh strategy needs some work. I've planned to at least add a script to download all assets and rebuild metadata, but haven't written it yet.
For persistent storage, I'd like to see everything kept in the project+assets rather than solely metadata.
…sets are treated as binary data.
…ject() for local filesystem has empty id.
|
Kudos, SonarCloud Quality Gate passed!
|
This adds an abstract StorageBackend class with AssetServiceBackend and LocalFileBackend. One backend can be chosen as the primary for reads, while any number of backends can be chosen for writes.
This initial API is modeled to support the existing asset service functionality for projects and to prioritize similarity between files saved locally and the asset service (to serve as a mirror). As such, the file organization is flat and uses uuid's. Future plans may involve a more convenient structure - and may leverage a suitable existing library.