Add testing for ComfyUI examples #95
Conversation
There was a problem hiding this comment.
Adds simple test for examples from https://github.com/comfyanonymous/ComfyUI_examples, just checks that loading the workflow and converting to prompt generates the same output as in the examples.
- Moved png & flac metadata extraction to own files so they can be imported without API etc being set up
LGTM
Add script
npm run test:generate:examples- This requires the ComfyUI examples repo to be cloned
- This extracts metadata from the examples and stores them in workflow/examples, noting any changes/issues
- I couldn't decide between storing them in this repo vs having an action to extract each time was better, however thought that having hard copies would be useful for spotting differences in future. Happy to reconsider.
I think we should do similar thing as we did for test:generate, i.e. ignore the examples. If we want to verify things are not changed, we should add the git ref when checking out https://github.com/comfyanonymous/ComfyUI_examples. So things will only break when we update the ref.
- Add
chalkas dev dep for color output, this was already included via other packages (e.g. jest)
LGTM
- Test currently provide a few simple "fixes" where new inputs have been added since their creation (these were backwards compatible)
Can you move the legacy prompt support to main app's code?
- Tests currently skip examples with group nodes as these don't generate the same ID as saved in the workflow so the examples will need regenerating
LGTM
Some potential TODOs:
- Add the test prep command to Github action workflow
- Update README about the extra test prep
|
|
||
| const dirname = path.dirname(fileURLToPath(import.meta.url)); | ||
| const repoPath = | ||
| process.env.EXAMPLE_REPO_PATH || path.resolve(dirname, "ComfyUI_examples"); |
There was a problem hiding this comment.
We should add EXAMPLE_REPO_PATH to .env_example and init dotenv here.
import dotenv from "dotenv";
dotenv.config();
The main app (litegraph) adds these values when generating using old example workflows (the problem is that when doing the comparison they dont exist in the example, but will in the test output) so adding the defaults at the point of comparison is the simplest approach. Adding/changing inputs isnt particularly common, and does need to be done carefully so in situations where it does happen, I think the tests failing and requiring double checking & updating to add the new default is probably fine? |
Adds simple test for examples from https://github.com/comfyanonymous/ComfyUI_examples, just checks that loading the workflow and converting to prompt generates the same output as in the examples.
npm run test:generate:examples- This requires the ComfyUI examples repo to be clonedchalkas dev dep for color output, this was already included via other packages (e.g. jest)