A/B Testing Notebook #155

dmarx · 2022-12-11T18:23:05Z

To do:

fix save_favorite_only behavior
Make notebook more robust to errors, resuming
- read outcomes.log if it exists, use it to populate running_score, SAMPLE_IDX, etc.

richer logging
- log time taken for generation
  - maybe writing images to gdrive is slower than writing them locally?
- log time taken by user to choose
- log incidence of safety mitigation trigger
- log incidence of errors and conditions that emitted them
- record "pass" - i.e. when user regenerates without selecting a favorite
optionally use different seed for samples
optionally request multiple samples per test case
use plotly (or some such) to permit user to interactively facet histogram on combinatorial options
- e.g. to see if there are specific prompts for which one or other test case is preferred
- check old checkins project for a potential solution here
  - https://medium.com/plotly/introducing-jupyterdash-811f1f57c02e
  - https://github.com/dmarx/checkin/blob/master/checkin/datavis%20sandbox.ipynb
Additional rating dimensions
- aesthetic preference
- relevance to prompt
- least weird / most devoid of artifacts
- best fit to style
- best fit to content
- best foreground
- best background
- best composition
- best small details
- best texture
- most photorealistic
- most artistic