Does prompt make sense?

[vwxyzjn]

This PR includes a simple script to judge the quality of the SFT prompts in the dataset.

[hamishivi]

As a very basic first pass this makes sense, but I wonder if:
(a) we can be more specific / ask for fine-grained scores (e.g., coherence, length, etc.). I feel like recent llm-as-a-judge work is trending to more fine-grained scores (e.g. ultrafeedback finegrained vs overall).
(b) If we can make this more quantitative somehow. Can we have some analysis of false positives or similar? Or maybe some small val set that we can reason about and quality-check with?

[hamishivi]

Maybe we could make some sort of experimental folder for a rough script like this? Unsure, this sort of thing feels way more rough that e.g. the submit_eval/finetune scripts.

[hamishivi]

should we use 4o?