A handful of fixed cases and a pass mark beat a feeling. Build a small test set and use it.
A prompt is a component, and components need tests. You do not need a research harness; you need a small set of cases you run every time you change the prompt, with a stated pass condition. Ten minutes of setup saves a week of arguing about whether the new wording is better.
Build the smallest useful set
Start with six to ten inputs. Include a typical case, a long input, an empty or near-empty input, an input with the wrong kind of content, and one genuinely ambiguous case. Save them beside the prompt. The point is coverage of failure modes, not statistical significance. When a real failure appears in production, add that input to the set — the test suite grows from experience rather than imagination.
Define pass before you run
Write the pass condition down before looking at outputs, or you will rationalise whatever you see. Conditions should be observable: every required field present, no invented numbers, under 120 words, correct label on all ten cases. A pass condition that requires a subjective judgement is allowed, but it needs a second reader — otherwise you are grading your own homework.
Example: a classification prompt with ten labelled examples. Pass means at least nine correct and no confident wrong label on the ambiguous case. Below that, the prompt is not ready, regardless of how good the prose reads.
Test the boundaries, not the average
Models are usually fine on typical inputs and unreliable at the edges: very short text, text with conflicting instructions, text in another language, text that contains the answer twice. A prompt that passes ten easy cases and fails the one empty input will fail in production, because empty inputs happen. Deliberately include the ugly cases you would rather not look at.
Compare, do not admire
When you change a prompt, run the old and the new against the same set and compare pass rates. This is the only reliable way to know whether a change helped. It also protects against the common illusion where a longer prompt feels more thorough and performs worse. Keep the comparison cheap: the same script, the same cases, a two-column result.
Record what you learned
A short note per version — what changed, what improved, what broke — turns a folder of prompts into institutional knowledge. It is also what lets someone else maintain the prompt without re-deriving every decision. For a marketplace listing, that history is a selling point: buyers can see the prompt has been tested, not just written.
Checklist
- Do you have six to ten saved cases including the ugly ones?
- Is the pass condition written down before running?
- Do you compare old and new on the same set?
- Does every production failure become a new case?