ChatGPT Prompts HubSearch
Menu

Guide

How to Build a Prompt Regression Test Table

A prompt can improve on one example and quietly fail on another. A small regression table keeps the same cases and success checks beside each version, so you can see what changed instead of trusting the latest polished answer.

Check whether a reusable prompt still works after you edit its instructions, context, examples, or answer format.

Use this when

  • A role or task prompt will be reused across similar briefs, records, lessons, analyses, or reviews.
  • A prompt edit fixes one answer but may change tone, factual discipline, refusal behavior, or output shape elsewhere.
  • Several people need a shared reason for accepting or rejecting a new prompt version.

Pause when

  • The task is still undefined and no one can state what a useful answer must contain.
  • The cases include private, regulated, or consequential data that has not been approved for testing.
  • A table score would be used to replace domain review, factual verification, policy checks, or a final human decision.

How to think about it

Write observable pass criteria before running the prompt

A useful criterion describes something a reviewer can point to in the answer. Instead of good tone, write uses the requested audience vocabulary without inventing familiarity. Instead of accurate, write every factual claim is supported by the supplied notes or labeled as a question. Keep the list short enough that the same reviewer can apply it consistently.

Separate answer-shape checks from judgment checks. Required headings, table columns, length limits, and uncertainty labels can be inspected directly. Whether advice is appropriate, a claim is current, or a decision is fair may need an owner with domain context. The test table should name that owner rather than hiding the handoff inside a single pass score.

Use cases that fail in different ways

The normal case proves the basic workflow. The edge case stretches one real variable such as a long reference bundle, an unusual audience, or two competing priorities. The missing-context case checks whether the prompt asks for what it needs instead of inventing it. The conflicting-instruction case checks which rule wins. The format-stress case checks whether the answer remains inspectable when the material is messy.

Use realistic but safe examples. Remove private identifiers and avoid copying production secrets into a convenience test. When a prompt controls tools, files, accounts, publication, or decisions, add cases that confirm it asks before acting and stops when permission or evidence is missing.

Record the failure, not just the score

A pass or fail cell is too thin for revision. Save the prompt version, test input, answer excerpt or artifact location, failed criterion, and reviewer note. Describe the observable problem: omitted one required column, added a claim not found in the notes, followed the later conflicting instruction, or returned prose instead of the requested table.

Make one prompt change per test round when possible. If you change the role, context order, examples, stop rules, and output schema together, the rerun cannot show which edit helped. A narrow revision record also makes it easier to reverse a change that improves one case while breaking another.

Rerun the same table after every meaningful edit

Regression testing depends on keeping the cases stable long enough to compare versions. Run the edited prompt on the same five inputs and apply the same criteria. Add a new case when a real failure appears, but do not remove an inconvenient case merely because the newest version fails it.

Model behavior can vary, so one run is not a guarantee. For important workflows, repeat sensitive cases and review the spread of answers. Treat the table as a practical change log and review aid, not proof that a prompt is universally safe, factual, or reliable.

Checklist

Task
Name the repeated job, user, allowed material, and final human owner.
Pass criteria
Write two to five observable checks before looking at the answer.
Normal case
Use a complete, representative input that should exercise the ordinary workflow.
Edge case
Stretch one realistic variable without changing every condition at once.
Missing context
Remove a required fact and check whether the prompt asks instead of guessing.
Conflicting instruction
Add a lower-priority request that conflicts with a stated rule and record which one wins.
Format stress
Use messy or long material and verify that the required answer shape remains reviewable.
Failure note
Record the exact criterion, answer evidence, reviewer, and prompt version.
One change
Revise one instruction, example, boundary, or output requirement at a time.
Same-case rerun
Rerun every saved case and keep regressions visible even when another case improves.

Worksheet

Test identity
Case name, prompt version, date, and reviewerMissing deadline | v4 | 2026-07-18 | content lead
Input
Paste or link the approved test inputA complete brief except for the publication deadline
Pass criteria
List the observable conditions for this caseAsks for deadline; does not invent one; preserves required headings
Result
Pass, fail, or needs domain review plus the exact evidenceFail: inserted Friday without support in the notes
Revision
Record one prompt change and the same-case rerun outcomeAdded missing-input stop rule; rerun passed all three criteria

Copy a working starter

Copy prompt regression table

Prompt regression test table
Prompt version:
Reviewer and domain owner:
Pass criteria:
| Case | Approved input | Expected behavior | Observed result | Pass / fail | Exact failure |
| Normal case | | | | | |
| Edge case | | | | | |
| Missing-context case | | | | | |
| Conflicting-instruction case | | | | | |
| Format-stress case | | | | | |
One prompt change:
Same five cases rerun:
New regression to keep:
Human approval still required for:

Reference notes

  • OpenAI prompt engineering guideOpenAI. Clear instructions, relevant context, examples, and explicit output requirements that can become test criteria.Use it for clear instructions, relevant context, examples, and explicit output requirements; do not treat any one prompt pattern as a guarantee.
  • OpenAI evaluation best practicesOpenAI. Task-specific success criteria, representative datasets, failure analysis, and continuous evaluation after changes.Use it for defining success criteria, representative test cases, failure analysis, and continuous evaluation; this page does not require API evaluation tooling.