We rebuilt PromptSewa — and your prompts were reset

Lesson 13 · 5 min

Classification and labelling prompts

Fixed labels, a rule for uncertainty, and a reason you can audit.

Classification prompts map an input to one of a small set of labels: support ticket category, sentiment, intent, priority, content type. They are deceptively simple. The hard part is not the model; it is the taxonomy. If two labels overlap or a case fits none, the model will pick something and you will never know it was a coin flip.

Define the label set exactly

List the labels and what each one means in one line. Use the labels you will actually act on, not a taxonomy borrowed from a blog post. If a label exists but no workflow uses it, remove it. A short, well-defined set produces consistent output; a long set with fuzzy boundaries produces confident nonsense that looks like a distribution.

Handle the middle

Real inputs sit between categories. Add an explicit fallback: "unknown" or "needs human review". A fallback is not a failure — it is the honest output for an input the taxonomy does not cover. Without it, every ambiguous case is silently forced into the nearest label, and your metrics look clean while the queue fills with misrouted work.

Ask for the reason, in one line

A label plus a one-line reason gives a reviewer something to audit. It also improves consistency, because the model has to commit to the rule it applied. Keep the reason short and structural: "mentions a refund and an order number" is checkable; three sentences of justification is not. For high-volume pipelines, a reason field can be dropped later once the prompt has proven stable.

Example: "Classify {{message}} as billing, technical, account or other. Reply with JSON: label (one of the four) and reason (max 12 words). If two labels apply, choose the one the customer asks about first; if none apply, use other."

Measure agreement, not just accuracy

A classifier that agrees with your reviewer nine times in ten is useful. To know that, you need a labelled sample, and you need to re-check the sample when the prompt or the taxonomy changes. Track disagreements by label: if every disagreement is on one label, the definition is the problem, not the model. This is the cheapest way to improve a taxonomy, and it is almost always a taxonomy problem.

Keep the input clean

Trim quoted threads, signatures and boilerplate before classifying. Long irrelevant context dilutes the signal and pushes the model toward the most common label in the conversation. If the input is a thread, say which message is being classified. Cheap preprocessing raises agreement more than most prompt wording changes.

Checklist

  • Is every label defined in one line?
  • Is there an honest fallback label?
  • Does the output include a short, checkable reason?
  • Have you measured agreement on a labelled sample?

Finished this lesson?

Mark it complete to generate a certificate you can print or save as a PDF. We ask for an email only so you can be reached — no account is created.

All lessons