Collecting the cases that break products in production

Average performance keeps improving while the same category of failure keeps reaching users. Those failures live in the thin tail of the distribution, and random collection barely touches it.

Four ways to find them

• Log every production failure with enough context to reproduce it
• Treat low confidence predictions as a queue for human review and collection
• Stage rare scenarios rather than waiting for them
• Commission capture in conditions your current set under represents

Keep the evaluation split honest

Hard cases belong in both production and evaluation, but not the same ones. An evaluation set that only holds easy examples will report improvement that users never feel.

Prioritise by consequence

Not every rare case deserves budget. Rank by how much damage the failure does, then collect against that ranking rather than by how interesting the case is.

Practical checklist

• Define the acceptance rule before any volume starts
• Review a small pilot before committing the full budget
• Track errors by category, language and reviewer
• Keep consent, source notes and version history with the files

Before you ask for a quote

A clear brief saves days. Share a sample file, target language or region, expected volume, deadline, quality threshold and any privacy restrictions. A supplier can then price the work on real effort rather than assumptions.

• Which languages, markets or user groups must be represented?
• What format does the final file need to arrive in?
• Who will approve ambiguous cases during the pilot?