In short
Good annotation guidelines define each label with positive and negative examples, state rules for ambiguous cases, give a decision order for conflicts and record a version history.
The guideline is the actual specification of the dataset. Everything downstream inherits whatever it left unclear.
What belongs in it
- A one line definition of each label, in plain language
- Three examples of what counts and three of what does not
- A rule for the ambiguous middle, even an arbitrary one
- An order of precedence when two labels both apply
- A version number and a change log
Test it on a stranger
Give the document to someone outside the project and have them label fifty items. Every question they ask is a gap. Fix the document rather than answering in chat, where the answer is lost.
Version changes carefully
Changing a rule mid project splits the dataset into two conventions. Record which batches used which version and decide deliberately whether earlier work needs revisiting.
Practical checklist
- Define the acceptance rule before any volume starts
- Review a small pilot before committing the full budget
- Track errors by category, language and reviewer
- Keep consent, source notes and version history with the files
Before you ask for a quote
A clear brief saves days. Share a sample file, target language or region, expected volume, deadline, quality threshold and any privacy restrictions. A supplier can then price the work on real effort rather than assumptions.
- Which languages, markets or user groups must be represented?
- What format does the final file need to arrive in?
- Who will approve ambiguous cases during the pilot?
- guidelines
- quality
- process
Need this done rather than read about it?
We run collection, annotation, transcription and localization projects for teams who would rather spend their time on the product.
Start a conversation