A multilingual support classifier looks convincing in the demo. Then German tickets are routed to the wrong teams, new product names confuse the model, and nobody can say whether French performance was ever tested separately.
The immediate proposal is often to label more data. Engineering asks for budget. Operations starts comparing annotation providers. Product expects the next training run to fix the issue.
But volume may not be the constraint. The labels could encode an unclear policy, the test set may not represent production, or the source documents behind a retrieval system may simply be stale. Scaling an annotation queue before identifying the failure mode creates expensive certainty around the wrong problem.
The real investment decision is whether the company needs a repeatable data, evaluation and human-review capability—and who should own it.
The bottleneck is often judgement, not label volume
Data annotation is essential for many supervised machine-learning systems. It can also support foundation-model evaluation, retrieval testing, safety review and human-feedback workflows.
It does not, however, power every AI product in the same way.
A product built on a pre-trained model may fail because retrieval returns irrelevant documents. A document assistant may have poor permission controls. A recommendation system may optimise a proxy metric that has little commercial value. A rules-based risk engine may need policy review rather than another training set.
Before funding annotation, identify the actual bottleneck:
- Missing examples: the model has not seen enough representative cases.
- Unclear task: people disagree about what the correct answer should be.
- Weak evaluation: the team cannot measure performance under expected-use conditions.
- Data quality: records are incomplete, duplicated, outdated or incorrectly joined.
- Retrieval failure: the right source material exists but is not being found or ranked.
- Workflow failure: the model output is acceptable, but escalation and human review are poorly designed.
- Integration failure: permissions, latency, monitoring or product instrumentation are blocking release.
More labels are a sensible response only to some of these conditions. They cannot repair an invalid success metric or an undefined product decision.
Define the decision before defining the workforce
Start with one sentence: For each input, what decision, prediction or recommendation must the system produce?
“Classify documents” is too broad. “Extract the renewal date from English and German commercial leases, flagging absent or ambiguous dates for legal review” is operationally useful. It defines the unit of work, languages, escalation condition and required expertise.
Then determine what the label represents.
Observable facts
Examples include whether a traffic light is visible or whether a document contains a signature. Generalist annotators may handle these tasks after training, although edge cases still need escalation.
Expert judgement
Examples include interpreting a medical image, identifying a contractual obligation or assessing whether a transaction pattern merits investigation. Domain specialists are likely to be necessary, at least for guideline design, difficult cases and adjudication.
Policy choices
Examples include deciding which support queue owns a mixed-intent ticket or what language should be treated as abusive in a community. There may be no objective ground truth. Product, risk or policy owners must make and document the decision.
This distinction changes the hiring plan. If reasonable experts disagree, adding more low-cost annotators may increase inconsistency rather than improve the dataset. The team may need clearer classes, an “ambiguous” state, confidence scoring or a named adjudication owner.
Put the project through an annotation readiness gate
A limited paid pilot is usually a better first commitment than a large workforce or long provider contract. Use seven gates to decide whether the work is ready to scale.
Scroll horizontally to see all columns.
| Gate | Decision test | Evidence required |
|---|---|---|
| 1. Task clarity | Can two qualified people explain each label and its business meaning? | Taxonomy, positive and negative examples, exclusions and ambiguity states |
| 2. Risk and privacy | What happens when a label or prediction is wrong, and is personal data necessary? | Risk register, access model, minimisation rationale and retention rules |
| 3. Sample design | Does the pilot cover the conditions that matter in production? | Coverage by language, region, customer type, device, format and costly edge case |
| 4. Human capability | Which decisions require domain or language expertise? | Role definitions, training material and escalation path |
| 5. Quality control | How will errors and legitimate disagreements be found? | Gold set, sampled double-labelling, adjudication rules and QA reporting |
| 6. Model value | Does better data improve a production-relevant outcome? | Frozen test set, subgroup results and comparison with the current baseline |
| 7. Operating model | Is the work durable and strategic enough to own internally? | Volume forecast, sensitivity assessment, expected rework and staffing plan |
A project that fails the task-clarity gate is not ready for annotation procurement. One that fails the model-value gate may have clean labels but no viable business case. A pilot should expose these outcomes cheaply.
The full cost is not the price per label. It includes guideline development, secure access, onboarding, QA, adjudication, project management, dataset versioning, model experiments and rework when the task changes.
Worked example: multilingual support routing
Consider a Europe-wide B2B platform routing English, German, French and Spanish support tickets. The company has a backlog of 100,000 historical tickets and is considering sending all of them for labelling.
A better pilot would begin with high-impact intents: account access, billing, data incidents, product defects and cancellation requests. The sample would be stratified by language, product area, customer segment and ticket complexity rather than drawn randomly from the entire archive.
The team would then:
- Define whether each ticket may have one intent, several intents or an ambiguous state.
- Separate the customer’s stated topic from the operational team that should receive it.
- Give annotators examples of mixed-intent, low-context and incorrectly translated tickets.
- Reserve a versioned test set that is not used to revise the model.
- Double-label a risk-informed subset and send meaningful disagreements to domain owners.
- Report results by language and intent, not only as one aggregate score.
- Compare model metrics with routing corrections, escalation rates and handling time in the actual workflow.
The pilot could lead to four different decisions:
- Low disagreement and meaningful operational improvement: scale the workflow.
- High disagreement between qualified reviewers: redesign the taxonomy or policy before scaling.
- Consistent labels but little model improvement: investigate features, retrieval, model choice or integration.
- Improvement in some languages only: add targeted language or domain expertise rather than expanding all annotation equally.
The point is not to prove that annotation works. It is to discover where human judgement creates measurable value and what operating capacity that requires.
Buy evidence, not a headline accuracy claim
A credible evaluation set must represent expected-use conditions. NIST’s AI Risk Management Framework calls for clearly defined, realistic test sets, documented test methodology and, where appropriate, results disaggregated across relevant segments.
“Held out from training” is therefore not enough. If the test set excludes difficult document formats, regional language or rare high-cost failures, it can remain technically clean while giving a misleading picture of production readiness.
The labels themselves also need auditing. A study covering ten widely used vision, language and audio datasets estimated at least 3.3% label errors on average and showed that correcting labels could change benchmark rankings. That does not establish an expected error rate for a company’s dataset. It does show why model comparisons are weak when the evaluation labels have not been checked.
Use this procurement and delivery checklist:
A concise dataset record is part of the product, not optional paperwork. Google’s Data Cards framework provides a useful reference for documenting sources, collection and annotation methods, intended uses and decisions affecting performance.
Choose the operating model around sensitivity and continuity
The build-or-buy decision is not binary. Most teams need a combination of internal accountability and flexible delivery capacity.
Scroll horizontally to see all columns.
| Model | Best fit | Main limitation |
|---|---|---|
| Internal team | Sensitive data, stable demand, proprietary domain knowledge and frequent product iteration | Slower to establish; fixed capacity can be inefficient when volume changes |
| Senior specialist hires | The workflow, evaluation harness or data platform does not yet exist | Specialists still need accountable product, privacy and domain stakeholders |
| Managed annotation provider | Well-defined, repeatable work with significant throughput or language coverage | Weak task design can be industrialised quickly; provider metrics may not reflect model value |
| Hybrid model | Internal ownership with variable external production capacity | Requires disciplined interfaces, documentation and access controls |
An internal capability does not mean every label must be created by an employee. It means the company retains ownership of intended use, task definition, risk decisions, acceptance criteria and the link between data changes and product outcomes.
A managed provider can be effective when guidelines are stable and volume is material. It is a poor substitute for an AI product owner or applied ML specialist when the team is still discovering what should be measured.
Similarly, a senior hire is not the right answer for a one-off, low-risk labelling batch with a settled specification. In that case, a provider may be faster and commercially cleaner.
When hiring, match the role to the missing capability:
- Applied AI engineer: evaluation harness, human-feedback loop, model experiments, monitoring and product integration.
- ML or data engineer: secure ingestion, lineage, versioning, quality tests, access controls and retraining pipelines.
- AI product lead: intended use, taxonomy, acceptance criteria, risk trade-offs and escalation design.
- Data-quality or operations lead: reviewer onboarding, work allocation, QA sampling, adjudication and reporting.
Hiring a generic “AI expert” without naming these outcomes usually produces an unfocused search.
Treat European risk and privacy as design inputs
For European deployments, legal and privacy review should begin before annotation data is distributed to a workforce or provider.
The EU AI Act explicitly includes collection, data origin, annotation, labelling, cleaning and enrichment within the data-governance requirements for covered high-risk systems. It also addresses relevance, representativeness, errors and completeness in relation to intended purpose under Article 10.
Recruitment and selection uses—including targeted job advertising, application filtering and candidate evaluation—are listed in Annex III. Classification and obligations still depend on the system and its intended use, so this requires case-specific legal assessment. Annotation controls alone do not establish compliance.
Data minimisation matters as much as data quality. The UK ICO’s AI and data-protection guidance advises organisations not to use more personal data than necessary and identifies options such as synthetic data and federated learning.
Before collecting more records, ask whether the same objective could be achieved through better sampling, clearer guidelines, redaction, tighter access, synthetic examples or a narrower intended use.
Turn the unresolved decision into a specialist brief
A useful hiring brief should state what must exist after the first delivery phase. For example:
Establish a versioned evaluation and human-review workflow for a multilingual support classifier. Define production-relevant test slices, implement secure data ingestion, document the label taxonomy, create QA and adjudication reporting, and recommend the long-term operating model.
Add the data types, languages, sensitivity, current stack, internal stakeholders and decision deadline. State whether the specialist is expected to build the workflow, lead internal staff, select providers or remain accountable through production release.
Deeptal connects companies with senior European specialists across applied AI, data engineering, AI product and related operational disciplines. It is not a crowd-labelling service, and a specialist cannot replace accountable legal, privacy, security and domain owners.
If the missing piece is senior ownership rather than annotation throughput, start with that outcome-based brief. For a qualified brief, Deeptal typically prepares a human-reviewed initial shortlist within two business days. Selected specialists typically start within 7–14 days, subject to fit, availability, interviews and agreed terms; Deeptal can also handle contracts, payroll, compliance administration and consolidated monthly billing without recruitment fees.
