TQE Systems, Validation and Verification

When do Translation Quality Scores Signal “Fitness for Purpose”?

Translation quality scores are supposed to predict success. But what happens when translations pass every quality check—and still fail in the market? The gap between measured quality and actual fitness reveals a fundamental flaw in how most organizations validate their translation workflows.

The stakes for translation quality validation have escalated significantly. Regulatory frameworks in medical devices and pharmaceuticals now require documented evidence that translations enable safe use—not just linguistic accuracy. AI and machine translation have compressed production timelines while introducing new uncertainty about output quality. Global brands face amplified reputational risk in markets where a single mistranslation can trigger viral social media backlash. Meanwhile, distributed teams and external vendors make specification alignment harder to maintain. Organizations can no longer afford to discover validation failures only after deployment, when correction costs multiply exponentially.

This article argues for a clear conceptual separation between verification and validation, and for applying that separation consistently at two different logical levels: the translation product and the translation quality evaluation (TQE) system that evaluates it. Without this separation, procedural compliance can be easily mistaken for effectiveness, and confidence in quality decisions is weakened.

Verification and Validation: The Core Distinction

In essence, verification is about checking compliance with specifications (stated and operationalized requirements) and validation is about checking fulfillment of actual requirements (stakeholder needs and expectations). Verification operates within the requirements space, while validation requires contextual evidence.

The distinction between meeting specifications and meeting requirements is well established in ISO quality management standards, yet its implications for translation quality evaluation are often underappreciated. A product can fully comply with its specifications and still fail to meet user needs.

This failure can arise from an array of causes, such as specifications that were incomplete or misaligned with requirements, or the use of a system that was not a reliable predictor of success.

When a metric is implemented incorrectly, scoring rules are applied inconsistently, or translators, evaluators, and validators rely on mismatched specifications, the issue is one of verification failure.

By contrast, when a metric is correctly implemented but is based on specifications that do not or no longer reflect current user requirements, the resulting scores may not support fitness for intended use.

Reliability issues are particularly important in this context: even a well-designed and correctly implemented TQE system cannot support valid decisions if its results are unstable. For example, inadequate evaluator training may lead to poor inter-rater or intra-rater agreement, undermining confidence in quality scores.

The following example illustrates product-level validation failure despite both product and system verification success. A healthcare organization consistently achieved 95% quality scores on patient-facing medication instructions using a rigorously implemented TQE system. Post-deployment analysis revealed critical comprehension failures: patients misunderstood dosing schedules and contraindication warnings. The specifications were being met—terminology was consistent, style guides were followed, error counts were low. But the specifications didn’t reflect how patients actually needed to process safety-critical information under cognitive load, time pressure, or health literacy constraints. The failure wasn’t procedural. It was a validation gap that specification compliance couldn’t detect.

Two Objects, Two Levels of Assurance: Product vs. System

To complicate matters, another source of confusion in translation quality discussions is the failure to distinguish between two different objects: 1) the translation product (the translated content) and 2) the TQE system (the metric, dimensions, weights, thresholds, sampling, and evaluators used to assess that content).

Verification and validation apply to both—but they do so at different logical levels. Activities at one level cannot substitute for activities at the other.

Product-Level Verification and Validation

Product Verification: Conformance to Specifications

At the product level, verification addresses the question of whether a translation conforms to its project specifications. Project specifications typically include terminology requirements, style guide adherence, and process constraints, such as revision requirements or use or non-use of MT.

Product verification checks whether such specifications have been implemented correctly. This is the domain of linguistic checks, QA tools, and analytic quality evaluation using metrics such as MQM.

For further detail on MQM dimensions, error taxonomies, and metric design, see resources published by the MQM Council.

It is critical to note that verification operates entirely within the space of specifications. It does not directly assess stakeholder needs, as it assumes that the specifications adequately represent those needs.

Product Validation: Fitness for Purpose

Product validation addresses a different question, namely, whether a translation is fit for its intended communicative purpose and usable by its target audience. This question cannot be answered by specification compliance alone. Product validation requires evidence from outside the verification process, such as:

• Task success rates,
• Stakeholder acceptance,
• Real-world deployment outcomes.

Product validation may occur occasionally rather than systematically, and when it fails, it should trigger an investigation on the root causes: Were the specifications implemented correctly? Are the specifications still appropriate? Have requirements changed?

System-Level Verification and Validation

TQE System Verification: Correct Implementation

At the system level, verification addresses whether a TQE system has been implemented and applied as defined. This includes confirming that processes are consistent and repeatable with:

• Metrics are applied as designed.
• Scoring rules are followed.
• Evaluators are trained to apply the metric consistently.

System verification ensures procedural correctness. It does not answer whether the system measures what actually matters.

TQE System Validation: Confidence for Decision-Making

System validation addresses a hard question: whether the TQE system reliably supports correct quality-related decisions for a given context. In practice, this means addressing whether:

• High scores reliably correspond to translations that meet stakeholder requirements.
• Low scores reliably flag translations that do not.
• Decision thresholds (publish, revise, reject) lead to appropriate outcomes.

Answering these questions requires meta-evidence, which emerges from outside the specifications space:

• Expert review of quality decisions,
• Alignment between evaluation outcomes and user acceptance,
• Correlation between scores and downstream outcomes (such as task success, user complaints, support tickets)

A TQE system can be fully verified and still be ineffective if it has never been validated for the decisions that it informs.

Risk, Uncertainty, and the Limits of Scores

When a Passing Score Fails to Meet Users Needs and Expectations (The Disney Problem)

Consider a scenario where a translation complies with all specifications. It is evaluated by a correctly implemented TQE system. It passes all thresholds. And yet… users reject it. Nothing has gone wrong at the level of verification. The failure occurs at the level of validation. This can happen when:

• Specifications no longer reflect user needs, leading to costly post-launch revisions or brand reputation damage in key markets.
• Error weights do not reflect real risk.
• Sampling introduces excessive uncertainty, potentially exposing entire product lines to undetected risk in user-critical segments.
• Evaluators lack access to relevant specifications.

The lesson is simple but uncomfortable: procedural correctness does not guarantee effectiveness.

Risk Management as Validation Variable

Quality scores are often treated as deterministic signals. But, in reality, they are inferences made under uncertainty. As such, their interpretation requires implementers to consider additional questions, including:

• How close is the score to the decision threshold?
• What is the level of evaluator agreement?
• How much uncertainty is introduced by sampling?
• Do different quality dimensions tell a consistent story?

Confidence intervals, inter-rater agreement, and internal consistency are not “nice to have” analytics; they are potential sources of validation evidence. They help determine whether a score is a reliable basis for decision-making or whether additional validation (such as user testing) is warranted.

Validation failures announce themselves late and expensively. Organizations discover that “compliant” translations don’t work only after launch, when typical correction costs run 10-30x higher than proactive validation. Emergency re-translations compress timelines and strain vendor relationships. Support tickets from confused users accumulate faster than quality scores predicted. Product launches delay while teams troubleshoot translations that technically passed all checks. In regulated industries, inadequate validation creates audit exposure that procedural compliance alone cannot address. These costs don’t appear in quality reports because unvalidated systems measure activity, not outcomes.

Seven Tips to Validate Without Overcomplicating Quality Management

Validation does not require continuous user testing or exhaustive experimentation. It requires targeted evidence that reduces the most relevant uncertainties. The following practices provide practical entry points for validation at both product and system level.

The following tips can guide validation for implementation in high-volume and risk-sensitive translation workflows.

1. Validate at Decision Boundaries, Not Everywhere

Validation efforts should concentrate where decisions carry risk, so validation should be prioritized when:

• Scores cluster near publish/reject thresholds,
• Quality dimensions conflict (e.g., fluency high, borderline accuracy),
• Content is user-facing, safety-critical, or brand-sensitive.

Rather than applying validation efforts uniformly, targeting boundary areas focuses efforts on high impact areas.

2. Use Targeted User Evidence, Not General Feedback

Validation evidence should be purpose-specific. Fitness for purpose is not a matter of whether users like a translation. Instead, concrete evidence can come from:

• Completing intended tasks,
• Understanding key messages without the need for clarification,
• Avoiding risk points (legal, medical, safety) based on content.

Task success is often more informative than subjective preference—and Likert scale preferences have been shown to be ill-aligned with task success metrics.

3. Exploit Disagreement as a Validation Signal

Evaluator disagreement is often treated as noise. In validation, it should be treated as data. Track inter-rater disagreement on high-impact dimensions and recurrent disputes on the same error types. Persistent disagreement may indicate that:

• Specifications are underspecified, or
• The TQE system is not aligned with real decision criteria, undermining confidence in every quality decision it informs.

4. Test the TQE System Against Known Outcomes

System validation requires ground truth, even if imperfect. Periodically compare TQE outcomes against content that previously triggered user complaints and content that performed well in real-world use.

Analyze false positives (“passed but failed in use”) and false negatives (“failed but worked fine”). Both outcomes become validation findings.

5. Treat Sampling as a Validation Risk

Sampling decisions introduce uncertainty that verification cannot remove. Unlike sampling ideally identical outputs of tangible manufacturing, translation sampling for quality evaluation is sampling across heterogeneous content with uneven user impact, where defects are not randomly distributed and consequences vary by context.

For sampled evaluations, explicitly ask:

• What failure modes could this sample miss?
• Is the sample representative of user-critical content?
• Would a different sample plausibly change the decision?

If the answer is “yes,” additional validation may be warranted.

6. Revalidate When Context Changes

Validation is not a one-time activity. Changes in context invalidate old assumptions faster than they invalidate procedures. Trigger revalidation when:

• Target audiences change
• Content type or risk profile shifts
• New MT or post-editing processes are introduced
• Quality scores drift without corresponding outcome evidence

7. Document Validation as a Rationale, Not a Score

Validation should support decision confidence rather than produce another metric. To keep validation lightweight while making uncertainty explicit, capture validation outcomes as:

• Short decision rationales.
• Assumptions tested and confirmed (or rejected).
• Residual risks accepted.

A Layered View of Translation Quality Assurance

Maintaining a clear separation between verification and validation ensures that 1) compliance is not mistaken for effectiveness; 2) confidence in quality decisions is justified; 3) risk is managed transparently; and 4) standards remain clear.

A robust translation quality framework respects the following chain:

1. Requirements are about stakeholder needs and expectations
2. Specifications are operationalized requirements
3. Verification (TQE) is the analytic evaluation of compliance
4. Scores & Decisions indicate inferred fitness
5. Validation is evidence that inference is reliable
6. Recalibration adjusts specifications and system to obtain valid and reliable scores.

Confusing the two may be operationally convenient, but it is conceptually unsound and increasingly risky in complex, high-stakes translation contexts.

Verification tells us whether we did what we said we would do. Validation tells us whether doing so actually works. Validation closes the quality evaluation loop. Without it, quality management becomes self-referential.

Building Justified Confidence in Quality Decisions

The separation between verification and validation isn’t academic—it’s the difference between measuring activity and measuring outcomes. Organizations that build validation into their quality frameworks gain not just better translations, but justified confidence in their quality decisions under uncertainty.

If your quality scores aren’t reliably predicting fitness for purpose, the problem isn’t the translators—it’s the system. And unlike verification failures, validation failures don’t announce themselves until correction becomes expensive. The question isn’t whether your translations comply with specifications. The question is whether your specifications—and the system that measures them—reliably predict what actually matters.

Validation closes the quality evaluation loop. Without it, quality management becomes self-referential, measuring its own compliance rather than stakeholder success.

Figure 1: Table comparing verification and validation at two levels. Rows distinguish the translation product and the TQE system. Columns distinguish verification and validation. Product-level verification concerns conformance to specifications such as terminology and style, while product-level validation concerns fitness for intended use based on user outcomes. TQE system verification concerns correct metric implementation and evaluator consistency, while system validation concerns whether quality scores reliably support correct decisions.