New Practice for Holistic Human and AI-based Quality Evaluation System for Translation
1. Scope
This practice, the Holistic Quality Evaluation System for Translation, henceforth referred to as HQuest, documents the core principles necessary to objectively measure translation quality using a holistic approach, including the AI-based holistic evaluation. Reliable and accurate measurement of translation quality is extremely difficult due to the fundamental fact that the impression a reader develops from a text is fundamentally subjective. This impression is shaped by the material perceived as a whole (on the macro level). Even meaning transfer is conditional on the entirety of experience, not to mention context, style, and other factors. This document identifies factors affecting the perceived quality of translated material and provides an overarching methodology for measuring those factors. It demonstrates the value of holistic assessment methods to complement, validate, or in some cases even replace detailed (analytic) approaches. It addresses the need for stakeholders in the translation industry to be able to reliably assess the quality of a translation with reasonable effort using an approach that is suitable for the task at hand. HQuest defines a methodology for assessing translation quality based on user experience, which includes perception of the material as a whole, sensitivity to particular details, fidelity of meaning transfer, context, literary impression, readability, text type, purpose, intended audience, language properties, and other factors. This standard practice does not define specific metrics for measuring language quality; rather, it establishes the fundamental properties of translation quality, demonstrates how these properties can be evaluated, and describes the necessary features of measuring tools and sets workable limits or boundaries for their use. These various parameters form a framework for creating a set of metrics. A key concept of well-formed metrics is introduced. An important method of creating well-formed analytic metrics is described. HQuest also differentiates between cases where a detailed analytic assessment is necessary, and cases where the more rapid holistic assessment is sufficient and recommended. Holistic assessments can also be used to establish the need for more detailed issue-based assessments. HQuest does not include an error typology, nor does it define procedures for carrying out analytic assessments. An error typology and procedures for performing analytic assessments are described in WK46396 Standard Practice for Analytic Evaluation of Translation Quality. While it does not specifically address analytic assessments, HQuest is nevertheless based on the principle that if an analytic metric is well-formed, it will produce an assessment that corresponds to holistic Adequacy and holistic Readability assessments. As such, correct analytic metrics can be validated by holistic assessment in accordance with this standard practice, and after that the two methods can be used interchangeably on the same type of material, which opens a way to a clear cut method of building a well-formed analytical metric for every given type of material, audience and purpose. Two approaches for holistic assessment are considered, one using expert reviewers, and another using volunteer reviewers.
government agencies and industry practitioners in the field of language industry, willing to use quick and most general way of evaluating translation quality
