Improving MT Evaluation at the Sentence Level: A Learning Approach

Abstract

Machine Translation (MT) systems are more complex to test than they appear to be at first, since many interpretations are also seen as equally reliable. The task becomes much more challenging in practice as short texts such as incremental system development and error analysis require tests to be carried out automatically. While a variety of automated metrics, including BLEU, have been suggested and implemented for discrimination against the large-scale MT system, there is still not enough sentence-level connection to judgments. A new metrics class is proposed in this paper based on machine learning. The requirement for broad client concentrations as a source of information preparation may be expelled by another methodology which classifies interpretations as CPs or human, as opposed to the immediate estimation of numerical human decisions. The resulting metric based on aid vector machines shows that the current artificial metrics are increasing dramatically and that the association with human decisions is only halfway up to that achieved by the objective human evaluator at sentencing level.