Spanish translations of Cochrane Plain language summaries: assessing the impact of a controlled language checker on machine translation quality

Session: 

Oral session: Linguistic translation of Cochrane content

Date: 

Sunday 16 September 2018 - 14:20 to 14:40

Location: 

Carrick 3

All authors in correct order:

Rossetti A1, Ried J2, O'Brien S1
1 School of Applied Language and Intercultural Studies, Dublin City University, Ireland
2 Cochrane Translation, Germany
Presenting author and contact person

Presenting author:

Alessandra Rossetti

Contact person:

Alessandra Rossetti
Juliane Ried
Abstract text
Background: Cochrane aims to make high-quality health information accessible for users worldwide (Chandler 2017). To this end, contributors simplify Systematic Reviews into Plain language summaries (PLS), which are translated (Cochrane Community 2016). Simplification and translation may represent onerous tasks when conducted without assistance from technology. Contributors produce PLS by manually implementing a set of guidelines (Cochrane Methods 2013), and translations are mainly produced by volunteers (Cochrane Community 2016). Cochrane is considering using machine translation (MT) and controlled language (CL) checkers (Birch 2017; Von Elm 2013). A CL is a set of rules used to make a text more comprehensible and translatable. A CL checker is the software that checks for adherence to those rules (O’Brien 2010). The adoption of a CL at the authoring stage can increase machine translatability and reduce the effort required to correct the MT outputs (Allen 2003).

Objectives: This study investigated whether introducing a CL checker called Acrolinx in the authoring process of PLS increases the quality of the Spanish translations of PLS produced by Google Translate, a freely available MT system.

Method and materials: The experimental materials were: a body of 12 PLS authored by implementing Cochrane guidelines and their Spanish MT translations manually; and a body of 12 PLS authored with assistance from Acrolinx and their Spanish MT translations. The MT evaluation task was conducted by 41 Spanish native speakers. Each evaluator was given two PLS (one per body) and their Spanish MT outputs. Evaluators assigned scores to the Spanish outputs on fluency and adequacy, commonly employed in MT quality evaluation.

Results and conclusions: The difference in quality scores between the two bodies of PLS was slight, possibly because neither approach to simplification was developed with MT quality in mind. Fluency and adequacy scores were relatively high, suggesting that MT produced reasonably good outputs. While human validation of the MT output is needed for accuracy, the high adequacy and fluency scores for Spanish indicate that integrating MT in the translation workflow might be a viable option.

Healthcare consumer involvement: This study investigated how technology might increase the accessibility of health content for healthcare consumers worldwide.

Relevance to patients and consumers: 

Cochrane’s audience is linguistically diverse and health content that is provided in English might prove inaccessible for non-native speakers of English, especially lay health consumers (Von Elm et al. 2013). Translation of Plain Language Summaries has already shown to increase access rates to Cochrane content substantially (Birch et al. 2017). By assessing the quality of translations produced by machine translation systems, this study might lead Cochrane contributors to rely more heavily on this technology which could speed up the translation process. In turn, the number of translations made available to health consumers might increase.