In the present paper, we present a study where we tested the newly proposed methodology for evaluating machine-translated texts (English-Slovene language pair). We first outline our ad hoc error categorization scheme, which served as the basis for the preparation of three types of translations (raw MT output, partially corrected, and completely corrected). Participants were instructed to read the texts for comprehension while their eye movements were recorded. After reading, they were asked to give their subjective evaluations as well. While the three types of translations were given significantly different scores by the participants, eye-tracking measures show difference between completely corrected and non-corrected MT output, but no difference between partially and non-corrected MT output.