Pragmatic assessment and consistency in rating are among the subject matters which are still in need of more profound investigations. The importance of the issue is highlighted when remembering that inconsistency in ratings would surely damage the test fairness issue in assessment and lead to much diversity in ratings. Our principal concern in this study was observing the criteria that American and Iranian EFL/ESL teachers consider when rating Iranian EFL learners’ pragmatic productions regarding the speech act of compliment. The instrument utilized in this study was WDCTs and a speech act rating questionnaire administered to sixty American and sixty Iranian EFL/ESL teachers. In order to come up with the criteria, the reasoning and justifications of the raters when rating learners’ pragmatic productions were analyzed carefully through content analysis. The results showed that overall the raters considered nine general criteria when rating. They included: “Strategy use”, “Affective factors”, “Politeness”, “Interlocutors’ relationships”, “Linguistic accuracy”, “Sincerity”, “Authenticity”, “Fluency”, and “Cultural issues”. Also, the most frequent criterion among the native and non-native raters was “Strategy use” and “Politeness” respectively. Finally, it was concluded that due to some inconsistencies and variations in the ratings and criteria of both native and non-native raters, it seems that both groups are in need of pragmatic workshops and training sessions. The results of this study can have important implications for EFL/ESL teacher educators who are considerate of the importance of pragmatic training and instruction.
