Ed.D. Students Yoonah Seong & Elizabeth Bottcher to Present at LTRC 2011

| February 24, 2011

Presentation Title: Assessing academic presentation performance: Does the rater matter?

Several studies have found differences in teacher and non-teacher raters’ attention and scoring behavior (e.g., Brown, 1995; Elder, 1993; Lumley, 1998). The findings all indicate that raters do not always interpret student performance and scoring criteria a similar way. This study which examines academic presentation performances yields similar results.

The oral presentation was a final project requirement for two English for Academic Purposes (EAP) courses at two different American universities. One class was an elective course for intermediate ESL learners in a non-degree intensive ESL program. The other class was for advanced level international students already matriculated in the university. Thus, the second class was a mandatory course which counted towards the students’ English language requirements. The presentations were video-taped and scored by two raters: 1) the teacher who taught both classes and 2) an ESL instructor with no EAP teaching experience. Using FACETS based on the Rasch Model, the examinees’ performance on this test task, the relative difficulty of the rubric components (Content, Organization, Visual, Delivery, Language) upon which the students were measured, and most importantly, the raters’ influence on the scoring process were examined.
This study shows that raters’ bias towards certain examinees and contradicting patterns of rater interaction with classes existed. Post-analysis discussion between the raters revealed that the raters’ awareness of the class context (e.g., class configuration, test purpose, and students) could have affected the raters’ scoring behavior and pattern. Although there is a need for further investigation, the results of this study raise important questions regarding the validity of teacher-based assessment in a classroom context.