Are student evaluations reliable?

A common belief among students is that Faculty Course Questionnaires (FCQs) hold little power. Consequently, many students feel that the evaluations are a waste of time and effort.

Communication Professor Don Morley, who teaches a communication research class based on statistical methods, has done an exceptional amount of research on FCQs, including his own results over time. Once just an inexperienced, 30-year-old teacher at UCCS, Morley believed that students related to him and liked him, and at the end of his first several semesters, his ratings were high. Since then, his evaluations have suffered.

Some of Morley's research, he believes, have begun to explain his and others' changing results. One study compared attractiveness and ease of a professor's class with the ending evaluations, and found that age is an important factor in overall results.

Morley has also studied the shift in student evaluation validity. Interestingly, evaluations in the ‘70s were valued more than studies more recently conducted. As Morley explained, "All kinds of noted people will defend them as valid," however, many times, student reliability is overlooked. Much of the current research, said Morley, finds students as both unpredictable and an occasionally useless source of information.

Striving to overcome this issue, Morley takes a mathematical perspective on reliability. Found on his web page is a computer program that statistically determines the reliability of any FCQ on campus. The ICC-AK formula is able to derive a number between 0 and 1, from the course evaluations, which determines the reliability of that particular set of FCQs based on consistency of results. The closer the ICC-AK is to 1, the more agreement found on the class's evaluations and the more accurate, he said, are the results.

For example, if Art History scored a 0.87, then the majority of the class felt a similar way about the course and the FCQ is relatively reliable. The courses scoring close to 1 are considered the most important to analyze because they are the most valid, according to Morley. 

When asked about how serious FCQs are taken by the administration and the possible repercussions of negative reviews, Dean's Review Committee Member and Psychology Professor Hasker Davis refused to be interviewed. Morley said that FCQs are of top priority to the committee in making faculty-related decisions. Still, Morley could not confirm that FCQs have ever led directly to a tenured professor's dismissal. "It is not to my knowledge that a tenured professor has ever been let go," Morley answered.

Nonetheless, poor results on FCQs, according to Morley, can influence professors' annual compensation increases, and if evaluations do not improve, a professor improvement plan can be implemented.