Pedagogy of priorities: a critical discourse analysis of instructor evaluations

Published online 24 January 2007.

In this diary, I will analyze the questions given on student evaluations of me (and other instructors where I was working) in order to say meaningful things about the discourses we use when we judge “good” and “bad” educational experiences.  It will be made clear that, in analyzing student evaluations (or any politicized educational discourse) the question of “what education is for” will be opened anew.

Deciding what education is for

When decisions are made in educational politics, there are always implicit judgments about “what education is for.”  In pedestrian discussion, the answer to the question of “what education is for” is a given: education is for teaching and learning, and you learn things so that you can think, or acquire skills.  What students should be thinking, and which skills students should acquire, then remain as open questions.

Arguably, these questions should be left open; instructors and students vary significantly as do all people, and their diversity can be counted as an asset to be used.  Instructors who have had special experiences (Holocaust survivors, for instance) can be counted on to give students learning experiences that they can’t get elsewhere.  Students, for instance, have different life-histories, which can to a certain extent be grouped in terms of their relation to a common history.  African-American students, for instance, belong to a historically-marginalized group, and this can be viewed as to a varied extent affecting their educational experiences (see, for instance, the educational debate about “ebonics”).  When teachers address “ebonics” as a reality of students who speak “African-American vernacular English” they, to a certain extent, diversify educational experience in schools.

When Congress passed the No Child Left Behind Act in 2001, then, it also made decisions about “what education is for.”  Education, it decreed, is for testing, and the tests are to determine whether students/ teachers/ schools/ districts are “doing well” or “doing poorly,” according to whether students have mastered particular bodies of knowledge.  Further standardizing the process of education (within NCLB) is its decree that “Schools are required to use “scientifically based research” strategies in the classroom and for professional development of staff.”  (source).  This decree apparently leaves out qualitative/ ethnographic research as a guide to professional development in favor of quantitative research.

The idea behind this sort of legislation appears to be that the same standards are to be applied to students and teachers regardless of their different circumstances, different personalities, different learning styles, and different life-goals.   NCLB, then, risks categorizing as “inferior” those who do not fit its mold.  Student and teacher bodies are in real life quite diverse; yet NCLB expects all to strive for “adequate yearly progress” regardless of how achievable or unachievable, meaningful or meaningless, this “adequate yearly progress” may be for different populations working with different resources.

Generally speaking, there are two main objections to NCLB circulating in political discourse in the United States: one is that NCLB is a good thing, but that it is being inadequately funded, and another is that NCLB, even if adequately funded, would not be a good thing.  The latter objection implies, according to this analysis, a disagreement over what NCLB says school should be for.

The point is that, when we try to “do educational politics,” we are reopening an ongoing debate about “what school is for,” a debate that cannot be avoided and should in fact be embraced.  The consequences of America’s aversion to this debate have been outlined most succinctly in Horse Philosopher’s diary of last month.  Pay heed!  Education is not for “whatever people think it’s for.”  That way lies legislative meddling by fools who think they can “improve the schools” without regard to the real-life conditions which affect said schools.

The pedagogy of priorities: how student evaluations guide “what college is for”

So, in sum, legislative decisions are also decisions about “what school is for.”  However, there are also plenty of other ways in which we decide “what school is for.”  One of these, which I will discuss in detail here, is the routine student evaluation of instructors in a collegiate setting.  This evaluation, of course, is typically done anonymously, and without the instructor’s knowledge of who is saying what about her or him.  At any rate, these evaluations can be said to have a certain power over instructors, to determine which instructors will be “popular” with students and which ones won’t.  They are often used as indices of an instructor’s employability, thus separating out part-time instructors (and remember, we’re talking about a significant percentage of the total here!) who will keep their jobs from those who won’t.

With so much at stake, we ought, I argue, to look at how the evaluation form is set up, so we can observe the criteria by which students are being asked to judge the employability of professors.  To examine these forms, I further argue, we need a theoretical tool, and this tool is what I will call “critical discourse analysis.”  For this analysis, I will use a form that was used to evaluate me in my last part-time college-professor assignment.

Critical discourse analysis

I’m not going to make you read a lot of the textual citations for this stuff, but it needs to be said (briefly) that critical discourse analysis is a product of the academy, of scholars like Norman Fairclough and James Paul Gee and Teun Van Dijk.  At any rate, critical discourse analysis is about discursive texts which are produced by writers and consumed by audiences.  In these texts, discourse “creates” a reality of its own.  This creation of reality occurs in three ways:

  1. Identity — the discourse establishes an identity for both author and audience
  1. Audience relationship — the discourse establishes a relationship between sender and receiver
  1. World-view — the discourse establishes a certain way of looking at the world, an ideology if you will

So, keeping that in mind, let’s take a look at the evaluation questions themselves:

The form

This form is called the “student opinion survey,” and it aims to assess the professor quantitatively by having the students fill in bubbles which will then be read by a computer.  There is, of course a “use other side for written comments” space, but since nobody besides the instructor reads those, they hardly bear upon the evaluative process.  (Moreover, students may fear that instructors will be able to use such commments to decipher the handwriting and figure out who is evaluating them negatively.)  Each question is to be answered with one of seven answers:

  1. Strongly agree
  1. Agree
  1. Somewhat agree
  1. Somewhat disagree
  1. Disagree
  1. Strongly disagree
  1. Don’t know/ Not applicable

So here are the questions themselves:

  1.  The INSTRUCTOR clearly defined the course requirements
  1.  The SYLLABUS clearly outlined the course requirements and grading criteria
  1.  The INSTRUCTOR clearly presented the subject matter
  1.  The READING MATERIAL, including the textbook, served well the purpose of this course
  1.  The examination QUESTIONS were a good measure of the material presented in the course
  1.  The instructor ADMINSTERED and SUPERVISED the examinations appropriately
  1.  In general INFORMATION about how well I was doing was readily available
  1.  In general, the instructor was ACCESSIBLE to provide requested help in the subject
  1.  The INSTRUCTOR INTERACTED with students in ways that were free of racia prejudice or discrimination
  1.  I would RECOMMEND this INSTRUCTOR to others
  1.  How would you rate this instructor’s OVERALL TEACHING ABILITY?

And there is a space below for “questions added by school or department are shown below.”

Remember, these are the criteria by which evaluation forms are asking students to ponder what counts as good teaching.  At any rate, let’s look at the questions, one after the other, using critical discourse analysis:

  1.  Now, it is easy to agree with common definitions of teaching, that one of its responsibilities is to “clearly define course requirements.”  One of the things that this suggests, however, is that the instructor has been clearly defined beforehand as a grader of students, and that the point of the educational process is to measure how well students have met “clearly defined course requirements.”  Question number one, therefore, is a corollary of the ideology of grading.  (There are, of course, other ways of measuring student class performance than grades, one of which, the Narrative Evaluation System, was applied to me as an undergraduate at UC Santa Cruz (pre-1998).)
  1.  This question seeks to measure the efficacy of the syllabus, as a contract which will bind the professor to the issuance of particular grades upon the student performance of grading criteria and course requirements.  Here we have an implied contractarian view of education.  Instructors are not your friends; they are like employers, who will pay you in grades for your student labors.
  1.  The assumption given in this question is that the subject matter should be “clear.”  Its given assumption about student identity, however, is that the student is already well-prepared to understand the subject matter, which may or may not be the case.  There is a relationship implied here in this question, that the instructor presents subject matter whilst the student absorbs the instructor’s presentation.  The question seemingly absolves students of responsibility to initiate dialogue (many don’t) about material they don’t understand.
  1.  This question assumes quite a bit about student identity.  Are students to imagine that they have such an expertise in the instructor’s field that they can say with certainty that the reading material chosen for a class is or isn’t appropriate for the “purpose of the course”?  It also assumes that students are capable of deciding what the “purpose of a course” should be.  Now, mind you, I have no problem with the idea that students should choose, or at least influence, a course’s reading material or its purpose.  I just want this choice to be made out in the open, rather than being insinuated in an underhanded way through a student evaluation.  (However, it needs to be added at this point that if students were to be allowed to choose the curriculum in any particular class, that would risk the professor’s scoring a poor evaluation on question #2.  If the students are really to choose, that means these choices cannot ultimately be specified beforehand in the syllabus in the way specified by question #4.)
  1. and 6. assume that the course contains an exam (mine didn’t), and that the relation between teacher and student is to be one of examiner.  Professors who choose not to give exams cannot benefit from good marks in either of these questions, fully one-fifth of the evaluation itself.
  1.  This question seems to imply that students are to make “progress” within a course steadily throughout the term.  A particular course, however, might make some large percentage of the total grade contingent upon work to be turned in at the end of the term.  Thus it would be difficult to tell students “how well (they were) doing” based on so little information.  Once again, this question fits snugly within the ideology of grading.  Professors grade, students earn grades, their relationship is contract-bound, and the overall ideology promotes “doing well” in those terms.
  1.  This question favors professors who are “accessible” — they have office hours, will meet up for appointments, and will answer emails promptly.  It also presumes that students are “request(ing) help,” which may or may not be the case.
  1.  This question implies that “racial prejudice” is something that can be determined from the interaction between professors and students.  Sometimes it can.  Racial prejudice, however, is a structural problem, and quite often it can be traced to the origins of students in segregated schools wherein lower-class, historically-marginalized racial groups (“Black,” “Latino”) are segregated into classrooms and schools which grant them an education which prepares them for a subordinate class existence.  (For more information, see Jonathan Kozol’s The Shame of the Nation).  Now, if a particular professor gives poor grades to students in historically-marginalized groups but it can also be proven that those students deserved those grades according to the criteria set forth in the syllabus, is the professor to be granted a poor scoring on question #9?  I can see why students would want to be asked a question #9 — racist professors are a bad thing.  However, I would like to suggest, also, that there remains the risk that professors who are overtly anti-racist could risk a poor showing on question #9 from students who interpret anti-racism as a form of racism.
  1. and 11.  These are general evaluative questions.  Question #10 presumes that if a student likes a professor, she will recommend that professor to others.  This risks the possibility that a particular professor may be good for some students and too difficult for others.  If I felt that, as a student, Professor X were good for me but not for those other students who in my judgment weren’t bright enough, would I feel safe in recommending Professor X to others?

These questions also do what student evaluations generally do — they conflate good teaching with popular teaching, and make it necessary for professors to appeal to a student body that 1) may not be homogeneous enough to agree on a particular professor’s worthiness and 2) may be taking that professor’s course for reasons which are irrelevant to that professor’s worthy qualities.  A particular professor may, for instance, be a brilliant scholar and effective conveyer of that erudition, yet be required to teach large numbers of students who are “there for an easy grade” and who will come away disappointed when the easy grade is not forthcoming.  Are we to judge that professor as a “poor professor” if her evaluations are “below the norm” calculated statistically?  (This outcome is more likely, I might add, if the statistical evaluation weighs negative evaluations more highly than positive ones.)  One can easily see, then, how questions such as the ones I have asked above open up the can-of-worms of what education is for.

Part of the problem, it might be added, is in the seemingly “objective” flavor in which statistical evaluations are typically couched.  When we attempt to compress a professor’s performance into a set of numbers, we lose the presupposition that evaluations are context-bound estimates of a professor’s compatibility with a particular group of students with particular needs.  A more accurate evaluation, relying upon such a presupposition, might rely entirely upon narrative estimates of a professor’s compatibility with a student body, given from multiple sources.

Conclusion: the pedagogy of priorities of the student evaluation

In conclusion, it must be said that the point of critical discourse analysis is not criticism, but rather to outline, with some degree of objectivity, what reality a particular discourse makes.  In this case I am analyzing student evaluations to show how they make that reality, for both student evaluators and faculty interpreters.  In many cases, however, a classroom full of student evaluators may deem the evaluation questions to be inappropriate, and will then use the evaluation process as a referendum on whether or not the professor deserves a good evaluation.  In such cases, we can say that the people designing the evaluation questions have “lost control” of the evaluation process.  They have failed, not partially but completely, to decide what education is for.

As it stands, however, the evaluation questions of this particular university (such as I have analyzed them above) model a particular type of education, contractarian, grades-oriented, clear and specific in its “requirements,” easy for students to understand, and racially “neutral.”  Is this particular type of education good for all students (regardless of diversity), all educational subjects, and all professors?  The imposition of uniform standards upon diverse populations is a judgment call, and like other varieties of evaluation, it opens the question of “what education is for” anew.

Actually, any sort of judgment about what is a “good teacher” opens that question.  But in everyday discourse we throw about judgments on teachers without asking ourself what education is for.  Are we so sure?

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s