. . . .

"The survey did not ask Do you like your teacher? Is your teacher nice? This wasn’t a popularity contest. The survey mostly asked questions about what students saw, day in and day out.

"Of the 36 items included in the Gates Foundation study, the five that most correlated with student learning were very straightforward:

1. Students in this class treat the teacher with respect.

2. My classmates behave the way my teacher wants them to.

3. Our class stays busy and doesn’t waste time.

4. In this class, we learn a lot almost every day.

5. In this class, we learn to correct our mistakes."

Here is earlier reporting (10 December 2010) from the New York Times about the same issue:

http://www.nytimes.com/2010/12/11/education/11education.html Here is the website of Ronald Ferguson's research project at Harvard:

http://tripodproject.wpengine.com/about/our-team/

And here are some links about the project from the National Center for Teacher Effectiveness:

http://www.gse.harvard.edu/ncte/news/NCTE_Conference_Using_S...

LAST EDIT: I'm amazed at how many of the comments in this thread appear to be about issues thoroughly discussed in the submitted article, but unresponsive to what the submitted article said. On this kind of issue, it's an especially good practice to read the fine article before assuming what is being discussed. We all know about school, but specific proposals for school reform have specific details that make some worse than others, and can be empirically tested.

I'm not, unfortunately. It seems like most people read a headline and perhaps a paragraph or two, then active their pre-existing beliefs about whatever the subject happens to me, and move on from there. That's certainly been my experience with commenters on my blog, anyway, and it's been experience in observing both online communities and in reading student papers.

I was a dick back in high school. The hacker I was back then would have figured out exactly how the testing and metrics were set up (public information) and organized a union of students to manipulate it. I can't do much with standarized test scores, they reflect on me. But a teacher quality survey? That's just a weapon.

Things like this make me wish that we had some kind of Hacker in Chief, to figure out how to circumvent new systems before they get implemented.

This is discussed extensively, midway through the article, in approximately four paragraphs:

Students were better than trained adult observers at evaluating teachers. This wasn’t because they were smarter but because they had months to form an opinion, as opposed to 30 minutes. And there were dozens of them, as opposed to a single principal. Even if one kid had a grudge against a teacher or just blew off the survey, his response alone couldn’t sway the average.

“There are some students, knuckleheads who will just mess the survey up and not take it seriously,” Ferguson says, “but they are very rare.” Students who don’t read the questions might give the same response to every item. But when Ferguson recently examined 199,000 surveys, he found that less than one-half of 1 percent of students did so in the first 10 questions. Kids, he believes, find the questions interesting, so they tend to pay attention. And the “right” answer is not always apparent, so even kids who want to skew the results would not necessarily know how to do it.

Even young children can evaluate their teachers with relative accuracy, to Kane’s surprise. In fact, the only thing that the researchers found to better predict a teacher’s test-score gains was … past test-score gains. But in addition to being loathed by teachers, those data are fickle. A teacher could be ranked as highly effective one year according to students’ test gains and as ineffective the next, partly because of changes in class makeup that have little to do with her own performance—say, getting assigned the school’s two biggest hooligans or meanest mean girls.

Student surveys, on the other hand, are far less volatile. Kids’ answers for a given teacher remained similar, Ferguson found, from class to class and from fall to spring. And more important, the questions led to revelations that test scores did not: Above and beyond academic skills, what was it really like to spend a year in this classroom? Did you work harder in this classroom than you did anywhere else? The answers to these questions matter to a student for years to come, long after she forgets the quadratic equation.

Fortunately the authors weren't so careless — that concern is obviously key to the repeated cautions about assigning too much weight to such tests. I'd like to believe anyone setting policy would be similarly slow to jump to conclusions but this is a field notoriously prone to chasing fads and quick fixes…

For example, in one programming summer camp I led, I've had kids come to me at the end of the 10 days and tell me that I was their favorite educator and a great director because I was tough but fair and expected them to learn and progress, while some other educator was lame because even though he was super nice and laid back, they didn't learn anything with him.

Now of course, during the camp itself, I've overheard more than one kid telling another that I was super mean and terrible. But in the end, most of them were vocal about how they appreciated me. Kids in general tend to be very aware of and appreciate fairness, consistency, and holding them up to standards and pushing them to get better.

Cases like that are not the exception, but sadly not the majority either (the specific example I cited was with French "gifted" kids between 14-18 years old). So it would really depend.

As a recent high schooler, I've seen kids who absolutely hated their teachers because they gave them bad grades because they challenge their students. However, this specific teacher was probably the most influential for those who appreciated him.

I'm just afraid that the majority will figure out the system and use it as a weapon, as was mentioned.

Only if it's used as part of a (simplistic) algorithm.

And besides, why would students want to fire the good teachers? That doesn't really make sense. (As someone who was sent to the principal's office on a semi-regular basis.)

Later, whenever I met a new teacher (at university, in the workplace) lessons always seemed easy to me. I had to work hard for a good grade at high school, and her tests demanded precision and attention to detail. I started to appreciate her more.

Students are likely to prefer teachers with a pleasant attitude and lower difficulty level. A teacher easy to distract with an off-topic conversation. But at the end of the day, a teacher's job is to teach. Sure, a great teacher would combine that with charisma, ability to inspire and natural charm, but nobody's perfect.

Win!

Part of the reason you see so much backlash from teacher's unions against these initiatives (despite the fact that virtually all teachers acknowledge that some are better than others, and have a pretty fair idea who the worst ones are) is that they don't see this as a way to fairly evaluate teachers and improve the quality of education. Rather, they see it as a weapon to be wielded by school districts, school administrators, and angry parents ("Mrs. X told little Johnny (age: 17) where babies come from!" "Mr. Y is a member of $MARGINALIZED_SUBGROUP!" "Ms. Z believes something that is the current scientific consensus" "Mr. Q gave my son an A-") against unpopular teachers. It is an unfortunate fact of life, and even more unfortunate that so few are willing to admit it (especially here on HN, where the quality of discussion is usually above average).

We would probably have much greater buy-in from teaching unions if these evaluations were perceived as something other than politics as usual. Unfortunately, the proposals all seem to keep a some level of human involvement in the firing process (so only the very most outstanding teachers are immune to the usual political pressures). It's useful at this point to recall a bit of history: Teachers today are so hard to fire because they have tenure. Teaching unions successfully forced tenure the tenure system on school districts because school administrators of the past abused their authority to fire good teachers who happened to be politically unpopular.

I'm also not so sure that everyone would be so willing to honestly critique their teachers at the end of the year, either. A lot of kids come to like their "bad teachers who lead fun classes" almost as much or more than the teachers who do their jobs well. Kind of like how many kids are unwilling to "snitch" on kids who have wronged them, even when guaranteed anonymity, I think a lot of them wouldn't want to get their teachers in trouble, even if they weren't doing their job properly. In my experience with such teachers, even poorly behaved students would act extremely respectfully when the principal came around to observe. Then again, maybe they were just afraid of getting on the teacher's bad side.

Even worse are instances where the administration knows fully well that a teacher leads a "joke class" and keeps them around to funnel the "bad kids" into. I grew up in a school system that straddled the border between a middle class college town and Section 8 housing packed with families ruined by the collapse of unskilled industrial labor in the area. I have no doubt that a large number of the elective courses existed solely to fill credits for disruptive students and keep them from causing problems in the "real classes" that the middle class and brighter/better-adjusted lower class students took. It didn't matter if the class actually learned anything, the school was just desperate to raise their graduation rates.

I do think these evaluations can help good teachers improve and possibly help to identify people who don't deserve to work as teachers, but they can't solve fundamental problems like "the early education and home environment of these students has failed them, and now the administration of their high school is ready to roll over to keep them from dropping out."

Having students grade the teachers I think is also the same way. One of the problems is that students don't know what makes a good teacher. The tests can gear students in that direction (ie "I feel challenged but not overwhelmed in this classroom"), but if we look for too much insight from the students I think we will be misdirected. Just like we are misdirected when we pay too much attention to standardized tests.

My fear is that some schools would start to look at these performance measurements as golden bullets sort the way that we've started to look at standardized tests as golden bullets. Most of jr. high was geared towards getting perfect scores on the state exams. My Junior and Senior years of high school were almost 100% (a few teachers went outside of the scope, but it was a teacher decision and not an administrative decision) geared around AP tests and the ACT.

The ACT and the AP tests have their place. And I think that student evaluations of teachers have their place as well. Both are very useful when applied appropriately.

I just don't want to see the system (d)evolve in such a way that too much emphasis is placed on empirical data.

Students don't need to know what makes a good teacher. They only need to be able to assess whether they learned, and whether they had fun in the process. Whether they learned enough is pretty much an orthogonal issue, and one that can and will be dealt with through standardized testing. Students also don't really need to give any thought to a teacher's specific methods in order to offer useful information.

This kind of thing (teachers grade students, students grade teachers) could improve the efficiency of the feedback loop and thus efficiency of the education system as a whole.

- Despite the fact that grades don't predict workplace performance at all, companies still use them to hire people. (Basically because it's a legal way to keep out black people.) Because of this, schools can boost their US News & World Report rank by giving people better grades, since starting salary factors into the rankings in many years.

- Schools are basically run as businesses these days, and that's what the customers want. Asking why schools keep inflating grades is kind of like asking why McDonald's keeps making their chicken nuggets taste so fucking delicious.

-Schools in Africa grade students, who are they trying to keep out?

-Some schools might be run as a business; I think most aren't (they're subpar and inefficient). If they are, they're run as a mom and pop. If San Francisco schools were businesses, they'd have gone bankrupt and displaced a while ago.

The science suggests that if you we're assigned both groups at random, you wouldn't be able to tell the difference.

There are only a handful of studies because it's considered kind of a dumb question to ask. But Alfie Kohn references some of them in his books, I think in What Does It Mean To Be Well Educated? But the ones that exist consistently show the same thing.

This includes the huge internal study that Google HR did:

"Unfortunately, most of the academic research suggests that the factors Google has put the most weight on — grades and interviews — are not an especially reliable way of hiring good people."

"When all this was completed, Dr. Carlisle set about analyzing the two million data points the survey collected. Among the first results was confirmation that Google’s obsession with academic performance was not always correlated with success at the company."

http://www.nytimes.com/2007/01/03/technology/03google.html?e...

That's what the results found. Google just didn't change their hiring policy accordingly.

Could you apply the principle of charity and think of any other possible reason why employers might care about grades?

Like, I dunno, employers believe grades are correlated with conscientiousness, agreeableness and intelligence, and employers find these traits useful?

If you knew twice as much about the issue, do you think you'd be more or less charitable than you are today?

https://en.wikipedia.org/wiki/Principle_of_charity

It's a far better way to learn from other people than simply assuming they are racist/communist/terrorist and dismissing them.

Now if law schools have a rational argument for why they still use grades to determine who gets into law school even though it has been shown that GPA has zero correlation with ability as a lawyer, and that its main effect is to keep black people out of law school, then I'd love to hear it. But note that they would need a rational reason for doing this, not just any reason.

(Which Malcolm Gladwell talks about here: http://www.newyorker.com/online/video/conference/2008/gladwe...)

A charitable interpretation of the actions of law school admissions departments would simply be that they don't believe this "fact" (a citation for which is desperately needed).

Further, you are attributing anti-black racism to a group of people (law school admins/profs) who routinely engage in pro-black racism (see, e.g., Grutter v. Bollinger). It doesn't even make sense.

"Finally, the "draft fear" explanation cannot explain why a very similar pattern of grade inflation has occurred over the decades in Canadian universities which were spared the tumult of the Vietnam protest era and student fears that they might be drafted and sent overseas to fight and die."

It has been written about, probably in the Atlantic or NYT, maybe 10 years ago.

For example the tough math teacher would get a bad evaluation, but the cream-puff teacher who never assigned homework and gave everyone A's would get a good evaluation.

I always hear this objection, and it's usually brought up to discourage even experimental data gathering. I don't buy it at all, and I think it says more about the objectors' mentality than about the students'.

Student/teacher situations are somewhat unique in that students don't choose to be there, and will usually be gone in 1 year, and definitely in a few years. Students have far less to lose in this situation. That may make some of the be more harsh, but it likely (based on the article) will just make more of them honest.

I can't be honest with my boss because he might make life a living hell or fire me. I can certainly be honest with a teacher in a school who I'll never see again in my life.

a) What's the purpose of having both?

b) If the evals are preferred over the tests, will "good" teachers continue to teach to a predictable, standardized curriculum?

c) Is the correlation additional evidence in favor of "differential compensation", that is, a compensation program based at least in part on exam scores?

d) Even if the information supplied is similar, doesn't this extra test/survey administration detract from instructional time? Is the information gleaned sufficient to compensate for the loss of instructional time?

e) Atlanta (Georgia, USA) is still reeling from a years-long cheating scandal. If such evaluations become "high stakes" (and there will likely be a push to do so, despite likely union opposition), won't these results be exploitable as well? (And perhaps even more so, through campaigning, social engineering, etc?)

What's the purpose of looking at multiple polls when you're trying to predict the outcome of the upcoming election? More evidence gives you higher confidence and lower margin of error. And as the article says, these student surveys provide clean, stable data that doesn't fluctuate very much from year to year and doesn't require much correction for race and family income.

These surveys take on the order of 10-15 minutes. That's nothing compared to a battery of standardized tests. They wouldn't have to be very informative at all in order to be worth the small sacrifice of instructional time, and if they're the second-best predictor of class achievement, then they're certainly worth the time (if the results are actually used).

I don't think there's going to be much movement to stop paying attention to standardized tests and curriculums, since these surveys don't measure the same thing - roughly speaking, the standardized tests seek to measure how much was learned, and these surveys add a dimension of why.

I agree the movement toward testing is likely to continue unabated. The point was the futility of additional questions; when a strong correlation is already known, the responses can be predicted with a high degree of accuracy, so the additional information is not really "additional". Yes, social scientist prefer multi-item measures and avoiding a single source bias (particularly when publishing theory), but in the end, if the correlation is maintained, not much new is really learned.

(And one of the reasons for multiple election polls is that the results change over time, and leading up to an election, that's relevant. Additionally, the entire electorate isn't polled each time, so the "cost" to the system is lower, relatively speaking.)

So, if both of those factors affect student performance on standardized tests, then there will be a strong correlation between the overall survey scores and the test scores. But analyzing the details of the survey results can offer actionable guidance that the test scores can't - the survey does provide useful information for how mediocre teachers can improve, even when it matches the test scores in predictive power for future test scores.

The message one takes away from that is that (i) yes, student evaluations are a good predictor of some objective properties of a class (and other measures don't even achieve that much), but those properties aren't what teachers should be optimizing. I'd grant that (ii) it does seem worthwhile for students to see that their interests make a difference to what happens in the classroom. I'd also grant that (iii) some classroom situations may be so bad that optimizing student satisfaction may, even if not educationally ideal, still be a big improvement. And for all I know, this may be widely true at the pre-university level; but on the other hand, for all I know, giving these evaluations a big institutional role at the pre-university level could also be counter-productive...the evidence cited in the article hardly enables us to say. Any deliberation about giving student evaluations an institutionalized role should take the evidence behind (i) seriously.

One promising message from the research reported in the Atlantic article is that (iv) the specific tests being discussed have been designed in ways that seem novel and especially revealing. But the article mixed that together with an indiscriminate enthusiasm for student evaluations quite generally. And I think many people will read this and say, "Duh, that's a no brainer." Yes, it is a brainer! These kinds of policy issues aren't settleable from the armchair. Even if we cleared all the political hurdles and made someone the educational policy dictator, he or she isn't going to be able to tell just from the armchair what the results of rolling out one policy rather than another is going to be. So I get frustrated with articles like this one, that report some interesting evidence but mix it together with the kind of insensitivity to the details exhibited in comments like "That research had shown something remarkable: if you asked kids the right questions, they could identify, with uncanny accuracy, their most—and least—effective teachers. The point was so obvious, it was almost embarrassing."

Neither does this inspire confidence: "Some studies...have shown that professors inflate grades to get good reviews. So far, grades don’t seem to significantly influence responses to Ferguson’s survey: students who receive A’s rate teachers only about 10 percent higher than D students do, on average." I hope that readers of this site don't need an explanation of why the clause after the colon is only barely relevant to whether grades get inflated because that leads to better evaluations. It's almost irrelevant. In the first place, teachers needn't be aware of the cited fact; they may experience grade-inflation pressures differently. Also, the cited fact is compatible with the majority of current A-getters scoring their teachers in ways that are largely insensitive to the grades they get, but a minority of current A-getters and a majority of current B-getters being extremely responsive to the grades they get. The cited fact is just not what we need to know.