Thursday, August 16, 2012

How to read graphs: The public policy version

I follow The New Teacher Project on ye olde Facebooks, and lately they've been on a rant/rave/something about what they call irreplaceable teachers. This morning, TNTP posted this statement and the following graph:

"For decades, principals have assumed that more years in the classroom mean better teaching. But while teacher experience helps, it doesn't guarantee excellence."

Here's what I think this graph really says: everyone improves over time, and partly because of that the lowest-performing teachers likely won't catch up with the highest performing ones.

Let's parse this, shall we?

From a scientific point of view, this graph makes two statements. First, the average low-performing teacher is worse at teaching than the average brand new teacher. What does that actually mean? It means that if you take the worst of the experienced teachers and compare them with all new teachers, the new teachers are better — the problem is, how did TNTP define "worst"?

If you dig deep into TNTP's report, you'll find out that in each of the four — yes, four — school districts they analyzed, they used a different methodology to define low-performing teachers, with the result that they classified between 10 and 24 percent of teachers as low performing.

Now look at the graph again. Despite making up at most a quarter of the teachers supposedly at the bottom of the pile, "Experienced Low Performers" are nonetheless doing better than about 35 percent of all teachers on average? And why is that? Without spending a lot more time on this than I'd like to, I can't be sure, but I'll bet it's because there are some new teachers that are really, really bad — but you wouldn't know that looking at this graph, because it compares bad but experienced teachers with all new teachers.

Takeaway part one: some experienced teachers are worse than the average new teacher, but likely there are many new teachers that are even worse or at least no better.

The second thing to take away from the graph is something TNTP sort of doesn't want you to pay attention to: over a three-year time period, both new teachers and low-performing experienced teachers improve. Why do I say TNTP sort of doesn't want you to pay attention to that? "For decades, principals have assumed that more years in the classroom mean better teaching." And, guess what, they're right.

(By the way, did you notice how the first claim above doesn't actually contradict the second? That's an elementary public relations, um, method.)

So, takeaway part two: everybody gets better with experience.

For me, the conclusion is this: there's a lot of variation in teacher quality, but everybody gets better with time. And, probably because everybody gets better with time, the low performers are going to continue to be low performers compared with everyone else — though, if you read the graph carefully, they won't continue to be quite so low performing as time goes on — in percentile terms, they actually improve a decent amount.

Now, I don't want to be too hard on TNTP, who are doing some interesting and maybe even useful work. What they're trying to say is not inaccurate, really — it really is true that there there are teachers that remain near the bottom of the barrel even with experience.

What I'd encourage, however, is that you think critically about what a graph like this means, and think about the methods — and the motivations — that someone put into this in order to try to persuade you of something.

Now back to another education nonprofit that shall remain nameless.

EDIT (18 Oct 2013): Oh, it was Teach For America. Whatever.