Friday, July 11, 2008

THE RESEARCH TRIANGLE. SORT OF.

As a student of education policy, my foremost personal goal is to keep my teacher sensibility and credibility from permanently draining away. Reading research reports, with their inevitable earnest calls for still more research, authoritative policy claims and suggestions, one understands that the base of the research-policy-practice triangle is practice, the apex being the conjunction of research and policy. Practice is a kind of collection pan (or audition stage) for ideas generated by scholarly investigation and clever policy-making.

Teachers serve as both subjects and objects of reform, but—with the exception of classroom-based action research, a phrase that causes your average doctoral student’s lip to curl—seldom go out looking to gather data or propose policy on a wider scale. They wait for the next finding, prescription or mandate to come down the pike, then either wrap the new guidelines around their old practice or attempt to ignore scholarship and policy altogether.

Some see this as evidence of a regrettable autonomy still present—despite the best efforts of education publishers—in the act of teaching. Others (people who work in schools, mainly) see these habits as a defense mechanism. No matter how many studies are conducted, no matter how large the data sets and innovative the statistical modeling, no matter how muscular the policy lever—kids keep coming to school, and teachers have no choice but to keep their heads down and teach them, somehow.

There’s been a little dustup over another one of Jay Greene’s papers, just released by the Manhattan Institute: Building on the Basics: The Impact of High-Stakes Testing on Student Proficiency in Low- Stakes Subjects. Greene’s shabby scholarship was roundly criticized about six months ago, when he and colleague Catherine Shock counted course titles in university catalogues, and developed a math-to-multiculturalism ratio, proving that Ed schools and teachers didn’t give a rip about mathematics achievement. Greene took some heat for that, eventually re-characterizing the data as “an amusement.”

This time, Greene and co-authors Marcus Winters and Julie Trivitt investigate the question of whether narrowing curriculum to put greater emphasis on two tested subjects (math and reading) in Florida schools might have a negative impact on student achievement in other subjects. Their answer? No.

Eduwonkette and other bloggers have raised lots of questions about technical aspects of the study’s research design, and the fact that the report was embargoed, not subject to traditional peer review, before hitting cyberspace; these are the kinds of inquiries that won’t be satisfied until long after the titular core message (“building on the basics” is a good and justifiable thing) has become conventional wisdom. Sherman Dorn cranked out an engaging essay on “the reworking of intellectual credibility in the internet age (which) will involve judgments of status as well as intellectual merit.” And an “Anonymous Peer Review” poster on the Ed Week blog set up, point by point, technical issues of concern in the piece, including some that are obvious even to research lightweights. For example, the study uses two years of math and reading achievement data and one year of test scores in science. That’s right—one year. I’m no statistician—but isn’t it hard to do growth comparisons with only one set of numbers?

Some tidbits from the report:

We find that students attending schools designated as failing in the prior year made greater gains on the state’s science exam than they would have done if their school had not received the F sanction.

There are two important reasons that we might expect schools deemed to be failing to respond positively. Those that have received an F grade for the first time may be shamed into improving their performance. Those that have received at least one failing grade may decide to raise their performance because they fear attrition of their student body.

Though there is some disagreement about which aspect of the accountability policy was effective (the threat of vouchers or the shame of an F grade), each of these analyses found that the policy improved the math and reading proficiency of students in public schools designated as failing.

While the hard-core researchers duke it out, let me step aside here and think like a teacher. Greene and his colleagues, through the Manhattan Institute (“turning intellect into influence”) have released a study strongly suggesting several things, some of which will be appealing to Florida legislators:

· Don’t worry too much about schools cutting back on science or other academic subjects to meet math and literacy targets, because it doesn’t really matter, in the long run. Science scores are likely to go up, statistically, if math and reading scores go up—and that’s good enough for us.

· Schools can “decide” to raise their performance after being shamed and threatened.

· The policy of giving schools failing grades improves their reading and math proficiency. And now we have evidence that it improves all subjects, whether we spend time teaching them or not. Failure, therefore, is a great motivator.

· Sanctions work, and are much less expensive than investing in improved instruction, engaging curriculum or retaining effective teachers.

Florida is a state with nearly 11,000 National Board Certified Teachers. The National Council on Teacher Quality, commenting acerbically on the recent—generally positive—National Research Council report on National Board Certification, said this:

Teachers from advantaged schools and states with financial incentives were more likely to participate in the certification process. Board-certified teachers are more likely to remain in the field than other teachers, and are more likely to move to assignments in high-performing schools with lower rates of poverty.

When policy and research “decide” that perhaps a robust science program really isn’t a necessity in a failing school, where are the Board-certified teachers going to migrate? Where might an accomplished science teacher seek refuge from the trickle-down effects of policy and research, along the bottom of the triangle?

8 comments:

Anonymous said...

"Bottom of the triangle"? You certainly reach for the odd metaphor, which I suppose is consistent with the random nature of your remarks.

I don't see anything "acerbic" in the comment by NCTQ that board-certified teachers look to work in schools where children are more advantaged. There is no news in that finding.

Arguing for the importance of raising basic skills in reading and math is not the same as arguing that science is unimportant.

TexMex said...

Arguing for the importance of raising basic skills in reading and math is not the same as arguing that science is unimportant.

Basic skills in reading and math can be developed in some kind of context like science or social studies or the arts.
But this would require teachers with the freedom to work together and make the best use of school resources especially in terms of the most important resource.... a talented and creative teacher.
So long as we are shunting kids to a corner to be supervised by a teacher's aid while they practice bubbling in practice test sheets.
But then again bright and educated teachers are in short supply and looking the other way while students cheat on exams is one way to "decide" to get better scores.

TexMex said...

http://www.sbac.edu/~werned/DATA/RESEARCH/journals/Excep%20Children/inclusion.pdf

Nancy Flanagan said...

Thanks for commenting, texmex and anonymous.

texmex' observation that strengthening math and reading will almost certainly have a spillover effect on science (and history, etc.) achievement is spot-on. Reading itself is skill more than "subject" and all academic achievement depends on basic literacy and numeracy. Greene makes this observation as well, but then--instead of demonstrating that all "subjects" are interdependent when measured by standardized tests--turns to his thesis about the importance of shame and threats, and disproving (with some extremely thin data) the arguments for keeping a strong science program in the school curriculum.

Perhaps I should have included more of NCTQ's commentary on National Board Certification, which can be summarized as--"Well, yes the NRC report does conclude that Board-certified teachers engender higher achievement levels, but they are responding to financial incentives, and don't stick around in disadvantaged schools, where they're most needed." In other words, Board-certified teachers prefer greater monetary, collaborative and psychological rewards. You're correct, anonymous--nothing new there.

My point in including it was noting that when research reports suggest (as this one clearly did) that paying attention to science instruction was not particularly important in low-achieving schools, good science teachers would go elsewhere. The NCTQ's remarks beg the question: why would they do anything else?

While Greene is careful not to say "science is unimportant," he does say that narrowing the curriculum does not impact overall achievement. And he does so with shaky evidence, using data from a state where policymakers are prone to the grand gesture. If you follow his conclusions to their logical ends, it would benefit every school in FL to get a shameful and threatening grade, then reduce its curriculum to "basics."

I doubt if the "bottom of the triangle" metaphor would feel "odd" to most teachers. Researchers and policymakers have a bi-directional influence on each other (the "Building on the Basics" study is a good example). And both have an influence on practice. But practitioners have little to say about the credibility or efficacy of research or policy, which trickle down into schools and classrooms.

Anonymous said...

Teachers who are talented and creative teach kids in creative ways instead of shunting them to corners to practice test sheets.

Anonymous said...

I don't think you (Nancy) understand the implications of Greene's study at all.

Greene suggests two ways that improved science scores could result from "high-stakes" policies for reading and math. First, if the high-stakes environment pushed schools into improving generally and, secondly, that improving reading and math makes it easier for kids to learn science. To the degree that these things are true, it seems that science teachers would welcome policies that improve kids' reading and math.

Why speculate about a flight of science teachers away from schools that are trying to pull up their reading and math?

Nancy Flanagan said...

From the report:

"If schools reallocate time and resources away from important but low-stakes subjects and toward the high-stakes subjects, with the result that students achieved in the high-stakes subjects at the expense of proficiency in the low-stakes subjects, we would say that the policy 'crowded out' learning in the low-stakes subjects." (But their research "disproved" this possibility.)

"We find that students attending schools designated as failing in the prior year made greater gains on the state’s science exam than they would have done if their school had not received the F sanction."

With due respect, anonymous, I understand the purpose of the report: to shoot some holes into that first "crowding-out" theory, a practice that has been widely reported anecdotally. Using one year of data at the 5th grade level, they suggest that shaming schools into higher levels of reading and math proficiency will raise science proficiency as well, a kind of side benefit of improving literacy and numeracy.

At the 5th grade level, in the first (and only reported) year of improvement, this may well be true, a kind of rising achievement tide lifting all boats. But generalizing from this--we don't need to worry overmuch about specific science instruction, because raising literacy and math scores lift science scores as well-- is dangerous. At some point (and one hopes that it's well before the 5th grade) kids deserve focused science instruction.

Real, skilled science teachers fleeing "F" schools is but one possible outcome of policymakers taking this report to heart. The policy implications in the 2nd paragraph above are only tangentially connected to the stated purpose of the report, which had several axes to grind, and didn't make a convincing case for any of its findings.

Anonymous said...

All you're doing is setting up straw men - maybe policy makers will read the report to undervalue science, maybe more emphasis on reading and math means less "robust" science, maybe "real" science teachers will jump ship.

Science scores went up in the "F" schools, possibly because, as Greene suggests, kids whose skills improve in reading and math learn science better. So, again, why would science teachers jump ship when things are looking up?