Showing posts with label research. Show all posts
Showing posts with label research. Show all posts

Wednesday, March 13, 2013

Twitter: Democratizing policy and research, or a HUGE waste of time?


I (@mlinic1) joined Twitter for several reasons. Professionally, it serves as an excellent way to promote my own research and that of the Forum on the Future of Public Education (@forum_future_ed). Further, Twitter provides an excellent opportunity to customize your own access to the news of the day by following people and organizations that are important to you. As such I’m able to keep abreast of recent developments in politics (@barackobama, @corybooker, @marcorubio), follow favorite entertainers (@neilhimself, @colbertreport, @nathanfillian), follow research organizations and think-tanks (@occrl, @heritage,  @nepctweet, @hooverinst), and keep up on the news (@newsbreaker, @educationweek). Perhaps the most engaging aspect of Twitter, for me, is following the discussions of individual educational researchers, advocates, and media figures (@clubienski, @schlfinance101, @mpolikoff, @saragoldrickrab, @michaelpetrilli, @shermandorn, @michellerhee, @leoniehaimson, @dianeravtich, @chingos). When I started using Twitter (@mlinic1), I was concerned that it was consuming too much of my time, reducing my productivity, and distracting me from meaningful work. I am amazed by productive scholars that are able to use Twitter engagingly, while promoting their scholarly pursuits, and challenging poor research (@saragoldrickrab and@schlfinance101 are great examples of such scholars).

In recent years, research and policy discussions have experienced a great democratization with the expansion of new media forms such as blogs, Twitter, and Facebook. Debates about policy are no longer contained to newspapers and news broadcasts, nor are debates about research validity contained to after-the-fact responses in journal articles (but, for a great example look here.) I have found the use of Twitter to provide two benefits to me as a new scholar. First, despite the restrictive nature of 140 characters, excellent and enlightening discussions can emerge on Twitter about the intricacies of education research. For example, in 2012 Matthew Chingos and Paul Peterson released a widely discussed study of voucher effectiveness; however, despite endorsement by the Wall Street Journal, the study’s methods were quickly attacked on Twitter. Additionally, wisdom can be found 140 characters or less. Following a discussion between several educational researchers about the validity of voucher studies, Dr. Morgan Polikoff asked “So, until such time as a review appears, how would you discuss voucher literature,” and Dr. Chris Lubienski responded, “Cautiously, considering the source, research design, etc.” While only requiring 57 characters, such advice is something all researchers (especially those working with policy) would do well to heed. 

Second, understanding how Twitter is used by organizations and individuals to disseminate and absorb information regarding policies and research is an interesting prospect, especially for any researchers interested in the use of research in policymaking. As discussed by Michael Petrilli, simply understanding who is following who, says quite a bit about how information is disseminated and echo chamber that exists (he provides a great image here). I am currently pursuing further research on this subject, examining organizational approaches to research dissemination. I will be presenting a paper on this subject at AERA 2013, and will share more about my findings in coming months.     

Thursday, January 17, 2013

Whose Opinions Count in Educational Policymaking?


Currently in the U.S., major educational reforms are being incentivized, which has effectively created pressure to innovate.  For instance, Race to the Top, a four-plus billion dollar federal competition sponsored by the U.S. Department of Education, has been designed to advance major, specific policies across the states (Race to the Top Fund, 2012).  Related, a narrative of U.S. educational crisis, whether or not it is overstated or dubious, continues to hold sway in many circles.  A crisis, real or manufactured, presents opportunity for would-be reformers.  As such, some individuals and organizations may be advancing their policy agendas by engaging media through individuals who possess little or no educational expertise. 

With this in mind, we (Malin and Lubienski, in review) became interested in assessing the relationship between expertise and media impact. To do so, we made use of two educational expert lists (Hess, 2012; and Welner, Mathis, and Molnar, 2012).  We treated educational press mentions, blog mentions, and newspaper mentions in combination as a dependent variable representing “media impact.”  Likewise, we treated four criteria— educational attainment, Google Scholar-listed publications, book points, and highest Amazon rankings— in combination as an independent variable measuring “expertise.”  We used linear regression to assess the strength and direction of relationships between these variables.

When these expert lists were combined, we found a non-significant positive relationship between our measure of expertise and our measure of educational impact (see figure below).  When we constrained our analysis to the NEPC list, however, expertise significantly predicted media impact.


We conclude that media impact is at best loosely related to expertise, which is troubling and points to the responsibility of the media to vet experts before citing them or their work.  Certainly, future research should be aimed at exploring and better understanding these relationships.  Perhaps most importantly, we join the growing chorus of individuals who seek to re-establish tighter relations between research, policy, and practice.  Education is immensely important and policy changes should be carefully discussed and weighed prior to implementation.  This is most likely to occur when individuals with educational expertise are positioned to inform the process.

Tuesday, January 15, 2013

Family background matters, so measure it better.


A few months ago, the Census Bureau released data based on a relatively new, more sophisticated measure of poverty.   The old measure had been in place since the 1960’s and did not account for the realities of today’s living expenses.  The new measure considers housing, medical, and child care costs and does a much better job adjusting for support received through federal assistance programs.   In areas with high costs of living like California and Hawaii, the new measure classified substantially more residents as poor, while the reverse was often true in areas with lower costs of living. 

This new measure is by no means perfect, and it certainly does not do anything directly to help poor families in the U.S.  But this measure may allow policy analysts to better assess the needs of American families and the relative effectiveness of safety net programs.  The change is solely on paper, but it is an important change nonetheless.

The K-12 education sector is long overdue for improvements in how it routinely measures the social background of children.  More often than not, a student’s participation in the free and reduced school lunch program and his or her LEP status are the only available indicators of family background.  Although additional indicators are sometimes collected for research or special programs and assessments, free/reduced lunch and LEP tend to be the only measures that are available for all schools in regular enrollment data.

Free/reduced lunch is a lousy indicator of socioeconomic status for a couple of reasons.  First, it classifies all students into just one of three categories (free lunch, reduced lunch, no lunch support), losing valuable detail in the process.  With this approach, a family of four making $28,000 per year will be indistinguishable from a family of four making $14,000 per year, as both would be classified as free lunch.  Second, free/reduced lunch is based on income primarily, and income by itself is not a very good indicator of social class.  In Class and Schools, Richard Rothstein points out how the use of income via free/reduced lunch as the primary measure of socioeconomic status can lead to the misrepresentation of some schools’ populations.   One school that was nationally recognized as being both high poverty and high performing was actually a public school where many Harvard and MIT graduate students sent their children.  True, graduate students don’t make much money, but few sociologists would regard this group as a high needs population. 

Social scientists have used hundreds if not thousands of different indicators to measure class and socioeconomic status, and the measures will often vary depending on available data.   However, a handful of variables emerge more often than the rest, due to both availability and their quality as predictors of outcomes in the social sector.  If I had to pick a single variable to add alongside family income, parental educational attainment would be a good choice.  A common way of representing SES in richer datasets is to combine information on income, parent education, and occupational status or occupational prestige (e.g.).  While converting occupational status into a number can be tricky, it’s a bit more straightforward for parental education levels. In many cases, measures of parent education are even reduced to maternal educational attainment due to the prevalence of single-parent households.  Thinking back to the Boston public school that enrolls the children of Harvard and MIT Ph.D. students, it is easy to see how a combination of income and parental education levels would give you a much more accurate sense of the average socioeconomic status of some families. 

I am not a lawyer, and I don’t know what legal justification the feds or states would need to collect additional personal information from parents; but, from a researcher’s perspective, the case is easy to make.  The link between social background and academic achievement is well established, but the debate over the extent to which these links should influence educational policy continues.  Achievement gaps between racial and socioeconomic groups remain large, and school segregation along these lines may be getting worse.  Meanwhile, wage, wealth, and income inequality in the U.S. continues to worsen, as it has been doing since the mid 1970’s.  In this context, there is substantial need for better measures of students’ social background, particularly given the shortcomings of current measures. 

Moreover, the weaknesses of free/reduced lunch as a socioeconomic indicator are not just an inconvenience for researchers these days.  For better or worse, many states and districts are now using statistical models to influence the retention, tenure, and promotion decisions of teachers.  Better background variables on students may help improve these models.  With all of these factors in mind, one could make the case that looking beyond free/reduced lunch is not only in the best interest of federal and state departments of education but also that it is their responsibility to do so. 

Monday, July 16, 2007

Community Organizing and Urban Education XI: Scholarly Participation in Organizing Campaigns: Research that Makes a Problem “Hit Home”













[To read the entire series, go here.]

[Note: post has been revised for more accuracy]

Given the enormous gulf between the realities of life in the academy—even in relatively marginal research institutions like mine—how might education scholars contribute to organizing campaigns?

First, I think it is important that scholars resist the urge to “help” by trying to turn organizers and leaders into scholarly researchers, into people like us. Organizers generally use research very differently than do scholars. In organizing, research done by participants is almost always deeply intertwined with the day-to-day activity of social action. As I noted in a review of a book by Oakes and Rogers, in social action groups

data collection is [usually] integrated into efforts to build relational power. Survey efforts, for example, become opportunities to engage with and recruit potential members. Leaders and organizers in Alinsky-based groups do not learn to be “scholars” or “researchers” separate from their identities as activists. In fact, in my experience “research” is sometimes pursued as a thinly veiled strategy for engaging people in an issue. And it isn’t unusual for these organizations to simply hire someone to get the data they need to act, allowing them to keep their limited resources focused on activities more directly related to organizing.

Community organizing groups need research to serve very particular purposes. (Exactly what skills a particular group or individual needs (or wants) at any particular time is always an emporical issue, of course.) In any case, at least three key uses of research come to mind: research that brings a problem into stark relief, research that helps define a particular solution, and research that contests assertions made by the opposition.

Here, I want to talk about the first use, making often vague social problems more concrete, making them “gut” issues around which one might recruit participation.

Of course, it would also be wonderful to have more scholars with a strong social action background. But given limited resources this may be more an issue of recruiting particular individuals than a strategy for supporting groups.

In the last few years, the education committee of the organizing group I work with, MOVE, has been focusing on health in schools as an area ripe for intervention. The reasons for this choice of issue are complicated and emerge to some extent from the specifics of our situation in Milwaukee. In the simplest sense, focusing on health allows us to seek out support for schools that is outside the usual funding streams and that, therefore, doesn’t “count” under the caps that currently limit funding in Wisconsin. Further, we believed that it would allow us to seek funding on a city instead of a state level, avoiding the need for power to move the state legislature that, as a city organization, we lacked.

To rally support from our own members and from other groups, we thought we needed some simple document that would lay out the challenges the health problems of poor children and their effects on learning in the starkest terms. We didn’t need a document that went through everything; we didn’t need a research study; we didn’t need a forty page review of the literature; and we didn’t need a document that would meet the requirements of a peer-reviewed journal publication. We needed a brochure.

The fact is that I knew little or nothing about health problems when I started working on the brochure. But I knew how to read through a mass of research documents and pull out key information, and I knew how to locate information that seemed to come from reputable sources.

So I pulled together the documents we’d been collecting through the preparatory interview research we had been doing, and searched on the Internet, and searched on the proprietary databases available at my university, and I searched through the archives of our local newspaper.

What I was looking for was data that would make readers stop in shock. What I needed was information that we could state publicly and not fear being attacked about for its accuracy.

The brochure that I completed, with the input of my committee is pasted in, above. You can see a more readable version here.

The first panel on the inside left is the most important. As I have noted before, it is crucial in organizing to find a “gut” issue that makes people want to stand up and act. An abstract crisis, a need for more “money” in general is not very motivating. But thousands of kids with their teeth rotting in their heads, thousands of kids that can’t see well enough to read easily, that’s motivating on a visceral level.

Importantly, I don’t waffle about the data. I make clear statements about the condition of child health. Only in the footnotes do I record where I got this information and possible limits of the data. For example, two large studies of poor inner-city children in different cities that both showed that 50% of the children had vision problems. In the full text of the brochure I simply state that 50% of poor kids have problems with vision, and then in the footnotes I note my extrapolation. In other cases, I have not bothered to put conditions on my knowledge, even though (as with most research) the data may not be as clear as the specificity of these numbers imply. For example, some of the data comes from statements by the Milwaukee Public Schools (MPS) superintendent. I think I know where he’s getting this data, and in one case I think it’s from a study that seems somewhat more limited in the population it covers than he indicates. The fact is, this is frequently true of data like this on a district level, often gathered for one purpose and used for other purposes. I don’t go into this in the brochure. It just isn’t important for the reader. The superintendent’s conclusion and the magnitude of his number is quite reasonable given the other research I’ve read, even if I’m not certain about the exact percentage he cites. There is no reason for us to complicate issues by constantly qualifying our statements because of small (possible) differences or uncertainties that, in the big picture, are pretty meaningless. These are the kinds of issues that matter to scholars, but not to policymakers, politicians, and people on the street. Statistics I could not defend in this way I did not include. In any case, the data is as accurate as I could make it, and readers know where to go to pursue it in more detail if they want. As far as I can tell, there is nothing in the brochure that could subject our organization to critiques from reputable research or professional groups (and I passed a draft by a key local stakeholder just to make sure).

Then, in the next panel, the brochure goes on to lay out the kinds of issues these health problems may raise for learning. And in the final panel I give some key quotes. Some of these are from Milwaukee, and some aren’t. And the picture isn’t of a Milwaukee kid, but it’s a good picture. None of this is really important to the reader.

On the outside of the brochure, I summarize the key limitations of MPS’s approach to solving these health problems. As a committee, we worked to lay these issues out without attacking the school district. And we tried to emphasize the district’s limited funding so that people wouldn’t misread us as saying the district should cut other programs to fund health instead.

Finally, in middle panel on the outside, we lay out our plan for fixing these problems. It may be surprising, but this is probably the least important part of the brochure. The reality of a campaign often necessitates changes in abstract plans stated ahead of time. But at least we make it clear that we’re not just complaining. We have plans to do something about the problems.

I don’t want to overstate my case, here. The brochure isn’t perfect. In retrospect there are some other things I wish I had done. But it's the first time I've tried to put something like this together, and I'm going to cut myself some slack. And I believe that it has been a useful organizing tool for us. It looks fairly professional and gives us credibility among health professionals, administrators, and politicians. It also gives readers the sense that there is organized, coherent, and well structured campaign. (This is true regardless of the "reality." As Alinsky said, what is key is not the power you actually have but the power others think you have.) Participation on our committee increased significantly after we passed it out to MOVE members. And instead of moaning about the fact that life is difficult, it lays out a path for some postitive change. It is empowering to some extent, just to have the brochure.

In fact, we have actually been successful at some important first steps in getting better health care for Milwaukee kids. I’m not going to talk about that right now, however. Our effort is still ongoing, and there are good reasons not to talk too much about it publicly until it is farther along.

Let me conclude this long post by contrasting this example of scholarly contributions to social action with that discussed by Oakes and Rogers in their book. They describe the development of a robust education policy round-table that they put together with a range of different organizations, providing them with research and facilitating dialogue.

Their's seems like a wonderful model. But it clearly requires dedicated funding and a significant commitment of other resources. I don’t have either of these available to me. Mostly it’s just me and my computer and my “enormously messy” office, as my daughter says.

This example shows how a relatively isolated scholar (like most of us, I bet) with access to the basic data available to all professors, some limited facility with Microsoft Word, and a week or so of time (spread over a couple of months) can put together a key “research” document to support what could end up being a major campaign. Perhaps my key skill, here, was in understanding just which data might be reliably leaned on without undermining the credibility of our effort.

It is crucial to emphasize, however, that I could only create this document because I am a long-term member of this organization. In our meetings, other members helped restructure the brochure and change the layout. In fact, it probably helps that I generally don’t emphasize the fact that I’m a scholar in my participation. I understood what this group needed because I’m a part of this group, and the group didn’t have any trouble working with me, or trusting me to put this together, because I’ve been there for a long time. Without this kind of embodied knowledge, I probably would have ended up creating yet another “lit review” that wouldn’t have really helped them that much.

Of course, the reverse could be true as well. This familiarity may have also made me less self-critical about what I was doing. Others may want to respond with their own opinion.