Policy Implications Are Discussed (Often To Poor Effect, in Sociology Journals)

I don’t know its origin, but it appears to have come (like so much else that we cherish because we despise ourselves) from economists. The Quarterly Journal of Economics was the first (in the JSTOR database) to use the term in an abstract, including it 11 times over the 1950s and 1960s before the first sociology journal (Journal of Health and Social Behavior) finally followed suit in 1971.

That 1971 article projected a tone that persists to this day. In a paragraph tacked onto the end of the paper, Kohn and Mercer speculated that inflated claims about the dangers of marijuana “may actually contribute to dangerous forms of drug abuse among less well-educated youth” (although the paper was a survey of college students). “If this is the case,” they continued, “then the best corrective may be to revise law, social policy, and official information in line with the best current scientific knowledge about drugs and their effect.” The analysis in the paper had nothing to do with anti-drug policy, instead pursuing an interesting empirical examination of the relationship of ideology (rebellious versus authoritarian) and drug use. The “implications” are vague and unconnected to any actually-existing policy debate (and none is cited). Being in this case both banal and hopelessly idealistic — intellectual bedfellows that find themselves miserably at home in the sociological space many in the public deride as “academic” — it’s hard to imagine the paper having any policy effect. Not that there’s anything wrong with that.

Fifty years later, “policy implications” has become an institution in academic sociology — by no means universal, but a fixed feature of the landscape, demanded by some editors, reviewers, advisors, and funders. The prevalence of this trope coincides with the imperative for “engagement” (which I’ve written about previously) driven both by our internal sense of mission and our capitulation to external pressure to justify the existence of our work. These are admirable impulses, but they’re poorly served by many of our current practices. I hope this discussion of “policy implications,” and the suggestions that follow, help push us toward more productive responses.

How it’s done

Most sociologists don’t do a lot of policy work. It’s not our language or social or professional milieu, and often not part of our formal training. So what do we mean, in theory and practice, when we offer “policy implications” for our research? There is a very wide range of applications, from evaluations of specific local policies to critiques of state power itself. I collected a lot of examples which I’ll describe, but first a very prominent one, from “Social Conditions as Fundamental Causes of Health Inequalities: Theory, Evidence, and Policy Implications, by Phelan, Link, and Tehranifar (2010). Their promise of policy implications is right in the title. From the policy implications section, here is a list of policies intended to reduce inequality in social conditions:

Policies relevant to fundamental causes of disease form a major part of the national agenda, whether this involves the minimum wage, housing for homeless and low-income people, capital-gains and estate taxes, parenting leave, social security, head-start programs and college-admission policies, regulation of lending practices, or other initiatives of this type.

Then, in the conclusion, they explain that in addition to leveling inequalities in social condition, we need policies that “minimiz[e] the extent to which socioeconomic resources buy a health advantage” — in the U.S. context, this is interpretable as universal healthcare.

These are almost broad enough — considered together — to constitute a worldview (or perhaps a party platform) rather than a specific policy prescription. If this were actual policy analysis, we would have to be concerned with, for example, the extent which policies to raise the minimum wage, raise taxes, house the homeless, and expand educational opportunity actually produce reductions in inequality, and which of these is most effective, or important, or feasible, and so on. But this is not policy analysis, and none is cited. These are one step down from documenting wage disparities and offering socialism as “policy implications.” This is a review paper, mostly theory and summarizing existing evidence — which makes it more suitable than the implications attached to many narrow empirical papers (see below). It has been very influential, influencing thousands of students and researchers, and maybe people in policy settings as well (one could try to assess that), by helping to establish the connection between health inequality and inequality on other dimensions. Important work. But the way I read the term, this is too broad to be reduced to “policy implications” — it’s more like social implications, or theoretical implications.

127 more examples

To generalize about the practice of “policy implications,” I collected some data. I used a “topic” search in Web of Science, which searches title, abstract, and keywords, for the phrase “policy implications,” in articles from 2010 to 2020. This tree map from WoS shows the disciplinary breakdown of the journals with the search term, which remains dominated by economics.

I chose the sociology category, then weeded out journals that were very interdisciplinary (like Journal of Marriage and Family), and some articles that turned out to be false positives, and ended up with 127 articles in these 52 journals.*

First I read all the abstracts and came up with a three-category code for abstracts that (1) had specific policy implications, (2) made general policy pronouncements, or (3) just promised policy implications. Here are some details.

Of the 127 abstracts, only two had what I read as specific policy implications. Martin (2018) wrote, “for dietary recommendations to transform eating practices it is necessary to take into account how, while cooking, actors draw on these various forms of valuation.” And Andersen and van de Werfhorst (2010) wrote, “strengthening the skill transparency of the education system by increasing secondary and tertiary-level differentiation may strengthen the relationship between education and occupation.” These aren’t as specific as particular pieces of legislation or policies, but close enough.

I put 29 papers in the general pronouncements category. For example, I put Phalen, Link, and Tehranifar (2010) in this category. In another, Wiborg and Hansen (2018) wrote that their findings implied that “increasing equal educational opportunities do not necessarily lead to greater opportunities in the labor market and accumulation of wealth” (reading inside the paper confirmed this is the extent of the discussion). This by Stoilova, Ilieva-Trichkova, and Bieri (2020) is archetypal: “The policy implications are to more closely consider education in the transformation of gender-sensitive norms during earlier stages of child socialization and to design more holistic policy measures which address the multitude of barriers individuals from poor families and ethnic/migrant background face” (reading inside the paper, there are several other statements at the same level). I read three other papers in this category and found similar general implications, e.g., “if the policy goal is to enhance the bargaining position of labour and increase its share of income, spending policy should prioritise the expenditures on the public sector employment” (Pensiero 2017).

“Policy implications are discussed”

The largest category, 97 papers (76%) offered no policy implications in the abstract, but rather offered some version of “policy implications are discussed.” It is an odd custom, to mention the existence of a section in the paper without divulging its contents. Anyway, to get a better sense of what “policy implications are discussed” means, I randomly sampled 10 of the papers in this category to read the relevant section. (I have no beef with these papers or their authors, they were selected randomly, and I’m only commented on what may be the least important aspect of their contributions.)

The first category among these, with 5 of the 10 papers, are those without substantive policy contributions. Some have banal statements at the end, which the author and most readers probably already believed, such as, “If these results are replicated, programs should be implemented that will solicit the help of grandparents in addition to parents” (Liu 2016). I also include here Visser et al. (2013), who conclude that their “findings show general support for basic ecological perspectives of fear of crime and feelings of unsafety,” e.g., that reducing crime in the absence of better social protection will not improve levels of fear and feelings of unsafety. I code this one as without substantive policy contribution because that’s a big claim about the entire state policy structure, which would require much more evidence to adjudicate, much less implement, and the paper offers only a small empirical nudge in one direction (which, again, is fine!).

Several in this category offered essentially no policy implications. This includes Wang (2010) who states at the outset that, “the question of motives for private transfers is one with important policy implications” for public transfer programs like food stamps and social security, but never comes back to discuss policies relevant to the results. And Barrett and Pollack (2011), who recommend that health practitioners develop better understanding of the issues raised and that “contemporary sexual civil rights efforts” pay more attention to sexual discrimination. Finally, Lepianka (2015), reports on media depictions of poverty and related policy, but doesn’t offer any implications of the study itself for policy. So, half of these had abstracts that were overpromising in terms of policy.

The other 5 papers do include substantive policy implications, explored to varying degrees. One is hard-hitting but brief: Shwed et al. (2018), whose analysis has direct implications which they do not thoroughly discuss. Their “unequivocal” result is that “multicultural schools, with their celebration of difference, entail a cost in terms of social integration compared to assimilationist schools—they enhance ethnic segregation in friendship networks. … While empowering minorities, it enhances social closure between groups.” The empirical analysis they did could no doubt be used as part of a policy analysis on the question of cultural orientation of schools in Israel.

Three offer sustained policy discussions, including the very specific: an endorsement of prison-based dog training programs (Antonio et al. 2017); a critique of sow-housing policy in the European Union (de Krom 2015); and recommendations for environmental lending practices at the World Bank (Sommer at al. 2017). The last one qualifies, albeit at a very macro level: Gauchat et al.’s (2011) analysis of economic dependence on military spending in metropolitan areas, the implications of which surpass everyday policy debates but are of course relevant.

To summarize my reading, with percentages based on extrapolating my subsample (so, wide confidence intervals): 23% of papers promising policy implications had none, and 38% had either vague statements or general statements that did not rely on empirical findings in the paper. The remaining 40% had substantive policy discussion and/or specific recommendations.

This is a quick coding and not validated. Others might treat differently papers that report an effect and then recommend changing the prevalence of the independent variable — e.g., poverty causes poor health; policies should reduce poverty — which I coded as not substantive. For example, I coded this from Thoits (2010) as not substantive or specific: “policies and programs should target children who are at long-term health risk due to early exposure to poverty, inadequate schools, and stressful family circumstances.” You could say, “policies should attempt to make life better,” but it’s not clear you need research for that. Anyway, my own implications (below) don’t depend on a precise accounting.

Implications

I am really, really not saying these are bad papers, or wrong to do what they did. I am not criticizing them, but rather the institutional convention that classifies the attempt to make our research relevant as “policy implications,” even when we have nothing specific to say about real policies, and then rewards sociologists for shoehorning their conclusions into such a frame.

Let me give an example of an interesting and valuable paper that is burdened by its policy implications. “The impact of parental migration on depression of children: new evidence from rural China,” by Yue et al. (2020) used a survey of families in China to assess the relationship between parental migration, remittances, household labor burdens, parent-child communication, and children’s symptoms of depression. After regression models with direct and indirect effects on children’s depression, including both children who were “left behind” by migrating parents and those who weren’t, they conclude: “non-material resources (parent-child communication, parental responsiveness, and self-esteem) play a much more important role in child’s depression than material resources (remittances).” Interesting result. Seems well done, with good data. The policy suggestions that follow are to encourage parent-child communication (e.g., through local government programs) and teach children in school that they are not abandoned by parents who migrate.

What is wrong with this? First, Yue et al. (2020) is an example of a common model that amounts to, “based on our regressions, more of this variable would be good.” It seems logical, but a serious approach to the question would have to be based on evidence that such programs actually have their intended effect, and that they would be better than directing the money or other resources toward something else. That would be an unreasonable burden for the authors, and slow the production of useful empirical results. So we’re left with something superficial that distracts more than it adds. Further (and here I hope to win some converts to my view), these policy implication sections are a major source of of peer review friction — reviewers demanding them, reviewers hating them, authors contorting themselves, and so on. Much better, in my view, would be to just add the knowledge produced by papers like this to the great hopper of knowledge, and let it contribute to a real policy analysis down the road.

Empirical peer-reviewed sociology articles should be shorter, removing non-essential parts of the paper that are major sources of peer review bog-down. Having different kinds of work reviewed and approved together in a single paper — a lengthy literature review, a theoretical claim, an empirical analysis, and a set of policy implications — creates inefficiencies in the peer review process. Why should a whole 60-page paper be rejected because one part of it (the policy implications, say) is rejected by one out of three reviewers? This is very wasteful. It puts reviewers in a position to review aspects of the work they aren’t qualified to judge. And it skews incentives by rewarding the less important parts of our work. Of course it’s reasonable to spend a few paragraphs stating the relevance of the question in the paper, but not a whole treatise (in the front and back) of every paper.

Advice for sociologists

1. Don’t try to pin big conclusions on a single piece of peer reviewed empirical research. That’s a sad legacy of a time when publishing was hard, sociologists had few opportunities to do so, and peer reviewed journals were the source of validation we were expected to rely on. So you devoted years of your life to a small number of “publications,” and those were the sum total of your intellectual production. We have a lot of other ways to express our social and political views now, and we should use them. The fact that you have a PhD, a job, and have published peer reviewed research, are all sources of legitimacy you can draw on to get people to pay attention to your writing.

2. Write for the right audience. If you are serious about influencing policy, write for staffers doing research for advocacy organizations, activists, or campaigns. If you want to influence the public, write in lay terms in venues that draw regular people as readers. If you want to set the agenda for funding agencies, write review pieces that synthesize research and make the case for moving in the right direction. These are all different kinds of writing, published in different venues. Crucially, none of them rely only on the empirical results of a single analysis, nor should they. The last three paragraphs of your narrow empirical research paper — excellent, important, and cutting-edge as it is — will not reach these different audiences.

3. Stop asking researchers to tack superficial policy implications sections onto the end of their papers. If you are a reviewer or an editor, stop demanding longer literature reviews and conclusions. Start rewarding the most important part of the work, the part you are qualified to evaluate.

4. If you are in an academic department, on a hiring committee, or on a promotion and tenure committee, look at the whole body of work, including the writing outside peer-reviewed journals. No one expects to get tenure from writing an op-ed, but people who work to reach different audiences may be building a successful career in which peer-reviewed research is a foundational building block. Look for the connections, and reward the people who make them.

*The full sample (metadata and abstracts) is available on Zotero. Some are open access, some I got through my library, but all are available from from Sci-Hub (which steals them so you don’t have to).

—

This post was previously published on Family Inequality with a Creative Commons License.

***