I have a somewhat embarrassing confession: I’m something of a Jennifer Doleac fanboy. Her findings are often uncomfortable for someone in my line of work, and watching humane reform intuitions get fed into the wood chipper of causal inference and study design is the kind of fun only a real nerd, or maybe a masochist, would acknowledge. But it’s possible to be a fan of skepticism and rigor, and Doleac and her econometric peers are trying to bring badly needed discipline into a field that has often run on good intentions, weak evaluations, heroic anecdotes, and sloganeering.
My view is that reformers are often too quick to believe that whatever matches our priors and shares our values must work. We can confuse moral attractiveness with effectiveness, leading us to fund and scale interventions that feel humane even if their outcomes are not, because staff believe in them and non-profit leaders defend them rigorously. At the heart of anticarceral advocates’ passion is the idea that when the punishments are cruel enough, any proposed program must surely be a better alternative. I’d say that this often justifies trying, but it’s rarely enough to justify triumphalism, even though that’s what foundations, media contacts, and conference organizers demand.
Bring on the skepticism and careful causal inference study designs, I say. We need economists to invade with their natural experiments, cleverly acquired datasets, and difference-in-differences designs. They’re not wrong to insist, for instance, that “people who got the program did better” is a whole different sentence than “the program caused them to do better.” (Though it can be an annoying refrain!)
But I’d just like to point out that criminal justice is not the first field to be visited by the economics fairy slipping identification strategies under our pillows. Development economics went through this first, and the lessons are instructive. The credibility revolution, in Joshua Angrist and Jörn-Steffen Pischke’s phrase, promised fewer just-so stories, fewer grand structural theories, and more credible answers to causal questions about poverty, education, health, and labor markets. Those have to be some of the most important questions in existence, so their attention was welcome. Instead of asking whether a program sounded enlightened and charitable, economists wanted to ask what happened to the control group, the folks who didn’t receive it. But the underlying appeal was something a bit darker: we could stop arguing, forever, about ideology and start checking the methods section. Swap theory for empiricism, and get rid of the culture wars to boot? Sign me up. And, as it turned out, the Nobel committee did.
But there’s a warning sign from that episode that criminal justice has yet to absorb. Consider child malnutrition projects: the Tamil Nadu Integrated Nutrition Project combined supplementary feeding, health measures, and nutrition education for pregnant mothers in rural India. It targeted areas with severe malnutrition. It was widely treated as a success, and with the best of intentions, Bangladesh tried to replicate it. The Bangladesh Integrated Nutrition Project borrowed the basic model: nutrition support, education, community-level intervention. Yet Bangladesh’s version did not produce the same result! Nutrition knowledge improved (you can almost always improve knowledge), and yet malnutrition did not fall in the way the Tamil Nadu numbers had promised.
The key difference seems to have been social rather than technical. In Tamil Nadu, the women receiving nutrition education were more likely to control the household decisions that mattered for the child’s diet. In Bangladesh, men often did the shopping, and mothers-in-law often controlled the distribution of food inside the household. Teaching pregnant mothers what to feed their children was not a useful intervention, it seems, because pregnant mothers were not the people with authority over food, and those who were did not always share the intervention’s priorities. The “same” intervention was not the same once it moved a thousand miles northeast and across social worlds.
This is a story that Nancy Cartwright made famous in her criticisms of the methods of the econometricians: “It worked there” is a different sentence from “it will work here.” Her 2012 book with Jeremy Hardie, Evidence-Based Policy: A Practical Guide to Doing It Better, lays out the case at length; the address she gave to the Philosophy of Science Association compresses it into a memorable form. Randomized controlled trials have become a gold standard, in part because they provide clear statistical evidence of causation. They seem to work best in medicine, but the ideal here is to be able to calculate metrics like “number needed to treat” for social problems like we do for vaccines, statins, or blood-pressure medications. Policies become treatments, and public health becomes a thinkable target. But while RCTs can tell us what worked in a place and time and for one group, they can’t tell us what will work in a different place, time, and with different people. A policy does not cause an outcome through the magical application of ceremonial spellcraft. It needs helpers, which Cartwright calls support factors: the surrounding conditions that allow a policy to do its causal work. Things like institutional norms, implementation staff, household authority structures, trust, incentives, administrative capacity, background law, and social meaning.
This is what philosophers of science call a ceteris paribus problem: the policy intervention is but one ingredient in the cake of outcomes, but you have to eat the cake you bake, not just the one line of the recipe. An experimental finding holds all things equal. But when it comes to human lives and social worlds, the other things are never equal. And criminal justice is where some of the deepest inequalities lie.
Still, the method works for medicine, right? Would you want to take a pill that didn’t have such high levels of evidence? (Forget the massive supplements industry.) Sadly, policy reforms are not pills. Here are some things that are not pills: diversion programs; mentoring programs; public defender trainings; restorative justice programs; therapeutic modalities. These are all social relationships. They’re routed through courts, schools, police departments, probation officers, hospitals, families, neighborhoods, and bureaucracies. Each of these sites and institutions will have its own culture, incentives, and pathologies. We know judges matter, to the extent that outcomes can be measured based on when they had lunch. We know prosecutors matter, and that background charging practices and negotiation strategies will change outcomes. The defense bar, youth culture, school discipline regimes, housing costs and availability, current levels of community organizing and trust, and the background levels of police harassment could all figure into the effectiveness of a small policy change you’re trying to measure. Change one without marking it and you could fully transform the effectiveness of the measured intervention, all while leaving the brand name and logic model untouched.
Which is why econometric skepticism is both necessary and dangerous: the cure can turn into a poison. Doleac and others are asking questions reformers might like to avoid: did this actually work, or did it merely flatter our biases? Did we reduce harm or tell ourselves a good story? Are we changing our clients’ lives or simply selecting clients most likely to succeed?
It’s dangerous to duck those questions, but it’s also dangerous to turn these questions into political weapons. “It didn’t work here and now” doesn’t mean “it won’t work anywhere ever,” either. The strongest econometricians know not to overgeneralize, but the policy communication environment sometimes confuses “this policy produced unintended consequences in one labor market” with “civil rights reform always backfires.”
Doleac’s own work is instructive here. Her work with Jillian Carr on juvenile curfews is a gift to reformers, and I’ve been using it myself the last few years: it suggests that a policy sold as public safety may actually increase shootings during the very hours it is supposed to control, plausibly because curfews thin out the streets and remove the witnesses and informal guardians whose presence deters violence. That is the kind of finding reformers should cite when law-and-order officials claim that restriction is safety.
But her Ban the Box work with Benjamin Hansen is more uncomfortable for us, because it suggests that a policy meant to reduce discrimination against people with criminal records may lead some employers to substitute race, age, and gender as proxies, with the effect concentrated on young Black and Hispanic men without college degrees. We have to be willing to consider the possibility that “racism finds a way.” We should be able to hear that possibility without treating the person who raises it as an enemy of reform.
It’s worth noting that the literature since Doleac and Hansen has grown messier rather than cleaner. Amanda Agan and Sonja Starr’s 2018 audit study in New Jersey and New York City reproduced something close to the Doleac-Hansen finding: at employers who had previously asked about records, the white-Black callback gap widened sharply after Ban the Box took effect. Evan Rose’s 2021 study of Seattle’s 2013 law, using administrative employment and conviction data, found otherwise: negligible employment effects for people with records, and therefore little evidence of either benefit or racial backfire in that setting, plausibly because employers were running background checks later in the hiring process or applicants were sorting into employers that did not screen at all.
Anne Burton and David Wasser’s 2025 reexamination went further: they returned to the Doleac-Hansen specification, corrected miscoded laws, and ran the same analysis on the larger American Community Survey. The original Current Population Survey estimates held up; the ACS estimates collapsed to null effect, with employment effects near zero and statistically inconsistent with the published results. Burton and Wasser argue that the original finding was driven by small-cell sampling fragility in a survey poorly suited to the question. Methodological rigor itself comes in degrees.
The four sets of findings present a more complicated picture than any single one. Different labor markets, different enforcement regimes, different employer pools, different measurement strategies. Where the dataset itself is the variable, the original effect does not always survive. The Ban the Box literature is, among other things, a working illustration of Cartwright’s thesis: a nominally identical policy producing different effects under different support conditions, with no easy aggregation into a single portable number.
We should therefore resist the idea of the magic mechanism: things that work once won’t necessarily work again. A single experiment can’t make natural laws. Both favorable and unfavorable findings demand that we ask tough questions: what is the mechanism? What is the target setting? What are the support factors? What complementary policies could prevent the backfire? Do we have to abandon Ban the Box, or should we pair it with stronger anti-discrimination enforcement, record sealing, and employer incentives? Would certificates of rehabilitation help? Should we pursue public hiring commitments, or direct subsidies for employment? I’m always thinking of Cartwright here, because she’s the philosopher asking us to look closer before we leap to conclusions from the unstable foundation of a single set of results.
This goes beyond an academic and intellectual enterprise, of course. Funders define what counts as evidence with some of these questions in mind. The field’s grammar is being rewritten to incorporate these programmatic and evidentiary demands. Organizations and programs are learning to describe ourselves in evaluable terms. We end up making outcomes promises with fundable time horizons. Researchers are being educated with the same institutional concerns, so they can have careers that are fundable and rewarding. Arnold Ventures did not invent this grammar, but with its money, research agenda, and commitment to rigorous causal evaluation it has become one of the places where the grammar is being formalized. Doleac is now one of its most visible figures.
In the end, interventions that can be counted, randomized, sorted, and tied to administrative datasets will be legible and fundable, and interventions that focus on trust-building, power-shifting, legitimacy-repair, and rights-enforcement will tend to fall by the wayside.
Some of that discipline is healthy. If you can’t say what you’re trying to change, for whom, and how, and how you can tell… that’s a problem no matter whether you use Stata or Microsoft Paint. But notice that there’s a dangerous slippery slope between “worth doing” and “evaluability.” The slide favors short term discrete and administratively measured interventions. What’s lost is long-term relationship work, community organizing, youth voice, legal advocacy, family stabilization, and institutional accountability. Even if an intervention has real benefits, it’ll tend to lose out if those benefits are diffuse, slow, or political rather than behavioral.
Then comes the deeper asymmetry, and this is where the field’s claim to rigor begins to wobble.
Look who must prove themselves: programs for youth development, mentoring, violence interruption, reentry, and diversion. Each must survive selection bias, attrition, spillover effects, measurement validity, model validity, site variation, and cost-benefit review. Honest reformers should say: good. Most of these programs should be evaluated more carefully than they are.
But detention, curfews, aggressive probation enforcement, warrior policing, gang databases, expulsions, pretrial monitoring, and GPS supervision seem somehow immune from the same evidentiary demands. They are treated as background conditions rather than interventions. The status quo is treated as the control group.
But the status quo is itself an intervention: an enormous coercive human-subjects experiment. Its core claims about deterrence, rehabilitation, public safety, and marginal incapacitation are far weaker and more context-dependent than its political authority suggests. Yet it is rarely asked to bear the burden of proof.
Punishment is evidence-based until definitively disproven. Support is suspect until it survives multiple rigorous and expensive evaluations.
That is the double standard. Cartwright’s ceteris paribus concerns make it worse: if even successful RCTs do not travel cleanly, then demanding portable proof from small humane programs without demanding comparable proof from the punitive default amounts to a political choice about whose uncertainty counts.
A better standard would start from a simpler principle of proportionality: the burden of evidence should scale with coercion. The least coercive supports should be expected to show plausible theory, community demand, implementation evidence, participant experience, and reasonable outcome tracking. RCTs may be useful here. They should not be mandatory. More intrusive, system-embedded interventions should face stronger evaluation, since they affect liberty, supervision, court outcomes, and access to services. Highly coercive state practices should face the strongest scrutiny of all.
Detention, incarceration, curfews, surveillance, exclusion, aggressive policing, and punitive supervision should be required to prove that they produce some measurable safety benefit and that they outperform less coercive alternatives once their full human, fiscal, and democratic costs are counted. A modest community program that gives young people somewhere to go after school does not need the same evidentiary burden as a policy that locks them in cells. Proportionality requires the difference.
Obviously, criminal justice should be evidence-based. But let’s make it fair. Let’s make everyone involved prove their worth, not just the bleeding hearts. Don’t just make the child and their family prove themselves. Make the cage and the curfew prove themselves too.
Nancy Cartwright taught me that evidence never travels without a passport. Jennifer Doleac reminds me that reformers need skepticism too. The task is to hold both lessons together. We need causal inference, and we need causal humility. We need to ask what works, but also where, for whom, under what conditions, compared to what, measured how, over what time horizon, and at what cost.
Skepticism can discipline power, or it can launder power. If it runs only in one direction, it becomes another way to fund doubt about help while leaving punishment untouched. That is the status quo with better footnotes.