Start with the question the evidence must answer
Suppose a city expands bus service and ridership rises the following year. The two events are correlated, but the increase may also reflect population growth, a change in fares or higher fuel costs. To ask whether the expansion caused the rise, we need to consider what ridership would have been without it. That unobserved alternative is called the counterfactual. Policy evaluation tries to estimate it rather than treating every change after a policy as an effect of the policy.
A useful first step is to state the proposed effect precisely. Which policy changed, for whom, in what place and over what period? Which outcome matters: total rides, the share of residents using transit or access to jobs? These measures can move differently. It also helps to describe how the policy is expected to work. More frequent buses might reduce waiting times, for example, but that mechanism depends on whether the added service reaches people who could use it.
Find a credible comparison
A simple before-and-after comparison is easy to understand but often weak evidence of causation. Conditions can change while a policy is being introduced. Comparing people or places that received the policy with those that did not may help, but the groups might have differed beforehand. A city that expands transit because demand is already growing is not automatically comparable to one that keeps its service unchanged.
A stronger comparison asks whether the groups were likely to have followed similar paths without the policy. Evaluators may examine their earlier trends and relevant characteristics, while remaining alert to differences they cannot observe. The comparison should also fit the question. If a service is aimed at neighborhoods with the least access, comparing their outcomes directly with those of well-served neighborhoods could mistake pre-existing inequality for a policy effect.
Understand what different methods can establish
When a policy or program can be assigned by chance, a randomized evaluation can make the groups similar on average before it begins. That makes differences in later outcomes easier to attribute to the intervention. Random assignment is not always feasible or appropriate in public policy, however, and the results still depend on what was implemented, who took part and whether outcomes were measured reliably.
Other approaches use the way a policy was introduced. A difference-in-differences study compares changes over time in a policy group with changes in a comparison group; its interpretation depends heavily on whether the groups would otherwise have had similar trends. A study of an eligibility cutoff may compare people just above and below it, where they may be relatively alike. Such a result can be informative near the cutoff without necessarily describing the policy’s effect on everyone. No method removes the need to examine its assumptions.
Look for rival explanations and measurement problems
Ask what else happened around the same time. A new fare rule, a major employer’s move or a change in how rides are counted could affect measured ridership. People may also respond before a policy formally starts if they know it is coming. And a policy in one area may affect a comparison area: new routes could draw riders away from nearby services. Each possibility changes how confidently a difference can be attributed to the policy.
Outcome measures deserve equal scrutiny. A rise in total rides might reflect repeated trips by existing riders rather than access for new riders. An average improvement could hide little change for people with limited mobility or those living far from stops. Conversely, a narrowly measured outcome may miss benefits or costs elsewhere. Evaluators should explain which outcomes were chosen, how consistently they were recorded and whose experiences those measures represent.
Draw a conclusion that matches the evidence
A careful conclusion separates the observed pattern from its causal interpretation. “Ridership rose after the expansion” describes a sequence of events. “The expansion increased ridership” requires a defensible comparison and an account of plausible alternative explanations. Even then, an estimate is usually more useful when paired with uncertainty and a clear description of the people, places and period it covers.
When reading an evaluation, ask five questions: What outcome was measured? What would likely have happened without the policy? Why is the comparison credible? What other changes might explain the result? To whom might the finding apply? These questions do not guarantee a definitive answer. They make it easier to distinguish evidence of an effect from a coincidence, and to identify what remains unknown before drawing broader lessons.
