Should We Believe in Economic Impact Assessments?

How can we ensure, as rigorously as possible, that a public policy has the desired economic effects? Clément de Chaisemartin, a researcher in the Department of Economics, focuses on this fundamental question in his ‘Really Credible’ project⁽¹⁾Completing the revolution: Enhancing the reality, the principles, and the impact of economics’ credibility revolution, which has received support from the European Research Council highly selective ERC Grants programme. Interview.

A common way of conducting economic research is to use observed data to study the causal links between economic variables. This is known as empirical economics. What is the advantage of this approach?

Clément de Chaisemartin: Economics is a discipline that has for a long time primarily drawn on theoretical models. However, models based on different assumptions predict different, and even opposite cause-and-effect relationships between economic variables. For example, a neoclassical economist would argue that raising the minimum wage will lead companies to use robots rather than workers, thus reducing employment. A neo-Keynesian economist would posit the opposite: raising the minimum wage can increase employment by stimulating demand or by increasing worker motivation and hence productivity. Who’s right? It’s hard to know without data! Empirical economics allows us to determine which of these divergent mechanisms is accurate.

Despite its promise, this approach has been challenged, notably by the American economist Edward Leamer, who has made a name for himself by dismissing it as biased and therefore not very credible. Why this criticism?

Clément de Chaisemartin: Empirical economics has been the subject of all kinds of criticism, including that of Leamer⁽²⁾Edward L. Leamer,  Let’s Take the Con out of Econometrics, The American Economic Review, Vol. 73, No. 1 (Mar., 1983), pp. 31-43. Unpacking these critiques requires understanding the fundamental problem in attempting to measure the effect of one variable on another. In my example, the effect of an increase in the minimum wage on employment at a given time and in a given place is the difference between the actual level of employment and the counterfactual level of employment that would have been observed in the same place and at the same time had the minimum wage not increased. By definition, this counterfactual level of employment is not directly observed: we do not directly observe the level of employment that would have prevailed in a region that chose to increase its minimum wage if it had made the opposite choice. Thus, the empirical economist evaluating a public policy will have to reconstruct the counterfactual situation that would have been observed without the policy. And unlike an architect, who can give free rein to subjectivity when developing a building, it is preferable for an applied economist not to give free rein to creativity when developing a counterfactual. Otherwise, the assessed effect of the economic policy will depend on methodological choices, and another economist making different choices could come to a different conclusion. Study results are sometimes sensitive to subjective researchers’ choices. This has undermined the credibility of empirical economics.

In response to this criticism, economists have sought to strengthen their econometric models and develop new methods in experimental economics, enabling the creation of ‘reproducible’ models. Many claim this addresses the credibility issue, but you don’t think it’s enough. What do you think is needed to close the gap?

Clément de Chaisemartin: Economists have been formally tackling the identification of causal links between economic variables since the 1980s. A toolbox of standardised techniques based on often plausible and testable hypotheses gradually took shape, notably under the aegis of Nobel Prize winners Joshua Angrist and Guido Imbens, and came to be known as the ‘credibility revolution’. But there was still one technique that, until recently, escaped this critical scrutiny of the valuation methods used by economists: two-way linear fixed-effects regressions – the awful name alone is suspect.
In a paper with my co-author Xavier D’Haultfoeuille, we showed that this technique, frequently used in studies based on natural experiments to evaluate the effect of policies, is only reliable if the effect of the evaluated policy is temporally and spatially identical. This assumption is often implausible. For example, the effect of a minimum wage on employment is unlikely to be the same in advanced economies, where few jobs can easily be automated, as in less advanced economies. Of concern is that a quarter of the most cited articles recently published in the American Economic Review, one of the most prestigious journals in economics, uses two-way fixed-effects regressions. In other words, much of the recent research in applied economics is still based on fragile assumptions, and therefore cannot be considered credible. The ‘credibility revolution’ is far from over!
The goal of my ‘Really Credible’ project is to propose evaluation methods that remain valid even if the effect of the policy being evaluated varies over time and space, as is often the case. Next, I want to develop computer programmes that researchers can easily use to apply the methods I propose. Finally, I will use these programmes to revisit a large number of articles that have used two-way fixed-effects regressions and use my method instead. This will allow me to see whether the conclusions that researchers obtained with two-way fixed-effects regressions withstand the scrutiny of comparison with the results obtained using my method.
This is important because the two-way fixed-effects regressions have been used to answer pressing questions, such as the effect of rising Chinese exports on employment in the USA and Europe, or the effect of rising temperatures on agricultural yields and population health. It is crucial to see whether the answers to these questions change with more robust methods.

Is your ultimate goal to ensure that decision-makers and the general public have a better understanding of economic research methods, and therefore more confidence in them?

Clément de Chaisemartin : Joshua Angrist⁽³⁾Joshua Angrist is Professor of Economics at the Massachusetts Institute of Technology (MIT) and co-winner of the 2021 Nobel Prize in Economics with Guido Imbens and David Card. He specialises in labor economics and the economics of education. once told me that a good criterion for knowing whether my research was good was that I should be able to explain it to my grandparents. The credibility revolution embraces the idea that research should be accessible to all and transparent. So it’s possible that one of the benefits of this revolution I’m partaking in is to make economic research understandable and credible to as many people as possible, potentially enabling it to foster more consensus on the pros and cons of different policy options. The way impact assessments shape the beliefs of citizens and public decision-makers is an interesting empirical question that is understudied. For example, an article⁽⁴⁾Sultan Mehmood, Shaheen Naseer and Daniel L. Chen “Training Policymakers in Econometrics”, working paper. showed that training Indian public decision-makers in modern public policy evaluation methods affected their beliefs and practices. It would be interesting to conduct such an experiment with Sciences Po students or graduates! I’d also love to carry out an experiment of this type with audiences much further removed from institutions (anti-vaxxers, conspiracy theorists, etc.). It’s paradoxical that, at a time when the social sciences are making great strides in producing robust research on public policy impacts, our societies are becoming increasingly polarised and clashing – sometimes violently – over choosing between different public policy options. Better dissemination of research findings and education in evaluation methods might help build consensus.

Interview by Hélène Naudet (Communications Department) and Melissa Mundell (Economics Department)

University Professor Clément de Chaisemartin joined the Department of Economics in 2021 after conducting research and teaching at the University of California, Santa Barbara. An econometrician, he specialises in public policy evaluation methods. He has also taken part in impact assessments of public education policies in France (Internats d’excellence) and abroad. He is also a Research Affiliate of the Abdul Latif Jameel Poverty Action Lab (J-PAL), created by the Department of Economics at the Massachusetts Institute of Technology.