External validity in program evaluation

LIEPP and the University of Maryland School of Public Policy are pleased to convene the seminar:

External validity in program evaluation

Tuesday June 6th 2023

Location: Sciences Po, Amphi Erignac, 13 rue de l’Université, 75007 Paris 
(and possibility to follow the event online)

Seminar organised by Doug Besharov (University of Maryland) and Anne Revillard (LIEPP, Sciences Po)

While program evaluation has been characterized by increased methodological sophistication to ensure the robustness of conclusions developed on the specific cases or sites under study, less attention has been paid to external validity, i.e., the extent to which these conclusions may apply to other contexts. This seminar brings together scholars and practitioners conducting evaluative research in a diversity of disciplines, to open a discussion on external validity from different methodological perspectives. 

Access videos of the seminar : 

External validity in program evaluation Session 1 [Video]

External validity in program evaluation Session 2 [Video]

External validity in program evaluation Session 3 [Video]

Programme: 

9:30-10:45: Session 1

• Introductions, by Anne Revillard (LIEPP, Sciences Po), Douglas J. Besharov (University of Maryland), Martin Weber (Director of Audit Chamber II, Investments for Cohesion, Growth and Inclusion of the European Court of Auditors), and Bastiaan de Laat (Head, OECD In-depth Evaluation).
• “Learning about External Validity from Multiple Literatures” (Jeff Smith, University of Wisconsin)

Discussant : Valérie Pattyn (Leiden University)  

10:45-11:00: Coffee break

11:00-12:30: Session 2

• “Differing Program Environments and Limited Replications Constrain Generalization: The Case of Published Evaluations of School Finance Policies” (Danielle Handel and Eric Hanushek, Stanford University)

 Discussant: Priscilla Alamos Concha (Radboud University)

• “Improving the Usefulness and Use of Systematic Reviews and Meta-Analyses to Inform Policy and Practice” (Rebecca Maynard, University of Pennsylvania)

Discussant : Ana Manzano (University of Leeds)

12:30-14:00: Lunch break

14:00-16:30: Session 3

• “The Logic of Generalization from Systematic Reviews to Policy and Practice” (Julia Littell, Bryn Mawr College)

 Discussant :  Thomas Delahais (Quadrant Conseil)

• “How Mixed-Methods Research Can Improve the Policy Relevance of Impact Evaluations”  (Burt Barnow and Sanjay Pandey, George Washington University)

 Discussants : Quan-Nha Hong (University of Montreal) and Sergi Fàbregues (Catalunya Open University)

• "Does Evidence-Based Policymaking Work for Local Policymakers When Rigorous Impact Evidence Comes from Other Localities?" (Larry Orr, Johns Hopkins Bloomberg School of Public Health)

Discussant: David Seidenfeld (American Institutes for Research, International Development Division)

• Concluding remarks, by Tom Ling (European Evaluation Society)

16:30-16:45: Coffee break

16:45-17:45: Internal work session with authors

Chairs: Doug Besharov (University of Maryland) and Anne Revillard (Sciences Po)

Invited participants: Irene Akua Agyepong (Ghana Health Service), Priscilla Alamos Concha (Radboud University), Burt Barnow (George Washington University), Douglas M. Call (University of Maryland), Bart Cambré (Antwerp Management School), Thomas Delahais (Quadrant Conseil), Sergi Fàbregues (Catalunya Open University), Bui Thi Thu Ha (Hanoi University of Public health), Danielle Handel (Stanford University), Eric Hanushek (Stanford University), Quan-Nha Hong (University of Montreal), Sumit Kane (University of Melbourne), Julia Littell (Bryn Mawr College), Ana Manzano (University of Leeds), Rebecca Maynard (University of Pennsylvania), Tolib Mirzoev (London School of Hygiene and Tropical Medicine), Sanjay Pandey (George Washington University), Valérie Pattyn (Leiden University), Valéry Ridde (Université Paris Cité, IRD), Benoît Rihoux (Université catholique de Louvain), Jeff Smith (University of Wisconsin), Geoff Wong (Oxford University)