Introductory description

This is a research-led module and will give students an insight into how economists evaluate policy implementation. The module begins with a review of the evaluation problem and discusses methods that are used to evaluate the outcomes of various interventions. The substantive portion of the module will take a series of recent research papers which are evaluating a policy that has been implemented and explain how the authors used appropriate statistical techniques in order evaluate the policy.

Module aims

Outline syllabus

This is an indicative module outline only to give an indication of the sort of topics that may be covered. Actual sessions held may differ.

Week 1 (2 hours Lecture)
Topic: Introduction to the evaluation problem.

Objective: This session will discuss the need for monitoring and evaluation, key concepts of monitoring and evaluation (causality, identification problems, selection on observable and unobservable characteristics), various elements that comprise a sound impact evaluation, data collection strategies, difficulties and limitations.

Week 2-10 (18 hours total)
Topic: Identifying the impact of an intervention

Objective: Within these eight weeks the lectures will take recent policy evaluation research papers and discuss the approach the papers took in order to evaluate the impact of the policy intervention and discuss the relative merits of alternative approaches. The approaches used for policy evaluation will include instrumental variables, differences-in-differences, matching methods, regression discontinuity design, synthetic control methods and randomized control trials, endogenous switching regression models, heterogeneous treatment effect models, Quantile Treatment Effect and event studies.
Students will also be taught to how to implement some of these procedures within appropriate statistical packages, such as STATA.

Learning outcomes

By the end of the module, students should be able to:

• Understand published empirical papers focused on policy evaluation.
• Demonstrate the ability to appreciate and implement empirical research using suitable techniques.
• Have developed the skills needed to understand, conduct and present empirical research related to policy evaluation.

Indicative reading list

The papers/books listed below highlight the gap between econometric theory and econometric application and the importance of using econometric tools in a “thoughtful and sensible” manner. The papers provide useful tips that you may wish to follow while undertaking a quantitative project.

Juying Zeng, Cristina Blanco-González-Tejero, F. Javier Sendra (2023): The spatial difference-in-difference measurement of policy effect of environmental protection interview on green innovation, Technological Forecasting and Social Change, Volume 191, 2023,

Myrto Kalouptsidi, Paul T. Scott, Eduardo Souza-Rodrigues (2021): Linear IV regression estimators for structural dynamic discrete choice models, Journal of Econometrics, Volume 222, Issue 1, Part C, 2021, Pages 778-804,

Wang, J. (2021). To use or not to use propensity score matching? Pharmaceutical Statistics, 20(1), 15-24.

James A. Reiffel (2020): Propensity Score Matching: The ‘Devil is in the Details’ Where More May Be Hidden than You Know, The American Journal of Medicine, Volume 133, Issue 2, 2020, Pages 178-181.

Salman, M., Long, X., Wang, G., & Zha, D. (2022). Paris climate agreement and global environmental efficiency: new evidence from fuzzy regression discontinuity design. Energy Policy, 168, 113128.

Stommes, D., Aronow, P. M., & Sävje, F. (2023). On the reliability of published findings using the regression discontinuity design in political science. Research & Politics, 10(2), 20531680231166457.

Uhr, D. D. A. P., de Oliveira Pinheiro, M. K., Francisco, D. J., de Oliveira, M. F. B., & Uhr, J. G. Z. (2023). The effect of 1987 Ireland’s Programme for National Recovery on sustainable economic growth: A synthetic control approach. Economics Letters, 222, 110929.

Addai, K. N., Ng’ombe, J. N., & Lu, W. (2023). Disaggregated impacts of off-farm work participation on household vulnerability to food poverty in Ghana. The Journal of Economic Inequality, 21(1), 83-104.

Angrist, J.D. and J. Pischke (2009), Mostly Harmless Econometrics: An Empiricist’s Companion, Princeton University Press.

Brigham R. Frandsen, Markus Frölich, Blaise Melly, (2012) “Quantile treatment effects in the regression discontinuity design”, Journal of Econometrics, Volume 168, Issue 2, 2012, Pages 382-395.

Casey, K., R. Glennerster and E. Miguel (2012), Reshaping Institutions: Evidence on Aid Impacts Using A Pre-analysis Plan, Quarterly Journal of Economics 127, 1755- 1812.

Cunningham, S. (2021), Causal Inference: The Mixtape, Yale University Press.

Duflo, Rachel Glennerster and Michael Kremer December 12, 2006. “Using Randomization in Development Economics Research: A Toolkit“ , https://www.povertyactionlab.org/sites/default/files/research-paper/Using-Randomization-in-Development-Economics.pdf

Gertler, P., S. Martinez, P. Premand, L.B. Rawlings and C.M.J. Vermeersch (2016), Impact Evaluation in Practice, Inter-American Development Bank and World Bank.

Glewwe, P. and Todd, P (2022) Quantile Treatment Effects;

Hamermesh, D.S. (2000), “The Craft of Labormetrics”, Industrial and Labor Relations Review 53, 363-80.

Hasebe, T. (2020). “Endogenous switching regression model and treatment effects of count-data outcome”. The Stata Journal, 20(3), 627-646.

Frolich, M. and Sperlich, S (2019) Impact Evaluation: Treatment effect and causal analysis, Cambridge University Press.

Kennedy, P.E. (2002), “Sinning in the Basement: What are the Rules? The Ten Commandments of Applied Econometrics”, Journal of Economic Surveys 16(4), 569-589.

Khandker S.R., G.B. Koolwal and H.A. Samad (2010), Handbook on Impact Evaluation: Quantitative Methods and Practices, World Bank.

Noltze, M., Schwarze, S. and Qaim, M., (2013). „Impacts of natural resource management technologies on agricultural yield and household income: The system of rice intensification in Timor Leste”. Ecological Economics, 85, pp.59-68.

David Powell, D. (2020) “Quantile Treatment Effects in the Presence of Covariates”. The Review of Economics and Statistics 2020; 102 (5): 994–1005.

Ravallion, M. (2001), “The Mystery of the Vanishing Benefits: An Introduction to Impact Evaluation”. The World Bank Economic Review 15(1), 115-140.

Vella, Francis, and Marno Verbeek. “Estimating and Interpreting Models with Endogenous Treatment Effects.” Journal of Business & Economic Statistics 17, no. 4 (1999): 473–78.

Vivalt, Eva. 2015. "Heterogeneous Treatment Effects in Impact Evaluation." American Economic Review, 105 (5).
Xie, Y., Brand, J. E., & Jann, B. (2012). “Estimating Heterogeneous Treatment Effects with Observational Data”. Sociological Methodology, 42(1), 314–347.

Stefan Wager, S. & Athey, S. (2018) “Estimation and Inference of Heterogeneous Treatment Effects using Random Forests”, Journal of the American Statistical Association, 113:523, 1228-1242

Wooldridge, J.M. (2019), Introductory Econometrics: A Modern Approach, South-Western, Cengage Learning, Chapter 13.

Research element

Apply critical econometric analysis to a policy, formulate concepts and hypotheses, and show how they are tested in the relevant policy context.

Interdisciplinary

The specific econometric skills concerning policy evaluation can also be extended beyond the economics discipline. Particularly, this module heavily interacts with policy evaluation's political and sociological aspects.

International

As a policy evaluation module, this module will introduce a whole host of public and social policies and their impact from different parts of the world, more particularly from the global South.

Subject specific skills

By the end of the course students will be equipped with the necessary methodological skills required to analyse any policy evaluation.

Transferable skills

Understand published empirical papers. Gain the ability to conduct empirical research using (modern) econometric techniques. Gain skills
i) in the use of computer software, including statistical software;
(ii) describe data and present it in a meaningful manner.
(iii) conduct individual research and investigate topics under their own initiative;
(iv) present their research to an audience;
(v) present their research conclusions in a written form.