QUANTITATIVE METHODS FOR POLICY EVALUATION

QUANTITATIVE METHODS FOR POLICY EVALUATION

Antonio Pacifico, Livia De Giovanni

Instructional goals

The course introduces students to the quantitative evaluation of policies in the public or private sector. The theoretical treatment of the course contents is accompanied by applications to real cases for the evaluation of policy interventions. Exercises, empirical exercises, case studies concerning the quantitative evaluation of public policies based on real data, also using statistical and econometric packages, are proposed. Practical lessons dealing with real-world examples with the use of statistical packages are designed to allow students to improve abilities in collecting, analysing, interpreting and presenting findings and data (EXCEL advanced, R). The EU digital competences DIGCOMP 2.2 are developed (Competence area 1: information and data literacy; Competence area 2: communication and collaboration; Competence area 3: digital content creation).

Intended learning outcomes

Knowledge and understanding: knowledge of the main quantitative methods for causality analysis and policy evaluation: Regression with Instrumental Variables, Statistical Matching, Difference in Difference, Discontinuous Regression. Applied knowledge and understanding: ability to analyze a policy, to identify treatment variable and outcome variable, and to apply appropriate quantitative methods of evaluation in microeconomic contexts in the public or private sector (labor, health, business strategies) and to interpret the results obtained, also using advanced spreadsheet and statistical software. Making judgments: ability to critically interpret data and statistical models used in evaluative analysis of policies in the public or private sector Communication skills: ability to effectively communicate evaluative analyses and present results. Learning skills: ability to independently learn policy evaluation methods to manage a completion of knowledge in professional activities or further study.

Course Contents

Review of probability and random variables. Review of statistical inference. The simple regression model. The multiple regression model. Instrumental variables estimation and two stage least squares. Causal inference for the quantitative evaluation of public policies. Statistical matching. Difference in differences. Regression discontinuity.

Reference Books

James H. Stock and Mark W. Watson, "Introduction to Econometrics", updated fourth edition, Pearson (This is the main textbook). Jeffrey Wooldridge, "Introductory Econometrics", fifth edition, South Western (selected Sections of this textbook are available on the course website) Seeing Theory: https://seeing theory.brown.edu Teacher's Notes Suggested readings: Joshua D. Angrist and Jörn-Steffen Pischke, (2009) "Mostly Harmless Econometrics: An Empiricist's Companion", Princeton University Press. Dehejia R.H. and S. Wahba (1999), Causal Effects in Nonexperimental Studies: Reevaluating the Evaluation of Training Programs, JASA. (Statistical Matching; available on the course website). Card D. e A,B, Krueger (1994), Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania, AER. (Difference in Differences; available on the course website). LaLonde, R.J. (1986), Evaluating the Econometric Evaluations of Training Programs, American Economic Review, 76, 604-620. (Difference in Differences; available on the course website). Angrist J.D. and V. Lavy (1999), Using Maimonides' Rule to Estimate the Effect of Class Size on Scholastic Achievement, QJE. (Regression Discontinuity; available on the course website). Mancini M. and C. Pappalardo (2006), Evaluating the Impact of Labour Market Regulation on the Size Growth of Italian Firms, Politica Economica, Vil. XXii, No. 3. (Regression Discontinuity; available on the course website). "EU Expenditure Programmes. A Guide. Ex Post and Intermediate Evaluation"; European Commission, (1997) (available on the course website). "Ex Post Evaluation of Cohesion Policy Programmes 2000-2006 Financed by the European Regional Development Fund", (GEFRA-IAB Final Report 2010) (available on the course website). "Counterfactual Impact Evaluation of Cohesion Policy. Impact and Cost Effectiveness of Investment Subsidies in Italy", Final Report to DG Regional Policy (2012) (ASVAPP) (available on the course website). Unless otherwise stated, the headings of the Chapters and Sections refer to the textbooks.

Teaching Methods

Lectures Exercises Empirical exercises Interactive visualization Flipped Classroom Project Work with use of advanced spreadsheet and econometric software Case analysis Case studies in the private or public sector.

Assessment Method

1. Written Exam. The final examination is the in the form of a written exam, consisting of both theoretical and empirical questions; during the final exam, it is not allowed to consult books or class notes. It verifies the acquisition of Knowledge and understanding, Applying knowledge and understanding, Making judgements. A midterm written assessment will be held and a final written assignment in the first session of exam. The Students attending the course are requested to solve 4 problem sets, two before the midterm written assessment and two before the final written assignment. The submission of the 4 problem sets is necessary to take the written assessments. The final grade is computed as the sum of the grades obtained on the 6 assessments. If the student is not satisfied with the assigned grade, she/he may not accept the grade and take the full exam. 2. Problem set 4 consists of replicating a public policy analysis presented in a scientific paper. The test is conducted in groups of three students using the advanced spreadsheet/R. It verifies the acquisition of Making judgments, Communication Skills, Learning Skills. Digital skills (1, 2, 3), teamwork, time management, development of new strategies and solutions to solve problems. WRITTEN EXAMINATION: this type of examination ("scritto verbalizzante") consists in a written exam without a subsequent oral examination. The student must book for the written test. At the end of the final examination, the teacher corrects the homeworks and publishes the results on the dedicated VOL web page (within one week from the end of the exam date). The students enrolled in the final exam will receive a communication with the grade earned on the written examination (the grade earned in the written examination will also be displayed on the web self service). Since the publication of the results, each student has 3 days to reject the assigned grade. Once the 3-day period is elapsed, the rule of "tacit consent" ("silenzio assenso") applies, and the assigned grade is verbalized by the teacher. The teacher must close down the verbal through the digital signature. Once the verbal is closed down, the grade earned is released to the student through an e-mail communication. The text of the written exam and the corresponding solution are made available on the course website before the publication of the grades. Each candidate, regardless of the final outcome of the examination, can access the solution of the written exam on a date set by the teacher, so that the student will be on time and able to not accept the assigned grade.

Thesis assignment criteria

The final paper is a research paper in which statistical methods are applied to the evaluation of public policies.

Week 1

Lecture 1 Introduction. Counterfactual analysis of treatment effects. Evaluating EU Expenditure Programmes: A Guide (European Commission) Chapter 1. Economic Questions and Data 1.1 Economic Questions We Examine 1.2 Causal Effects and Idealized Experiments 1.3 Data: Sources and Types Chapter 2 Review of Probability 2.1 Random variables and Probability Distributions Probability, the Random Space and Random Variables 2.2 The Expected Values, Mean and Variance Fundamentals of Probability (Appendix B: B1, B3. Introductory Econometrics - Wooldridge) Lecture 1 Exercises, empirical exercises, case studies on the quantitative evaluation of public policies based on real data, also using statistical and econometric packages.

Week 2

Lecture 2 Chapter 2 Review of Probability 2.3 Two Random Variables (Joint and Marginal Distributions, Conditional Distributions, Independence, Covariance and Correlation, The Mean and Variance of Sums of Random Variables) 2.4 The Normal, Chi-squared, Student t, and F Distributions 2.5 Random Sampling and the Distribution of the Random Average Random Sampling 2.6 Large-Sample Approximations to Sampling Distributions Fundamentals of Probability (Appendix B: B2, B4, B5. Introductory Econometrics - Wooldridge) Teacher's Notes Lecture 2 Exercises, empirical exercises, case studies on the quantitative evaluation of public policies based on real data, also using statistical and econometric packages.

Week 3

Lecture 3 Chapter 3 Review of Statistics 3.1 Estimation of the Population Mean 3.2 Hypothesis Tests Concerning the Population Mean 3.3 Confidence Intervals for the Population Mean 3.4 Comparing Means from Different Populations 3.5 Differences of Means Estimation of Causal Effects Using Experimental Data 3.6 Using the t-Statistic when the Sample Size Is Small 3.7 Scatterplots, the Sample Covariance and the Sample Correlation The California Standardized Testing and Reporting (STAR) dataset (caschool). Teacher's Notes. Lecture 3 Exercises, empirical exercises, case studies on the quantitative evaluation of public policies based on real data, also using statistical and econometric packages.

Week 4

Lecture 4 Chapter 4 Linear Regression with One Regressor 4.1 The Linear Regression Model 4.2 Estimating the Coefficients of the Linear Regression Model 4.3 Measures of fit 4.4 The Least Square Assumptions 4.5 Sampling Distribution of the OLS Estimator The simple regression model (Chapter 2 Introductory Econometrics - Wooldridge). 2.1 Definition of the Simple Regression Model 2.2 Deriving the Ordinary Least Square Estimates. The California Standardized Testing and Reporting (STAR) dataset (caschool). Lecture 4 Exercises, empirical exercises, case studies on the quantitative evaluation of public policies based on real data, also using statistical and econometric packages.

Week 5

Lecture 5 Chapter 5 Regression with a Simple Regressor: Hypothesis Tests and Confidence Intervals 5.1 Testing Hypothesis About One of the Regression Coefficients 5.2 Confidence Interval for a Regression Coefficient 5.3 Regression when X Is a Binary Variable 5.4 Heteroskedasticity and Homoskedasticity Chapter 6 Linear Regression with Multiple Regressors 6.1 Omitted Variables Bias 6.2 The Multiple Regression Model 6.3 The OLS Estimator in Multiple Regression Omitted Variable Bias: The Simple Case (Section 3.3 Introductory Econometrics - Wooldridge). The California Standardized Testing and Reporting (STAR) dataset (caschool) Lecture 5 Exercises, empirical exercises, case studies on the quantitative evaluation of public policies based on real data, also using statistical and econometric packages.

Week 6

Lecture 6 Chapter 6 Linear Regression with Multiple Regressors 6.4 Measures of Fit in Multiple Regression 6.5 The Least Square Assumptions in Multiple Regression 6.6 The distribution of OLS Estimators in Multiple Regression 6.7 Multicollinearity Chapter 7 Hypothesis Tests and Confidence Intervals in Multiple Regression 7.1 Hypothesis Tests and Confidence Intervals for a Single Coefficient Standard Errors for the OLS Estimators Hypothesis Tests for a Single Coefficient. The California Standardized Testing and Reporting (STAR) dataset (caschool) Lecture 6 Exercises, empirical exercises, case studies on the quantitative evaluation of public policies based on real data, also using statistical and econometric packages.

Week 7

Lecture 7 Chapter 7 Hypothesis Tests and Confidence Intervals in Multiple Regression 7.2 Test of Joint Hypothesis Testing Hypothesis on Two or More Coefficients The F-Statistic 7.6 Analysis of Test Scores Data Set Chapter 8 Nonlinear Regression Functions 8.3 Interactions Between Independent Variable. The California Standardized Testing and Reporting (STAR) dataset (caschool) Lecture 7 Exercises, empirical exercises, case studies on the quantitative evaluation of public policies based on real data, also using statistical and econometric packages.

Week 8

Lecture 8 Chapter 11 Logistic Regression Teacher's Notes and published journal articles. Lecture 8 Exercises, empirical exercises, case studies on the quantitative evaluation of public policies based on real data, also using statistical and econometric packages.

Week 9

Lecture 9 Chapter 13 Experiments and Quasi-Experiments 13.1 Potential Outcomes, Causal Effects and Idealized Experiments Potential Outcomes and Average Causal Effects Econometric Methods for Analyzing Experimental Data 13.2 Threats to Validity of Experiments 13.3 Experimental Estimates of the Effect of Class Size Reduction Statistical Matching (Teacher's Notes and published journal articles). Case analysis: National Supported Work Demonstration Lecture 9 Exercises, empirical exercises, case studies on the quantitative evaluation of public policies based on real data, also using statistical and econometric packages.

Week 10

Lecture 10 Chapter 13 Experiments and Quasi-Experiments 13.4 Quasi-Experiments The Difference in Differences Estimator (supplemented with Teacher's Notes and published journal articles). Case analysis: National Supported Work Demonstration Lecture 10 Exercises, empirical exercises, case studies on the quantitative evaluation of public policies based on real data, also using statistical and econometric packages.

Week 11

Lecture 11 Chapter 13 Experiments and Quasi-Experiments 13.4 Quasi-Experiments Regression Discontinuity Estimator (supplemented with Teacher Notes and published journal articles). Case analysis. Raffaello Bronzini and Eleonora Iachini. 2014. Are Incentives for R&D Effective? Evidence from a Regression Discontinuity Approach. American Economic Journal: Economic Policy, 6(4): 100-134. Case studies Lecture 11 Exercises, empirical exercises, case studies on the quantitative evaluation of public policies based on real data, also using statistical and econometric packages.

Week 12

Final Practice