QUANTITATIVE METHODS FOR POLICY EVALUATION
Instructional goals
The course introduces students to the quantitative evaluation of policies in the public or private sector. The theoretical treatment of the course contents is accompanied by applications to real cases for the evaluation of policy interventions. Exercises, empirical exercises, case studies concerning the quantitative evaluation of public policies based on real data, also using statistical and econometric packages, are proposed.
Practical lessons dealing with real-world examples with the use of statistical packages are designed to allow students to improve abilities in collecting, analysing, interpreting and presenting findings and data (EXCEL advanced, R).
The EU digital competences DIGCOMP 2.2 are developed (Competence area 1: information and data literacy; Competence area 2: communication and collaboration; Competence area 3: digital content creation).
Intended learning outcomes
Knowledge and understanding: knowledge of the main quantitative methods for causality analysis and policy evaluation: Regression with Instrumental Variables, Statistical Matching, Difference in Difference, Discontinuous Regression.
Applied knowledge and understanding: ability to analyze a policy, to identify treatment variable and outcome variable, and to apply appropriate quantitative methods of evaluation in microeconomic contexts in the public or private sector (labor, health, business strategies) and to interpret the results obtained, also using advanced spreadsheet and statistical software.
Making judgments: ability to critically interpret data and statistical models used in evaluative analysis of policies in the public or private sector
Communication skills: ability to effectively communicate evaluative analyses and present results.
Learning skills: ability to independently learn policy evaluation methods to manage a completion of knowledge in professional activities or further study.
Course Contents
Review of probability and random variables. Review of statistical inference. The simple regression model. The multiple regression model. Instrumental variables estimation and two stage least squares.
Causal inference for the quantitative evaluation of public policies. Statistical matching. Difference in differences. Regression discontinuity.
Reference Books
James H. Stock and Mark W. Watson, "Introduction to Econometrics", updated fourth
edition, Pearson (This is the main textbook).
Jeffrey Wooldridge, "Introductory Econometrics", fifth edition, South Western
(selected Sections of this textbook are available on the course website)
Seeing Theory: https://seeing theory.brown.edu
Teacher's Notes
Suggested readings:
Joshua D. Angrist and Jörn-Steffen Pischke, (2009) "Mostly Harmless
Econometrics: An Empiricist's Companion", Princeton University Press.
Dehejia R.H. and S. Wahba (1999), Causal Effects in Nonexperimental Studies:
Reevaluating the Evaluation of Training Programs, JASA.
(Statistical Matching; available on the course website).
Card D. e A,B, Krueger (1994), Minimum Wages and Employment: A Case Study of
the Fast-Food Industry in New Jersey and Pennsylvania, AER.
(Difference in Differences; available on the course website).
LaLonde, R.J. (1986), Evaluating the Econometric Evaluations of Training
Programs, American Economic Review, 76, 604-620.
(Difference in Differences; available on the course website).
Angrist J.D. and V. Lavy (1999), Using Maimonides' Rule to Estimate the Effect of
Class Size on Scholastic Achievement, QJE.
(Regression Discontinuity; available on the course website).
Mancini M. and C. Pappalardo (2006), Evaluating the Impact of Labour Market
Regulation on the Size Growth of Italian Firms, Politica Economica, Vil. XXii, No. 3.
(Regression Discontinuity; available on the course website).
"EU Expenditure Programmes. A Guide. Ex Post and Intermediate Evaluation";
European Commission, (1997) (available on the course website).
"Ex Post Evaluation of Cohesion Policy Programmes 2000-2006 Financed by the
European Regional Development Fund", (GEFRA-IAB Final Report 2010) (available
on the course website).
"Counterfactual Impact Evaluation of Cohesion Policy. Impact and Cost
Effectiveness of Investment Subsidies in Italy", Final Report to DG Regional
Policy (2012) (ASVAPP) (available on the course website).
Unless otherwise stated, the headings of the Chapters and Sections refer to the textbooks.
Teaching Methods
Lectures
Exercises
Empirical exercises
Interactive visualization
Flipped Classroom
Project Work with use of advanced spreadsheet and econometric software
Case analysis
Case studies in the private or public sector.
Assessment Method
1. Written Exam. The final examination is the in the form of a written exam, consisting of both theoretical and empirical questions; during the final exam, it is not allowed to consult books or class notes. It verifies the acquisition of Knowledge and understanding, Applying knowledge and understanding, Making judgements.
A midterm written assessment will be held and a final written assignment in the first session of exam.
The Students attending the course are requested to solve 4 problem sets, two before the midterm written assessment and two before the final written assignment. The submission of the 4 problem sets is necessary to take the written assessments. The final grade is computed as the sum of the grades obtained on the 6 assessments. If the student is not satisfied with the assigned grade, she/he may not accept the grade and take the full exam.
2. Problem set 4 consists of replicating a public policy analysis presented in a scientific paper. The test is conducted in groups of three students using the advanced spreadsheet/R. It verifies the acquisition of Making judgments, Communication Skills, Learning Skills. Digital skills (1, 2, 3), teamwork, time management, development of new strategies and solutions to solve problems.
WRITTEN EXAMINATION: this type of examination ("scritto verbalizzante") consists in a written exam without a subsequent oral examination. The student must book for the written test. At the end of the final examination, the teacher corrects the homeworks and publishes the results on the dedicated VOL web page (within one week from the end of the exam date).
The students enrolled in the final exam will receive a communication with the grade
earned on the written examination (the grade earned in the written examination will also be displayed on the web self service).
Since the publication of the results, each student has 3 days to reject the assigned grade. Once the 3-day period is elapsed, the rule of "tacit consent" ("silenzio assenso") applies, and the assigned grade is verbalized by the teacher. The teacher must close down the verbal through the digital signature.
Once the verbal is closed down, the grade earned is released to the student through
an e-mail communication.
The text of the written exam and the corresponding solution are made available on the course website before the publication of the grades. Each candidate, regardless of the final outcome of the examination, can access the solution of the written exam on a date set by the teacher, so that the student will be on time and able to not accept the assigned grade.
Thesis assignment criteria
The final paper is a research paper in which statistical methods are applied to the evaluation of public policies.
Week 1
Lecture 1
Introduction. Counterfactual analysis of treatment effects.
Evaluating EU Expenditure Programmes: A Guide (European Commission)
Chapter 1. Economic Questions and Data
1.1 Economic Questions We Examine
1.2 Causal Effects and Idealized Experiments
1.3 Data: Sources and Types
Chapter 2 Review of Probability
2.1 Random variables and Probability Distributions Probability, the Random Space
and Random Variables
2.2 The Expected Values, Mean and Variance
Fundamentals of Probability (Appendix B: B1, B3. Introductory Econometrics - Wooldridge)
Lecture 1
Exercises, empirical exercises, case studies on the quantitative evaluation of public
policies based on real data, also using statistical and econometric packages.
Week 2
Lecture 2
Chapter 2 Review of Probability
2.3 Two Random Variables
(Joint and Marginal Distributions, Conditional Distributions, Independence,
Covariance and Correlation, The Mean and Variance of Sums of Random
Variables)
2.4 The Normal, Chi-squared, Student t, and F Distributions
2.5 Random Sampling and the Distribution of the Random Average Random
Sampling
2.6 Large-Sample Approximations to Sampling Distributions
Fundamentals of Probability (Appendix B: B2, B4, B5. Introductory Econometrics -
Wooldridge)
Teacher's Notes
Lecture 2
Exercises, empirical exercises, case studies on the quantitative evaluation of public
policies based on real data, also using statistical and econometric packages.
Week 3
Lecture 3
Chapter 3 Review of Statistics
3.1 Estimation of the Population Mean
3.2 Hypothesis Tests Concerning the Population Mean
3.3 Confidence Intervals for the Population Mean
3.4 Comparing Means from Different Populations
3.5 Differences of Means Estimation of Causal Effects Using Experimental Data
3.6 Using the t-Statistic when the Sample Size Is Small
3.7 Scatterplots, the Sample Covariance and the Sample Correlation
The California Standardized Testing and Reporting (STAR) dataset (caschool).
Teacher's Notes.
Lecture 3
Exercises, empirical exercises, case studies on the quantitative evaluation of public
policies based on real data, also using statistical and econometric packages.
Week 4
Lecture 4
Chapter 4 Linear Regression with One Regressor
4.1 The Linear Regression Model
4.2 Estimating the Coefficients of the Linear Regression Model
4.3 Measures of fit
4.4 The Least Square Assumptions
4.5 Sampling Distribution of the OLS Estimator
The simple regression model (Chapter 2 Introductory Econometrics - Wooldridge).
2.1 Definition of the Simple Regression Model
2.2 Deriving the Ordinary Least Square Estimates.
The California Standardized Testing and Reporting (STAR) dataset (caschool).
Lecture 4
Exercises, empirical exercises, case studies on the quantitative evaluation of public
policies based on real data, also using statistical and econometric packages.
Week 5
Lecture 5
Chapter 5 Regression with a Simple Regressor: Hypothesis Tests and Confidence
Intervals
5.1 Testing Hypothesis About One of the Regression Coefficients
5.2 Confidence Interval for a Regression Coefficient
5.3 Regression when X Is a Binary Variable
5.4 Heteroskedasticity and Homoskedasticity
Chapter 6 Linear Regression with Multiple Regressors
6.1 Omitted Variables Bias
6.2 The Multiple Regression Model
6.3 The OLS Estimator in Multiple Regression
Omitted Variable Bias: The Simple Case (Section 3.3 Introductory Econometrics -
Wooldridge).
The California Standardized Testing and Reporting (STAR) dataset (caschool)
Lecture 5
Exercises, empirical exercises, case studies on the quantitative evaluation of public
policies based on real data, also using statistical and econometric packages.
Week 6
Lecture 6
Chapter 6 Linear Regression with Multiple Regressors
6.4 Measures of Fit in Multiple Regression
6.5 The Least Square Assumptions in Multiple Regression
6.6 The distribution of OLS Estimators in Multiple Regression
6.7 Multicollinearity
Chapter 7 Hypothesis Tests and Confidence Intervals in Multiple Regression
7.1 Hypothesis Tests and Confidence Intervals for a Single Coefficient
Standard Errors for the OLS Estimators
Hypothesis Tests for a Single Coefficient.
The California Standardized Testing and Reporting (STAR) dataset (caschool)
Lecture 6
Exercises, empirical exercises, case studies on the quantitative evaluation of public
policies based on real data, also using statistical and econometric packages.
Week 7
Lecture 7
Chapter 7 Hypothesis Tests and Confidence Intervals in Multiple Regression
7.2 Test of Joint Hypothesis
Testing Hypothesis on Two or More Coefficients
The F-Statistic
7.6 Analysis of Test Scores Data Set
Chapter 8 Nonlinear Regression Functions
8.3 Interactions Between Independent Variable. The California Standardized Testing and Reporting (STAR) dataset (caschool)
Lecture 7
Exercises, empirical exercises, case studies on the quantitative evaluation of public
policies based on real data, also using statistical and econometric packages.
Week 8
Lecture 8
Chapter 11 Logistic Regression
Teacher's Notes and published journal articles.
Lecture 8
Exercises, empirical exercises, case studies on the quantitative evaluation of public
policies based on real data, also using statistical and econometric packages.
Week 9
Lecture 9
Chapter 13 Experiments and Quasi-Experiments
13.1 Potential Outcomes, Causal Effects and Idealized Experiments
Potential Outcomes and Average Causal Effects
Econometric Methods for Analyzing Experimental Data
13.2 Threats to Validity of Experiments
13.3 Experimental Estimates of the Effect of Class Size Reduction
Statistical Matching
(Teacher's Notes and published journal articles). Case analysis: National Supported Work Demonstration
Lecture 9
Exercises, empirical exercises, case studies on the quantitative evaluation of public
policies based on real data, also using statistical and econometric packages.
Week 10
Lecture 10
Chapter 13 Experiments and Quasi-Experiments
13.4 Quasi-Experiments
The Difference in Differences Estimator
(supplemented with Teacher's Notes and published journal articles). Case analysis: National Supported Work Demonstration
Lecture 10
Exercises, empirical exercises, case studies on the quantitative evaluation of public
policies based on real data, also using statistical and econometric packages.
Week 11
Lecture 11
Chapter 13 Experiments and Quasi-Experiments
13.4 Quasi-Experiments
Regression Discontinuity Estimator
(supplemented with Teacher Notes and published journal articles). Case analysis. Raffaello Bronzini and Eleonora Iachini. 2014. Are Incentives for R&D Effective? Evidence from a Regression Discontinuity Approach. American Economic Journal: Economic Policy, 6(4): 100-134.
Case studies
Lecture 11
Exercises, empirical exercises, case studies on the quantitative evaluation of public
policies based on real data, also using statistical and econometric packages.
Week 12
Final Practice