Regression discontinuity design

Strong internal validity

... But limited external validity

Missing data and no overlap

Can we find a generalization interval?

This paper

Identification of generalization interval and estimation of ATT for population within such interval:

• Pre-intervention period informs the generalization bandwidth
  (Wing & Cook, 2013; Keele, Small, Hsu, & Fogarty, 2020)

• Leverage the use of predictive covariates for breaking link between running variable and outcome
  (Angrist & Rokkanen, 2015; Rokkanen, 2015; Keele, Titiunik, & Zubizarreta, 2015)

• Based on idea of local randomization near the cutoff
  (Lee, 2008; Cattaneo, Frandsen, & Titiunik, 2015)

This paper

Main advantages:

• Gradual approach
  • No need for "All or Nothing"
  • Interval informed by the data (Cattaneo et al., 2015)
• No extrapolation of population characteristics
  • Compare like-to-like (Rosenbaum, 1987)
  • Makes overlap region explicit
• Generalization to population of interest
  • Use of representative template matching (Silber et al., 2014; Bennett, Vielma, & Zubizarreta, 2020)
• Sensitivity analysis to hidden bias (Rosenbaum, 2010; Keele et al., 2020)

Outline

1. Motivation
   
   

2. Generalized Regression Discontinuity Design (GRD)

   2.1 Framework

   2.2 GRD in practice
   
   

3. Application: Free Higher Education in Chile
   
   
4. Conclusions

Generalized Regression Discontinuity Design (GRD)

Two-part problem with pre- and post-intervention periods:

1) Identification of generalization interval H^*

(using pre-intervention period)
2) Estimation of ATT for population within H^*

(using post-intervention period)

The setup

• Two periods: pre- and post-intervention, $t=0$ and $t=1$.

• Running variable $R$ determines assingment $Z$ in $t=1$. E.g.: $$Z_{it}=I(R_{it}<c)$$

• Potential outcomes under treatmet $z=0,1$: $$Y_{it}^{\left(z\right)}=g_z(X_{it},u_{it},r_{it})+z_{it}\cdot \underset{\text{Treat. Effect}}{\underset{}{\tau (X_{it},u_{it},r_{it})}}+\underset{\text{Period FE}}{\underset{}{{\alpha }_t}}$$

  • $X_{it}$: Predictive covariates

  • $u_{it}$: Unobserved confounders

  • $\tau (\cdot )$: Causal effect

Two periods for GRD

A gradual approach

• Conditional expectations of potential outcomes, $Y_t^{\left(z\right)}\left(R\right)$: $$Y_0^{\left(0\right)}\left(R\right)=E\left[Y_{i0}^{\left(0\right)}|R\right]={\mu }_0\left(R\right)$$ $$Y_0^{\left(1\right)}\left(R\right)=E\left[Y_{i0}^{\left(1\right)}|R\right]=\underset{\text{Avg. Outcome by R}}{\underset{}{{\mu }_0\left(R\right)}}+\underset{\text{Treat. Effect by R}}{\underset{}{{\tau }_0\left(R\right)}}$$

• Identify generalization interval $H=[H_{-},H_{+}]$ for $t=0$: $$R_i=h\left(X_i\right)+{\eta }_i\forall R_i\in H$$

• If $H^{\ast }=max\left\{|H|\right\}$ exists, then for a set of covariates $X=X_T$: $$Y_0^{\left(0\right)}\left(R^{\prime }\right)|X_T=Y_0^{\left(0\right)}\left(R^{\prime \prime }\right)|X_T\text{for any}{R}^{\prime },R^{\prime \prime }\in H^{\ast }$$

Conditional outcome within the generalization interval

Main assumption for generalization to t=1

• No changes in unobserved confounders between $t=0$ and $t=1$ for units within $H^{\ast }$

• Partially testable for $Z=0$ in $t=1$

Estimating an effect away from the cutoff

Context: Representative template matching

Diagram for GRD

Step 0: Identification of narrow interval

Step 1: Template selection

Step 2: Matching units in pre-intervention period

Step 2: Matching units in pre-intervention period

Step 2: Matching units in pre-intervention period

Step 2: Matching units in pre-intervention period

Step 2: Matching units in pre-intervention period

Step 2: Matching units in pre-intervention period

Step 3: Fit a local polynomial

Step 4: Identify new generalization interval

Step 5: If generalization interval > template...

Step 5a: ... expand template and do it again.

Step 5b: ... until template is the same as the interval

Step 5b: ... until template is the same as the interval

Step 6: ATT estimation

Step 6: ATT estimation

Straightforward estimation given the matched sample:

• E.g. paired t-test:

$${\hat{\tau }}_{ATT}=\sum _{k=1}^N\frac{Y_{k\left(1\right)1}-Y_{k\left(0\right)1}-(Y_{k\left(1\right)0}-Y_{k\left(0\right)0})}{N}=\sum _{k=1}^N\frac{d_k}{N}$$

$Y_{k\left(z\right)t}$: Outcome within matched group $k$ with treatment $z$ for period $t$.

Free Higher Education (FHE) in Chile

Context of higher education in Chile:

• Centralized admission system (deferred admission mechanism)

• Admission score: PSU score + GPA score + ranking score

• Before 2016: Scholarships + government-backed loans

FHE policy:

• Introduced in December 2015 (unanticipated)

• Eligibility: Lower 50% income distribution + admitted to eligible program

Research question

What was the effect of being eligible for FHE on application and enrollment to university?

• Treatment: SE eligibility for FHE

• Two outcomes: Application to university and enrollment

  • Lower-income students $\to$ financial constraints
  • Saliency of the policy

• Larger effects for students away from the cutoff?

  • Compare RD to GRD results

What data do I have?

• 3 cohorts: 2014, 2015, and 2016 (~ 200,000 students)

• Rich baseline data: Demographic and socioeconomic data at student level, 10th (8th) grade standardized scores, school characteristics.

• Application data: Scores by subject, application, and enrollment.

How does the RD look like?

GRD for Free Higher Education

Steps for GRD:

• Select template size: $N=1,000$

• 20 bin for grid

• MIP matching:

  • Restricted mean balance (0.05 SD) : Academic performance, school characteristics, demographic/socioeconomic variables.

  • Fine balance: Gender, mother's and father's education (8 cat.), PSU language score (deciles), PSU math score (deciles), HS GPA (quintiles).

Generalization interval: [-M$500.3, M$300.9]

For what population are we generalizing for?

For what population are we generalizing for?

Testing time invariance on control side for t=1

Balance for the entire sample

Balance within generalization interval (before matching)

Balance within generalization interval (after matching)

Effects of introduction of FHE: RD and GRD

Effects of introduction of FHE: Application

Effects of introduction of FHE: Enrollment

How does the effect change with interval width?

Conclusions

• GRD as a gradual approach (not all or nothing)
  
  
• Use data to inform interval for generalization
  
  
• Use of matching to avoid extrapolation
  
  
• Limitations:
  • More data: two periods
  • Conditional time invariance assumption for $t=1$
    
    
• Multiple applications for DD-RD: e.g. geographic RDs.