Motivation

• Results from high-stakes tests widely used in education policy

  • E.g. funding, promotions, school closures, school choice, etc.
    
    

• Assumption: Standardize tests used as a proxy of school quality

Is it so?

Motivation

Motivation

• Beyond explicit cheating and socioeconomic sorting: Students' exclusion

  • E.g.: Reclassification of low-performers as students with disabilities (Figlio & Loeb, 2011)
  • Use of disciplinary measures to exclude low-performers (Figlio, 2006)
    
    

• Less attention on non-representative attendance patterns

  • Differences between scores before and after imputation (Cuesta et al., 2020)
    
    

• Schools have incentives to game the system

  • Especially in high-accountability settings

This paper

Attendance Patterns

• Event study approach:

  • How do these exclusions patterns look like? Are these the same for every (type of) school and every grade?

  • Focus beyond bottom performers

  • Robustness checks for alternative mechanisms

Imputation Policies

• Machine learning prediction:

  • Consequences of blanket policies in imputation of scores

  • Identification of schools that are most likely gaming the system

Outline

1. Motivation

2. Chilean educational context

3. Attendance patterns:

   • Event study for different years, grades, and performance
   • Potential mechanisms

4. Prediction approach:

   • Difference between predicted and observed distributions
   • Potential consecuences of imputation

5. Conclusions and next steps

The Chilean context: Standardized testing

• Universal standardized testing since 1980's (SIMCE)
  
  

• SIMCE as high-stake testing:

  • Results widely available in a universal voucher system

  • Tied to teachers' bonuses

  • Tied to budget restrictions and school closures

SIMCE and absenteeism

• Use of pre-filled communication for parents to be sent out by schools

  • Evidence that parents from lower-income students are less likely to receive information
    
    

• No real consequences for low attendance:

  • Between 2005-2007, non-representative results where marked with symbols

  • No imputation strategy so far
    
    

• Improvement of regulation for justifying students exclusion

  • E.g. specific disabilities (blindness) or non-Spanish speakers.

How to evaluate the effect of "day of the test" on abstenteeism?

• Some studies assessing the effect of attendance manipulation:

  • Focus on distortions (difference between imputed and observed scores) (Cuesta et al., 2020)

  • Manipulation for specific vulnerable schools (SEP) to raise scores (Feigenberg et al., 2019; Quezada & Hippel, 2017)
    
    

• This paper: Event study between 2011 and 2018 for all tested grades.

  • Focus on attendance by within-school performance

  • Use of alternative non-high-stake test to analyze potential mechanisms

  • Use of unpublished survey for communication and incentives around SIMCE

Data Available

• Standardized tests 2011-2018 (SIMCE)

  • Scores at student and school level for different subjects (Math, Language, History, and Science)

  • Student's socioeconomic characterization (parental questionnaire)
    
    

• Daily attendance data 2011-2018 (SIGE)

  • Use for voucher payments (each day has ~ 2.5 million records)
    
    

• GPA Performance 2011-2018 (Rendimiento)

  • Use GPA performance deciles within school-grade

Observations from our data

Empirical approach for difference in attendance

• Event study centered around the day of the test:

$$Y_{ipsgt}=\sum _{P=1}^5\sum _{T=-4}^5{\tau }^{PT}{D}_{ipsgt}^{PT{G}^{\ast }}+{\gamma }_{pt}+{\alpha }_i+{ϵ}_{ipsgt}$$

Where

• $Y_{ipsgt}$: Binary attendance for student $i$, from GPA group $p$, in school $s$ and grade $g$, for day $t$ (centered around the day of the test).

• $D_{ipsgt}^{PT{G}^{\ast }}$: Indicator variable $I(p=P,t=T,g=G^{\ast })$, where $G^{\ast }$ is the tested grade. Coefficient of interest is $D^{P0G^{\ast }}$.

Clear difference in attendance by performance for 2nd grade

No effect on lower performers for 10th grade

Attendance patterns differ by grade

Potential mechanisms that explain these patterns

• Schools directly (des)incentivize attendance of (lower)higher performers

• Students are excluded due to other reasons (justified).

• Students experience a disutility from testing

Differences in communication and incentives between high and low performers

• Schools directly (des)incentivize attendance of (lower)higher performers

  • 2017 survey for students in test-taking grades.

Differences in communication and incentives between high and low performers

• Schools directly (des)incentivize attendance of (lower)higher performers

  • 2017 survey for students in test-taking grades.

Use of exemptions to exclude students don't tell the whole story

• Students are excluded due to other reasons (justified).

  • Change in exemption policy in 2012 $\to$ reduction in exempted students (flattened)

  • Results remain similar after 2012

No self-selection from students because of testing

How do these results compare to predicted counterfactual?

• Can we use this existing rich panel data to predict attendance on the day of the test as if it was a regular day?
  
  

• Use XGBoost with panel data for attendance prediction

  • Model includes fixed effects by day of the week, school, grade, and student.

  • Also includes sibling attendance and attendance lag.
    
    

• Use data for 4th grade (2017):

  • Data before the test to predict attendance on the day of the test.

Overall predictions over performance distribution

Example: Comparisons between schools?

Example: Comparisons between schools?

Can we characterize these schools?

Schools that appear to exclude lower-perfoming students are also more vulnerable

Implications for imputation policies

• How to handle this absenteeism problem?

  • Scenario 1: Observed attendance (no imputation)

  • Scenario 2: Attendance as if the test hadn't happened (impute "typical day")

  • Scenario 3: Everybody is present
    
    

• Proposals to impute lowest scores for absent students to desincentivize arbitrary exclusion

  • Most vulnerable schools have higher absenteeism rates $\to$ Increase inequality and non-representativeness

Conclusions and next steps

• Non-representative patterns of absenteeism beyond exclusion of low-performers

  • High heterogeneity between schools
    
    

• Communication strategies play important role for lower-performing students
  
  

• Impact of imputation policies?

  • Work in progress: How does non-representativeness and different imputation strategies impact policies and information provision?