Motivation

• Results from high-stakes tests widely used in education policy

Motivation

• Results from high-stakes tests widely used in education policy

Motivation

• Results from high-stakes tests widely used in education policy

Motivation

• Results from high-stakes tests widely used in education policy

Motivation

• Results from high-stakes tests widely used in education policy

Motivation

• Results from high-stakes tests widely used in education policy

• Assumption: Standardize tests used as a proxy of school quality
  
  

• Multiple issues with the use of standardized tests:

  • Teaching to the test/explicit cheating.

  • Correlation with SES.

  • Non-representative patterns of attendance.

Motivation

• Prior literature related to student exclusion:

  • Reclassification of low-performers as students with disabilities (e.g. Figlio & Loeb, 2011; Figlio & Getzler, 2007; Cullen & Reback, 2006)
  • Use of disciplinary measures to exclude low-performers (Figlio, 2006)
    
    

• Some studies analyzing the effect of attendance manipulation in Chile

  • Distortion in quality signals (Cuesta et al., 2020)
  • Manipulation for specific vulnerable schools (SEP) to raise scores (Feigenberg et al., 2019; Quezada & Hippel, 2017)
    
    

• Schools have incentives to game the system

  • Especially in high-accountability settings

This paper

Attendance Patterns

• Event study approach:

  • How do these exclusions patterns look like? Are these the same for every (type of) school and every grade?

  • Focus beyond bottom performers

  • Robustness checks for alternative mechanisms

What can be done?

• Machine learning prediction:

  • Identification of schools that are most likely gaming the system

  • Consequences of blanket policies in imputation of scores

The Chilean context: Standardized testing

• Chile has a universal voucher system (school choice)
  
  

• Universal standardized testing since 1980's (SIMCE)

  • For all 4th graders; then extended to other grades.
    
    

• SIMCE as a high-stake test:

  • Results widely available in a universal voucher system

  • Tied to teachers' bonuses

  • Tied to budget restrictions and school closures

SIMCE and absenteeism

• Use of pre-filled communication for parents to be sent out by schools

  • Evidence that parents from lower-income students are less likely to receive information
    
    

• No real consequences for low attendance:

  • Between 2005-2007, non-representative results where marked with symbols

  • No imputation strategy so far
    
    

• Improvement of regulation for justifying students exclusion

  • E.g. specific disabilities (blindness) or non-Spanish speakers.

Data Available

• Standardized tests 2011-2018 (SIMCE)

  • Scores at student and school level for different subjects (Math, Language, History, and Science)

  • Student's socioeconomic characterization (parental questionnaire)
    
    

• Daily attendance data 2011-2018 (SIGE)

  • Use for voucher payments (each day has ~ 2.5 million records)
    
    

• GPA Performance 2011-2018 (Rendimiento)

  • Use GPA performance deciles within school-grade

Observations from our data

Empirical approach for difference in attendance

• Event study centered around the day of the test (T=0):

$$Y_{ipsgt}=\sum _{P=1}^5\sum _{T=-4}^5{\tau }^{PT}{D}_{ipsgt}^{PT}+{\gamma }_{pt}+{\alpha }_i+{ϵ}_{ipsgt}$$

Where

• $Y_{ipsgt}$: Binary attendance for student $i$, from GPA group $p$, in school $s$ and grade $g$, for day $t$.

• $D_{ipsgt}^{PT}$: Indicator variables (lags and leads) for students that belong to a tested grade.

Clear difference in attendance by performance for 2nd grade

No effect on lower performers for 10th grade

Attendance patterns differ by grade

Potential mechanisms that explain these patterns

• Students are excluded due to other reasons (justified)

• Low-performing students are less aware of the test

• Students experience a disutility from testing

• Schools directly (des)incentivize attendance of (lower)higher performers

Differences in communication and incentives between high and low performers

• 2017 survey for students in test-taking grades.

Differences in communication and incentives between high and low performers

• 2017 survey for students in test-taking grades.

Differences in communication and incentives between high and low performers

• 2017 survey for students in test-taking grades.

Differences in communication and incentives between high and low performers

• 2017 survey for students in test-taking grades.

No evidence of self-selection from students because of testing

How do these results compare to predicted counterfactual?

• Can we use this existing rich panel data to predict attendance on the day of the test as if it was a regular day?
  
  

• Use Extreme Gradient Boosting (XGBoost) with panel data for attendance prediction

  • Predictor variables include lags for attendance, day of the week, grade, GPA group, and sibling's attendance, in addition to student and schools' characteristics.
    
    

• Use data between 1st and 5th grade (2017) in the Metropolitan region - 4th grade is treated:

  • Data before the test to predict attendance on the day of the test.

Overall predictions over performance distribution

Overall predictions over performance distribution

Example: Comparisons between schools?

Example: Comparisons between schools?

Can we characterize these schools?

Schools that appear to exclude lower-perfoming students are also more vulnerable

Schools that appear to exclude lower-perfoming students are also more vulnerable

Schools that appear to exclude lower-perfoming students are also more vulnerable

Schools that appear to exclude lower-perfoming students are also more vulnerable

Schools that appear to exclude lower-perfoming students are also more vulnerable

Schools that appear to exclude lower-perfoming students are also more vulnerable

Implications for imputation policies

• How to handle this absenteeism problem?

  • E.g.: Observed attendance (no imputation), attendance as if the test hadn't happened (impute "typical day"), everybody is present.
    
    

• Proposals to impute lowest scores for absent students to disincentivize arbitrary exclusion

  • Most vulnerable schools have higher absenteeism rates $\to$ Increase inequality and non-representativeness

Some imputation exercises

How can we impute missing scores?

• Scenario 1: Not impute at all. Show observed distributions.

• Scenario 2: Impute by decile only for the difference between predicted and observed attendance.

  • Imputed score: (a) overall min, (b) decile min, (c) min school, or (d) min decile by school.

• Scenario 3: Impute every missing student.

  • Imputed score: overall min

Some caveats:

• Difference between predicted and obs. captures total incentives/disincentives in attendance.

• Imputed score might be too optimistic (e.g. real score would be lower than observed distribution)

Scenario 1 vs Scenario 3: No imputation and Impute all

Imputing Predicted - Observed is less extreme

Conclusions and next steps

• Non-representative patterns of absenteeism beyond exclusion of low-performers

  • High heterogeneity between schools
  • Identification of gaming schools (e.g. persistance)
    
    

• Communication strategies play important role for lower-performing students
  
  

• Impact of imputation policies?

  • Work in progress: How does non-representativeness and different imputation strategies impact policies and information provision? What score do we impute and for whom?