mbennett_attendance.knit

class: inverse, center, middle

<br>

.left[
<h1 class="title-own">Beyond Exclusion:<br/>The Role of High-Stake Testing on Attendance the Day of the Test</h1>]

.pull-left-little_l[
<br>
<br>
<br>
<br>
<br>
<br>

.left[.small[PRIISM Seminar<br>April 3rd, 2024]]]

.pull-right-little_l[
<br>
.small[.right[Magdalena Bennett&nbsp;&nbsp;&nbsp;<br>*The University of Texas at Austin*&nbsp;&nbsp;&nbsp;]]

<br>
.small[.right[Christopher Neilson&nbsp;&nbsp;&nbsp;<br>*Yale University*&nbsp;&nbsp;&nbsp;]]

<br>
.small[.right[Nicolás Rojas&nbsp;&nbsp;&nbsp;<br>*Columbia University*&nbsp;&nbsp;&nbsp;]]

]
<br>
<br>
<br>

---
# Motivation

- Results from **.coolblue[high-stakes tests]** widely used in education policy

---
# Motivation

- Results from **.coolblue[high-stakes tests]** widely used in education policy
   
.center[
![:scale 50%](https://raw.githubusercontent.com/maibennett/presentations/main/content/presentations/Attendance/PRIISM_20240402/images/iitems1.svg)]

---
# Motivation

- **.coolblue[Assumption]**: Standardize tests used as a proxy of school quality
<br>
<br>
--

- **.coolblue[Multiple issues]** with the use of standardized tests:

- Teaching to the test/explicit cheating.
  
  - Correlation with SES.
  
  - **.coolblue[Non-representative patterns of attendance.]**
  
---
# Motivation

- Prior literature related to student exclusion:

- Reclassification of low-performers as students with disabilities .small[(e.g. Figlio & Loeb, 2011; Figlio & Getzler, 2007; Cullen & Reback, 2006)]
  - Use of disciplinary measures to exclude low-performers .small[(Figlio, 2006)]

--
<br>
<br>
- Some studies analyzing the **.coolblue[effect of attendance manipulation in Chile]**
  
  - Distortion in quality signals .small[(Cuesta et al., 2020)]
  - Manipulation for specific vulnerable schools (SEP) to raise scores .small[(Feigenberg et al., 2019; Quezada & Hippel, 2017)]

--
<br>
<br>
- Schools have **.coolblue[incentives]** to game the system

- Especially in **.coolblue[high-accountability settings]**

---
# This paper

**.coolblue[Attendance Patterns]**

- Event study approach:

- *How do these exclusions patterns look like? Are these the same for every (type of) school and every grade?*
  
  - Focus beyond bottom performers
  
  - Robustness checks for alternative mechanisms

**.coolblue[What can be done?]**

- Machine learning prediction:

- Identification of schools that are most likely gaming the system
  
  - Consequences of blanket policies in imputation of scores

---
background-position: 50% 50%
class: left, middle, inverse
<br>
<br>
<br>
<br>
<br>
<br>
<br>
.big[
The Chilean Educational Context
]

---
# The Chilean context: Standardized testing

- Chile has a **.coolblue[universal voucher system]** (school choice)

--
<br>
<br>

- Universal standardized testing since 1980's (SIMCE)
  
  - For all 4th graders; then extended to other grades.

--
<br>
<br>

- SIMCE as a **.coolblue[high-stake]** test:

- Results widely available in a universal voucher system
  
  - Tied to teachers' bonuses
  
  - Tied to budget restrictions and school closures
  
---
# SIMCE and absenteeism

- Use of **.coolblue[pre-filled communication]** for parents to be sent out by schools

- Evidence that parents from lower-income students are less likely to receive information
  
--
<br>
<br>

- **.coolblue[No real consequences for low attendance]**:
  
  - Between 2005-2007, non-representative results where marked with symbols
  
  - No imputation strategy so far
  
--
<br>
<br>

- Improvement of regulation for **.coolblue[justifying students exclusion]**

- E.g. specific disabilities (blindness) or non-Spanish speakers.

---
background-position: 50% 50%
class: left, middle, inverse
<br>
<br>
<br>
<br>
<br>
<br>
<br>
.big[
Attendance Patterns for the Day of the Test
]

---
# Data Available

- **.coolblue[Standardized tests 2011-2018 (SIMCE)]**

- Scores at student and school level for different subjects (Math, Language, History, and Science)
  
  - Student's socioeconomic characterization (parental questionnaire)

--
<br>
<br>

- **.coolblue[Daily attendance data 2011-2018 (SIGE)]**

- Use for voucher payments (each day has &#126; 2.5 million records)
  
--
<br>
<br>

- **.coolblue[GPA Performance 2011-2018 (Rendimiento)]**

- Use GPA performance deciles within school-grade
  
---
# Observations from our data

---
# Empirical approach for difference in attendance

- Event study centered around the day of the test (T=0):

`$$Y_{ipsgt} = \sum_{P=1}^5\sum_{T=-4}^5 \tau^{PT}D^{PT}_{ipsgt} + \gamma_{pt} +\alpha_i + \epsilon_{ipsgt}$$`

Where
- `$Y_{ipsgt}$`: Binary attendance for student `$i$`, from GPA group `$p$`, in school `$s$` and grade `$g$`, for day `$t$`.

- `$D^{PT}_{ipsgt}$`: Indicator variables (lags and leads) for students that belong to a tested grade.

---
# Clear difference in attendance by performance for 2nd grade

---
# No effect on lower performers for 10th grade

---
# Attendance patterns differ by grade

---
# Potential mechanisms that explain these patterns

<br>

- Students are **.coolblue[excluded due to other reasons]** (justified)

- Low-performing students are **.coolblue[less aware of the test]**

- Students experience a **.coolblue[disutility from testing]**

- Schools directly **.coolblue[(des)incentivize attendance of (lower)higher performers]**

---
# Differences in communication and incentives between high and low performers

- 2017 survey for students in test-taking grades.

<table class=" lightable-paper" style='font-family: "Arial Narrow", arial, helvetica, sans-serif; width: auto !important; margin-left: auto; margin-right: auto;'>
<caption>Results for 4th Grade</caption>
 <thead>
  <tr>
   <th style="text-align:left;"> GPA Decile </th>
   <th style="text-align:center;"> Told </th>
   <th style="text-align:center;"> Notification </th>
   <th style="text-align:center;"> Preparation </th>
   <th style="text-align:center;"> Grades </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> D1 </td>
   <td style="text-align:center;"> -0.06*** </td>
   <td style="text-align:center;"> -0.11*** </td>
   <td style="text-align:center;"> -0.08*** </td>
   <td style="text-align:center;"> 0.14*** </td>
  </tr>
  <tr>
   <td style="text-align:left;">  </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> D10 </td>
   <td style="text-align:center;"> 0.06*** </td>
   <td style="text-align:center;"> 0.05*** </td>
   <td style="text-align:center;"> 0.05*** </td>
   <td style="text-align:center;"> -0.2*** </td>
  </tr>
  <tr>
   <td style="text-align:left;">  </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Baseline </td>
   <td style="text-align:center;"> 0.89*** </td>
   <td style="text-align:center;"> 0.87*** </td>
   <td style="text-align:center;"> 0.89*** </td>
   <td style="text-align:center;"> 0.39*** </td>
  </tr>
  <tr>
   <td style="text-align:left;">  </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
  </tr>
</tbody>
</table>

---
# Differences in communication and incentives between high and low performers

- 2017 survey for students in test-taking grades.

<table class=" lightable-paper" style='font-family: "Arial Narrow", arial, helvetica, sans-serif; width: auto !important; margin-left: auto; margin-right: auto;'>
<caption>Results for 4th Grade</caption>
 <thead>
  <tr>
   <th style="text-align:left;"> GPA Decile </th>
   <th style="text-align:center;"> Told </th>
   <th style="text-align:center;"> Notification </th>
   <th style="text-align:center;"> Preparation </th>
   <th style="text-align:center;"> Grades </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> D1 </td>
   <td style="text-align:center;"> <span style=" font-weight: bold;    color: #A5AC20 !important;border-radius: 4px; padding-right: 4px; padding-left: 4px; background-color: rgba(165, 172, 32, 0.3) !important;">-0.06***</span> </td>
   <td style="text-align:center;"> <span style=" font-weight: bold;    color: #A5AC20 !important;border-radius: 4px; padding-right: 4px; padding-left: 4px; background-color: rgba(165, 172, 32, 0.3) !important;">-0.11***</span> </td>
   <td style="text-align:center;"> -0.08*** </td>
   <td style="text-align:center;"> 0.14*** </td>
  </tr>
  <tr>
   <td style="text-align:left;">  </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> D10 </td>
   <td style="text-align:center;"> 0.06*** </td>
   <td style="text-align:center;"> 0.05*** </td>
   <td style="text-align:center;"> 0.05*** </td>
   <td style="text-align:center;"> -0.2*** </td>
  </tr>
  <tr>
   <td style="text-align:left;">  </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Baseline </td>
   <td style="text-align:center;"> 0.89*** </td>
   <td style="text-align:center;"> 0.87*** </td>
   <td style="text-align:center;"> 0.89*** </td>
   <td style="text-align:center;"> 0.39*** </td>
  </tr>
  <tr>
   <td style="text-align:left;">  </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
  </tr>
</tbody>
</table>

---
# Differences in communication and incentives between high and low performers

- 2017 survey for students in test-taking grades.

<table class=" lightable-paper" style='font-family: "Arial Narrow", arial, helvetica, sans-serif; width: auto !important; margin-left: auto; margin-right: auto;'>
<caption>Results for 4th Grade</caption>
 <thead>
  <tr>
   <th style="text-align:left;"> GPA Decile </th>
   <th style="text-align:center;"> Told </th>
   <th style="text-align:center;"> Notification </th>
   <th style="text-align:center;"> Preparation </th>
   <th style="text-align:center;"> Grades </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> D1 </td>
   <td style="text-align:center;"> <span style=" font-weight: bold;    color: #A5AC20 !important;border-radius: 4px; padding-right: 4px; padding-left: 4px; background-color: rgba(165, 172, 32, 0.3) !important;">-0.06***</span> </td>
   <td style="text-align:center;"> <span style=" font-weight: bold;    color: #A5AC20 !important;border-radius: 4px; padding-right: 4px; padding-left: 4px; background-color: rgba(165, 172, 32, 0.3) !important;">-0.11***</span> </td>
   <td style="text-align:center;"> -0.08*** </td>
   <td style="text-align:center;"> 0.14*** </td>
  </tr>
  <tr>
   <td style="text-align:left;">  </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> D10 </td>
   <td style="text-align:center;"> <span style=" font-weight: bold;    color: #4493AE !important;border-radius: 4px; padding-right: 4px; padding-left: 4px; background-color: rgba(68,147,174, 0.3) !important;">0.06***</span> </td>
   <td style="text-align:center;"> <span style=" font-weight: bold;    color: #4493AE !important;border-radius: 4px; padding-right: 4px; padding-left: 4px; background-color: rgba(68,147,174, 0.3) !important;">0.05***</span> </td>
   <td style="text-align:center;"> 0.05*** </td>
   <td style="text-align:center;"> -0.2*** </td>
  </tr>
  <tr>
   <td style="text-align:left;">  </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Baseline </td>
   <td style="text-align:center;"> 0.89*** </td>
   <td style="text-align:center;"> 0.87*** </td>
   <td style="text-align:center;"> 0.89*** </td>
   <td style="text-align:center;"> 0.39*** </td>
  </tr>
  <tr>
   <td style="text-align:left;">  </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
  </tr>
</tbody>
</table>

---
# Differences in communication and incentives between high and low performers

- 2017 survey for students in test-taking grades.

<table class=" lightable-paper" style='font-family: "Arial Narrow", arial, helvetica, sans-serif; width: auto !important; margin-left: auto; margin-right: auto;'>
<caption>Results for 10th Grade</caption>
 <thead>
  <tr>
   <th style="text-align:left;"> GPA Decile </th>
   <th style="text-align:center;"> Told </th>
   <th style="text-align:center;"> Notification </th>
   <th style="text-align:center;"> Preparation </th>
   <th style="text-align:center;"> Grades </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> D1 </td>
   <td style="text-align:center;"> <span style=" font-weight: bold;    color: #A5AC20 !important;border-radius: 4px; padding-right: 4px; padding-left: 4px; background-color: rgba(165, 172, 32, 0.3) !important;">-0.02***</span> </td>
   <td style="text-align:center;"> <span style=" font-weight: bold;    color: #A5AC20 !important;border-radius: 4px; padding-right: 4px; padding-left: 4px; background-color: rgba(165, 172, 32, 0.3) !important;">-0.01***</span> </td>
   <td style="text-align:center;"> -0.02*** </td>
   <td style="text-align:center;"> 0.05*** </td>
  </tr>
  <tr>
   <td style="text-align:left;">  </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> D10 </td>
   <td style="text-align:center;"> <span style=" font-weight: bold;    color: #4493AE !important;border-radius: 4px; padding-right: 4px; padding-left: 4px; background-color: rgba(68,147,174, 0.3) !important;">0.01***</span> </td>
   <td style="text-align:center;"> <span style=" font-weight: bold;    color: #4493AE !important;border-radius: 4px; padding-right: 4px; padding-left: 4px; background-color: rgba(68,147,174, 0.3) !important;">0.00</span> </td>
   <td style="text-align:center;"> 0.00 </td>
   <td style="text-align:center;"> -0.03*** </td>
  </tr>
  <tr>
   <td style="text-align:left;">  </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Baseline </td>
   <td style="text-align:center;"> 0.95*** </td>
   <td style="text-align:center;"> 0.78*** </td>
   <td style="text-align:center;"> 0.82*** </td>
   <td style="text-align:center;"> 0.33*** </td>
  </tr>
  <tr>
   <td style="text-align:left;">  </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.00) </td>
  </tr>
</tbody>
</table>

---
# No evidence of self-selection from students because of testing

.pull-left[
- Students experience a **.coolblue[disutility from testing]**?

- Use of **.coolblue[no-stake test]** applied to schools (~300 schools per year) `$\rightarrow$` No effect on attendance

]

.pull-right[
.small[
<table class=" lightable-paper" style='font-family: "Arial Narrow", arial, helvetica, sans-serif; width: auto !important; margin-left: auto; margin-right: auto;'>
<caption>Results for No-Stakes Test</caption>
 <thead>
  <tr>
   <th style="text-align:left;"> Grade - Year </th>
   <th style="text-align:center;"> D1 </th>
   <th style="text-align:center;"> D2 </th>
   <th style="text-align:center;"> D3D8 </th>
   <th style="text-align:center;"> D9 </th>
   <th style="text-align:center;"> D10 </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> 2nd 2011 </td>
   <td style="text-align:center;"> -0.01 </td>
   <td style="text-align:center;"> 0.01 </td>
   <td style="text-align:center;"> 0.01* </td>
   <td style="text-align:center;"> 0.02 </td>
   <td style="text-align:center;"> 0.00 </td>
  </tr>
  <tr>
   <td style="text-align:left;">  </td>
   <td style="text-align:center;"> (0.01) </td>
   <td style="text-align:center;"> (0.01) </td>
   <td style="text-align:center;"> (0.01) </td>
   <td style="text-align:center;"> (0.01) </td>
   <td style="text-align:center;"> (0.01) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 5th 2012 </td>
   <td style="text-align:center;"> 0.00 </td>
   <td style="text-align:center;"> -0.01 </td>
   <td style="text-align:center;"> 0.00 </td>
   <td style="text-align:center;"> 0.01 </td>
   <td style="text-align:center;"> 0.01 </td>
  </tr>
  <tr>
   <td style="text-align:left;">  </td>
   <td style="text-align:center;"> (0.01) </td>
   <td style="text-align:center;"> (0.01) </td>
   <td style="text-align:center;"> (0.01) </td>
   <td style="text-align:center;"> (0.01) </td>
   <td style="text-align:center;"> (0.01) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 6th 2011 </td>
   <td style="text-align:center;"> 0.02* </td>
   <td style="text-align:center;"> 0.01 </td>
   <td style="text-align:center;"> 0.01** </td>
   <td style="text-align:center;"> 0.01 </td>
   <td style="text-align:center;"> 0.00 </td>
  </tr>
  <tr>
   <td style="text-align:left;">  </td>
   <td style="text-align:center;"> (0.01) </td>
   <td style="text-align:center;"> (0.01) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.01) </td>
   <td style="text-align:center;"> (0.01) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 6th 2017 </td>
   <td style="text-align:center;"> 0.00 </td>
   <td style="text-align:center;"> 0.03 </td>
   <td style="text-align:center;"> 0.01 </td>
   <td style="text-align:center;"> 0.01 </td>
   <td style="text-align:center;"> 0.00 </td>
  </tr>
  <tr>
   <td style="text-align:left;">  </td>
   <td style="text-align:center;"> (0.02) </td>
   <td style="text-align:center;"> (0.02) </td>
   <td style="text-align:center;"> (0.01) </td>
   <td style="text-align:center;"> (0.02) </td>
   <td style="text-align:center;"> (0.01) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 11th 2012 </td>
   <td style="text-align:center;"> 0.00 </td>
   <td style="text-align:center;"> 0.00 </td>
   <td style="text-align:center;"> 0.00 </td>
   <td style="text-align:center;"> -0.02** </td>
   <td style="text-align:center;"> 0.00 </td>
  </tr>
  <tr>
   <td style="text-align:left;">  </td>
   <td style="text-align:center;"> (0.01) </td>
   <td style="text-align:center;"> (0.01) </td>
   <td style="text-align:center;"> (0.00) </td>
   <td style="text-align:center;"> (0.01) </td>
   <td style="text-align:center;"> (0.01) </td>
  </tr>
</tbody>
</table>
]
]

---
background-position: 50% 50%
class: left, middle, inverse
<br>
<br>
<br>
<br>
<br>
<br>
<br>
.big[
Predicting the Counterfactual
]

---
# How do these results compare to predicted counterfactual?

- Can we use **.coolblue[this existing rich panel data]** to predict attendance on the day of the test *as if it was a regular day*?

--
<br>
<br>

- Use **.coolblue[Extreme Gradient Boosting (XGBoost)]** with panel data for **.coolblue[attendance prediction]** 
  
  - Predictor variables include lags for attendance, day of the week, grade, GPA group, and sibling's attendance, in addition to student and schools' characteristics. 
  
--
<br>
<br>

- Use data between 1st and 5th grade (2017) in the Metropolitan region - 4th grade is treated:

- Data **.coolblue[before]** the test to **.coolblue[predict attendance on the day of the test]**.

---
# Overall predictions over performance distribution

---
# Overall predictions over performance distribution

---
# Example: Comparisons between schools?

.pull-left[
.center[
.box-7trans[**School1**<br>
.small[Math: 258<br>
Language: 260]]
]]

.pull-right[
.center[
.box-7trans[**School2**<br>
.small[Math: 259<br>
Language: 256]]
]]
--
<img src="mbennett_attendance_files/figure-html/prediction_example-1.svg" style="display: block; margin: auto;" />

---
# Example: Comparisons between schools?
.pull-left[
.center[
.box-7trans[**School1**<br>
.small[Math: 258<br>
Language: 260]]
]]

.pull-right[
.center[
.box-7trans[**School2**<br>
.small[Math: 259<br>
Language: 256]]
]]

---
# Can we characterize these schools?

.small[
- **.coolblue[K-means clustering]** Use differences between predicted and observed attendance.
  - 2 optimal clusters]
--
<img src="mbennett_attendance_files/figure-html/cluster_plot-1.svg" style="display: block; margin: auto;" />