Course Type

Since Fall 14, the gender distribution has become closer to equal, with slightly more females than males since Spring 16. This difference is moderately statistically significant (\(\chi^2_{3}\)=6.5, p=.09).

Performance - GPA

There is no statistical difference in the average GPA between males and females, regardless of cohort.

A multiple linear regression model found that the redesign did not have a statistically different impact on males and females.

Performance - DFW rate

There is no difference in proportion of males and females getting a repeatable grade on the entire sample, or when using data on the F14 and F16 cohorts only.

A logistic regression model on the full data indicates that students in the redesigned section (CRT and CRT + SI) have significantly lower odds of failing Chem 111 compared to students in the traditional courses after adjusting for gender.

Summary: The redesign had a positive effect on GPA and DWF rate after controlling for gender, and while it appears as if males improved more than females between Traditional and CRT + SI, the difference is not statistically significant.

Course Type

Students in F14 were on average 0.68 (0.01, 1.34) years older than those in the two CRT semesters (p=.04), and 0.87 (0.1, 1.65) years older than students in Fall 16 (p=.02).

Performance - GPA

GPA and age appear to have a non-linear relationship. Zooming in on students under 25, the relationship between age and GPA is decreasing until about 20, then increases afterward.

A linear spline model was fit on the full sample using a knot at 20 to allow for a change in slope while controlling for the effect of the course type.

Under 20, there is a very slight (\(\beta=-0.09\)) and non-significantly different from zero downward trend in GPA as age increases. For each year older than 20 a student is, the average GPA increases by 0.19 (p=.06). Students in the CRT + SI course have on average 0.34 higher GPA compared to students in the traditional class (p=.005).

An interaction term between age and course type was tested but found to be non-significant. This indicates that the intervention has a similar positive effect on a student’s GPA regardless of their age.

Performance - DFW rate

There is no difference in the average age for students receiving a repeatable grade or not, with our without the older students included in the analysis.

A logistic regression model on the full data across all ages indicates that students in the redesigned section (CRT and CRT + SI) have significantly lower odds of failing Chem 111 compared to students in the traditional courses after adjusting for age.

Summary: The redesign had a positive effect on GPA and DWF rate after controlling for age.

Course Type

The proportion of students in each URM category listed above has changed significantly since F14 (\(\chi^2_{6}\)=18.0, p=.006). Most notably is that the proportion of URM students has increased greatly from F14 to S16, becoming about the same proportion (40%) as the non-URM group.

Performance - GPA

The distribution of GPA across URM status is similar for the entire sample and when only considering F14 and F16.

When comparing the average GPA across URM status for all four semesters combined, Non URM students have on average 0.39 (95% CI 0.18-0.60, p<.001) higher GPA compared to URM students.

Next we examine how the average GPA for students in each URM classification changes with the different redesign models. The plot below shows the mean with 95% CI for each combination of URM status and course type. The confidence intervals are added to this plot to more easily identify which pairs of URM status within course type are different. If the CI bars do not overlap (e.g. Non-URM vs URM in the CRT) this is a significant difference. However, if the bars overlap that does not mean there is no difference. You cannot make a statistical conclusion if the bars overlap.

A multiple linear model of GPA using course type and URM status and the interaction between the two as predictors was run to test for a significant decrease in the gap between URM and non-URM students from each of the two redesigned course types compared to the traditional class. This is called a difference of differences analysis.

The table to the right shows that under the traditional course model the gap in GPA between URM and non URM was 0.38 (with URM being significantly lower), the gap widened to 0.53, with URM still being statistically significantly lower, but under the CRT + SI model of redesign, the gap decreased such that URM students do not have a significantly lower GPA compared to NonURM students.

The decrease from 0.32 to 0.28 however is not statistically significant (p=0.48).

Performance - DFW rate

The failure rate for URM status has decreased as the level of intervention increased from Traditional to CRT to CRT + SI.

Summary: The CRT+SI redesign is associated with an improvement in GPA and reduction in DFW rates in URM students.

Course Type

The proportion of students that are first generation has remained statistically constant for the past four semesters.

Performance - GPA

When considering the entire four semesters, first generation students have on average 0.29 gpa points lower than non-first generation students. The difference is non-significant when only considering the F14 and F16 cohorts, indicating that these samples are not the same.

Next we examine the average GPA for first generation and non-first generation students under the different redesign models. The mean profile plot with 95% CI below indicate that just flipping the classroom improves the overall GPA of the class, but primarily for non-first generation students only. However when adding SI to the course model the GPA for the first generation students rises to meet that of the non-first gen. This explains why the difference in average GPA is significantly different when considering the full sample, but not when considering the Traditional and CRT + SI cohorts only.

Performance - DFW rate

The failure rate for each group decreases in a similar pattern to the improvement in GPA.

For both groups the DFW rate went from 43-45% under the traditional course type, to 32-33% under redesign with SI.

Summary: Both the course type and first generation status are associated with a difference in GPA and DFW rate, but the redesign type has different effects for first generation students, specifically the redesign was associated with an improvement in performance for non-first generation students only. Adding on SI to the redesigned class bridged that gap for the first generation students.

Course Type

There is no significant difference in average admissions index across the three course types.

Performance - GPA

Admissions index does not have a strictly linear relationship with GPA, and that the relationship differs across the course types. For the Traditional and CRT + SI , there appears to be a flat relationship between HSEI and GPA until about HSEI=3.5, then a positive relationship for HSEI above 3.5.

• The slopes below HSEI < 3.5 do not significantly differ based on course type - this is primarily due to the large standard error of the slope rather than the magnitude of the slope itself.
• The GPA for students in the CRT + SI is moderately (p<.10) larger than the average GPA for students in the traditional classes (0.38, 95% CI -0.07, 0.83)

A linear spline model was fit on the full sample using a knot at 3.5 to allow for a change in slope. No interaction terms were included since there was no significant difference in slope across CRT group.

• There no effect of HSEI on GPA when HSEI is below 3.5 (p=.7)
• There is a strong effect of HSEI on GPA when HSEI is above 3.5 (p=.0089)
• After controlling for HSEI there is still a significant improvement in GPA for students in the CRT + SI redesign compared to those in the traditional course.

Course Type - DFW

Students who pass CHEM 111 have on average a 0.2 GPA higher admissions index compared to students who receive a repeatable grade (3.62 vs 3.82, p<.0001).

Summary:

• After controlling for admissions index, there is no direct effect of the course redesign on the likelihood a student will pass the course.
• There is a positive relationship between admissions index and GPA, but only for a high admissions score.
• The type of course redesign does not significantly change the relationship between admissions index and GPA.

Collapsing and grouping varibles.

• The number of college prep math, English, and lab science classes were grouped into more reasonable ranges such as under 6 units, 6 to 8, and over 10. Reference groups are always no units of college prep classes.
• The college in which the student is currently a major of was collapsed into
  • Other: BSS, BUS, CME, HFA, UGE
  • ECC
  • NS: NS and AGR

Restrict on age

Since 95% of the students are below 25 years of age, the multivariable model will be fit on only students less than 25 years of age.

Imputing missing values

About 15% of the records in the data were missing admissions index, so for the purpose of machine learning modeling missing values are imputed using predictive mean matching via the MICE (multiple imputation using chained equations) package in R. Only one computed data set is used (instead of multiple) for exploratory purposes only.

Notes and Comments

• These models are an additive, each covariate is modeled as if it modifies GPA or the probability of passing the class in a linear manner.
• This model is simple in that unless interactions between variables are explicitly specified, they are not considered.
• This differs from the supervised learning model such as a Random Forest model, that allows for all variables to interact with each other.
  • The purpose of these models is classification, not assessing the impact of a covariate on an outcome.
• The effect for all categorical variables represent a change from baseline or reference group, to the group listed.
• This analysis only examined courses taught by the same instructor across four semsters to assess the effectiveness of a single individual course redesign.