CircleIn – First term outcomes assessment

Monroe Community College serves over 11,000 students across two campuses, two centers and multiple other locations, offering more than 100 degree and certificate programs at a significantly lower cost than most higher education institutions in the country. One third of Monroe students are 25 years or older and 43 percent are underrepresented minorities. MCC is a board member of the League for Innovation in the Community College. Like many community colleges, Monroe has seen a decrease in enrollment over the last decade. As retention has remained relatively flat, operating cost per FTE has increased- meaning that we want to be innovative in our approach to student success but cannot afford to support initiatives that are not having a measurable impact. As such, when we launched CircleIn at the end of our spring 2020 term we developed an assessment plan to measure the impact of this peer to peer study app. 

Underlying our interest in the app, Monroe had studied student course persistence and found that classroom community has a significant impact on persistence. Our Teaching and Creativity Center shares strategies for creating community in the classroom, but we wanted more. CircleIn appeared to provide a classroom community that all students could access whether they were non-traditional students with competing demands, had prior experience with the material, or were taking online courses. In addition, the app uses game design to provide incentives for students to build a study habit. 

At the conclusion of the spring term we looked at the difference in demographics of students who used the app and the general population. CircleIn app users were more likely to be

• Female
• Black/African-American
• Full-time

*And they were less likely to use college learning centers/tutors. This was an interesting finding since this academic support appears to reach a new segment of students. 

There was also a significant difference in student outcomes. The average term GPA for app users was 3.06 vs. 2.418 for non-users. This trend held across gender and ethnicity. For example, we found:

• 50% higher term GPA for Black men using CircleIn
• 41% higher term GPA for Black women using CircleIn
• 46% higher term GPA scores for Hispanic men using CircleIn
• 31% higher term GPA scores for Hispanic women using CircleIn

Monroe recognizes that many of the students who took advantage of the app may be already more likely to persist. This kind of selection bias is common when it is not possible to execute a randomized control trial (RCT) or when it is difficult to obtain a representative sample. During the fall 2020 terms, the college applied prediction-based propensity score matching (PPSM) to find the group of students who participated in the initiative, match them to a group of students who did not participate but who shared many similar characteristics, and then, compare the two - providing a matched comparison for analysis. We analyzed more than 500 student pairs - half of whom used the app in the spring term, and half of whom did not. The result was a 13.5 percentage point overall lift in persistence for students who used CircleIn. This is one of the highest lifts in persistence of all initiatives we have analyzed in the last three years. The analysis also revealed that the CircleIn was even more impactful for Black/African-American students, part-time students, and students with the lowest probability to persist. 

We had similar success in the summer 2020 term will be completing a more detailed analysis in early 2021 on the fall 2020 term.