Skip to main content
Header Logo Background Graphic Denver Preschool Program

Cohort 10 Report

Denver Preschool Program

Child Outcomes Study 2017 – 2018

Part A

Preschool Progress and Kindergarten Readiness Report,
Cohort 10

Document Summary

This report describes short-term outcomes for Cohort 10 of the Denver Preschool Program Child Outcomes Study. Specifically, the progress children made over the preschool year and kindergarten readiness. The Cohort is a stratified sample comprised of 240 randomly selected children who participated in DPP during the 2017-2018 school year. The outcomes focus on pre-academic, cognitive skills, and social-emotional development. Subgroup comparisons by primary language and income are included, in addition to details on the effects of high-quality preschool settings.

Clayton Early Learning Clayton Early Learning Research and Evaluation Department
Research Contacts:

Sheridan Green, Ph.D., Vice President, Research and Evaluation, sgreen@claytonearlylearning.org

Diana Mangels, M.A., Director, Research and Evaluation, dmangels@claytonearlylearning.org

Recommended Citation:

Green, S., Mangels, D. & Reale, M. (2018). Denver Preschool Program Child Outcomes Study 2017-2018: Part A:

Preschool Progress and School Readiness Report, Cohort 10. Denver, CO.

Executive Summary 2017-2018, Cohort 10

In 2017-2018, 240 preschool students participated in the Denver Preschool Program Child Outcomes Study. The students were directly assessed in the pre-academic domains of vocabulary, early literacy, and math skills in both the fall and spring semesters. Spanish-English dual language learners completed the direct assessments in both English and Spanish. Parents and teachers also completed surveys rating students’ social emotional development and executive function.

Do students make progress in their development while participating in DPP (i.e., vocabulary, early literacy, mathematics, social-emotional development and executive function)?

On average, preschool students enrolled in DPP developed at or above expectations over the course of the 2017-2018 school year. There were statistically significant increases from fall to spring in students’ standard scores on vocabulary and math assessments administered in English, meaning that students developed to a greater degree than expected based on their age. For vocabulary and math assessments administered in Spanish and early literacy assessments administered in either language, students developed at a steady rate that is typical for their age. Students also exhibited statistically significant improvements in social emotional development (including self-regulation, initiative, and attachment) and executive function (including working memory and inhibition skills) over the course of the year.

Do students from different income levels and with different primary languages make similar progress in their development while participating in DPP?

Students from different income levels progressed at similar rates from fall to spring on English and Spanish direct assessments. That is, there were no differences in students’ rates of progress by income level. However, differences were found in students’ developmental progress by primary language. Overall, English monolingual learners scored higher on English vocabulary, early literacy, and math than did dual language learners, but dual language learners showed greater improvements on the math assessment as compared to English monolingual learners. That is, dual language learners outpaced English monolingual learners on the math assessment in terms of growth, making developmental progress at a faster rate from fall to spring.

It is important to note that income level and primary language are strongly associated in this sample. Specifically, 90% of dual language learners come from the two lowest DPP-defined income tiers (equivalent to <100% and 100%-185% of the Federal Poverty Threshold) compared to only 37% of English monolingual learners. As a result, it is difficult to disentangle the effects of income and primary language on child outcomes, and thus the observed differences by primary language may be the result of the co-occurrence of these two factors

To what extent and in what areas are DPP students ready for kindergarten?

The vast majority of children met the kindergarten readiness benchmark of a standard score of 85 or higher (as defined by test publishers). Specifically, the number of students who demonstrated kindergarten readiness on direct assessments of vocabulary, early literacy, and math administered in spring 2018 ranged from 83 to 92%, with the exception of Spanish vocabulary (42%). Nationally, 84% of students are expected to score in the typical range (85 or above) on these assessments.

The likelihood of demonstrating kindergarten readiness on these assessments was strongly associated with children’s primary language. For instance, 96 and 83% of English monolingual learners scored 85 or above on English vocabulary and English literacy, respectively, compared to 50 and 54% of dual language learners. An even more pronounced pattern of differentiated results emerged when a score of 100 (average score as defined by test publishers) was used as the cutoff. Because language development is expected to progress at a different pace for children learning more than one language as compared to those learning only one language, the research team also examined children’s highest score on each assessment type in either English or Spanish as a measure of kindergarten readiness. The number of children who demonstrated kindergarten readiness on each assessment type in at least one language was consistently high across assessments, with 88% of children scoring 85 or higher in vocabulary, 86% in early literacy, and 88% in math.

A large majority of children also exhibited kindergarten readiness in social emotional development and executive function, as rated by parents and teachers. Similar to the direct assessments, nationally 84% of children are expected to be rated in the “typical” or “no concern” ranges of scores. In Cohort 10, parents and teachers rated 87 to 96% of students as demonstrating kindergarten readiness in the spring in social emotional domains (Total Protective Factors and Behavioral Concerns). For executive function, 88% of children were rated in the adaptive range for Working Memory (e.g., remembering instructions, remembering several things asked to do) by parents compared to 80 % rated by teachers. Parents rated 81% and teachers rated 82% of children in the adaptive range for Inhibition (e.g., thinking before acting, being able to stop an activity when asked).

Do classroom environments make a difference in progress and kindergarten readiness for DPP students?

While it is known from the research literature that classroom quality is essential to promoting positive outcomes for children, it is difficult to examine the associations between classroom quality and child outcomes in this sample because the majority of programs have moderate to high quality ratings. These consistently high ratings provide insufficient variability to statistically explore the relationships between classroom quality and child outcomes. Thus, no significant associations were observed between any of the child outcome measures and the CLASS® and Colorado Shines quality ratings of DPP programs.

Overall, Denver Preschool Program students in Cohort 10 were ready for kindergarten in preacademic, social-emotional development, and executive function domains. The majority of dual language learners demonstrated kindergarten readiness in at least one language and showed progress over the school year in both English and Spanish.

Introduction

The Denver Preschool Program (DPP) is a taxpayer-funded initiative, created in 2006 and reauthorized by voters in 2014, that champions, funds, and increases access to high quality preschools. DPP supports every family in Denver with a 4-year-old by offering tuition credits to access a high-quality preschool of their choosing and gives approved providers access to quality improvement resources. The vision of DPP is that children in Denver enter kindergarten ready to reach their full potential.

The Clayton Early Learning Research and Evaluation Department conducts a cross-sequential study, which gathers year-of and longitudinal data for each DPP cohort. The study started during the 2008-2009 school year and has continued every year, uninterrupted. The study helps DPP understand student progress during preschool, kindergarten readiness, and the impact of receiving tuition credits on academic outcomes through the end of high school. During the 2017-2018 school year, the Clayton research team welcomed Cohort 10 and Cohort 1 was expected to be enrolled in eighth grade (see Table 1).

This report focuses on the short-term outcomes; progress made during the preschool year and kindergarten readiness, in the areas of vocabulary, early literacy, math, and social-emotional and executive functioning. A companion report, Denver Preschool Program Child Outcome Evaluation 2017–2018: Part B: Longitudinal Follow-up, Evaluation Cohorts 1-9 Elementary Report, describes the long-term outcomes.

Table 1. DPP Evaluation Cohorts and Expected Grade Levels by School Year

Short-Term Child Outcome Questions

  1. Do students make progress in their development while participating in DPP (i.e., vocabulary, early literacy, mathematics, social-emotional development and executive function)?
  2. Do students from different income levels and with different primary languages make similar progress in their development while participating in DPP?
  3. To what extent and in what areas are DPP students ready for kindergarten?
  4. Do classroom environments make a difference in progress and kindergarten readiness for DPP students?

Data Collection and Measurement

The study involves a tremendous amount of coordination, partnerships, and collaboration to collect direct child assessments, parent and teacher surveys, and classroom observations. At the beginning of the school year, the research team stratifies students from the pool of all enrollees and randomly selects a sample from the strata to ensure representativeness. Parents of selected students are contacted by the research team and are invited to participate in the study. The researchers also invite lead teachers of selected students to participate in providing information on the student and the classroom environment. Parents and teachers are compensated for their time with a gift card. Parents also receive a brief report on the results of their children’s assessments.

Direct Child Assessments

Because of the rapid growth that happens during the first five years of life and the group atmosphere of the classroom, it is often difficult to know how an individual child is developing without a direct child assessment. The research team uses standardized direct assessments to assess vocabulary, early literacy, and math abilities for Standard English and Spanish languages (see Table 2).

Table 2. Direct Child Assessments

Students take part in direct child assessments in the fall and again in the spring. Highly trained research assistants (assessors) conduct the child assessments. The assessments are conducted at the student’s school in an area designated by the preschool staff. The student and assessor generally sit in a quiet space at a child-size table outside the classroom or main area of care in provider homes. The direct assessments are spaced about four to five months apart (Cohort 10: M = 4.1; SD = .26). Direct child assessments are conducted in English for all children regardless of their primary language. Spanish-English dual language learners receive additional assessments in Spanish at both time points.

Standard scores based on a nationally normed sample, are used for every direct child assessment. Scores are based on a representative sample for each age and can range from 0 to 160, with an average score of 100 and standard deviation of 15. Standard scores are helpful for interpreting changes in scores exhibited across the year because these scores are based on age. Scores that are steady from fall to spring indicate that the student had a steady rate in development. However, if a student learned more than what was expected due to typical development, a change in scores from fall to spring would reflect accelerated development.

The standard score is also useful for understanding readiness in the spring before students enter kindergarten. A student who has a score of 100 is meeting the expectations for their age. A student’s proficiency level for their age can be determined based on the standard deviation from the score of 100. In this study, five categories of readiness expectations were derived from student’s scores on vocabulary, early literacy, and math: Lagging, Approaching, Meeting, Exceeding, and Excelling (see Table 3). What these categories mean in terms of children’s readiness is that students with scores in the lagging or approaching category may need additional support in kindergarten to “catch up and keep up” with their peers with scores categorized as “meeting, exceeding, or excelling”. Students who are meeting, exceeding, or excelling in terms of expectations (based on national averages) are primed for learning and need generally less support to engage in academic instruction.

Table 3. Readiness Categories Indicating Kindergarten Readiness Expectations Level

Parent and Teacher Surveys

Social-emotional development and thinking and memory skills require a more in-depth and background knowledge of the child that is hard to capture in a one-time direct child assessment. For these domains (see Table 4), the researchers rely on surveys from important adults in the child’s life: parents and teachers. Parents and teachers fill out child surveys during the fall and spring around the same time that the direct child assessments occur.

T-scores are computed from the Devereux Early Childhood Assessment (DECA) parent and teacher surveys. An average T-score is 50 and has a standard deviation of 10. The interpretation of the scores still allows comparison of growth over time and an examination of readiness benchmarks similar to the standard scores in the direct child assessments. Composite scores are used for the CHEXI subscales measure computed from parent and teacher surveys.

Table 4. Teacher and Parent Surveys

Classroom Observations

Highly trained and reliable observers conduct classroom observations in classroom or homes that hold a DPP student included in the study. In certain circumstances, the research team uses secondary data or retrospective (same school year) scores for the classroom observation data (see Table 5). Teachers also fill out information about their classroom, curriculum, and teaching credentials.

Table 5. Classroom Quality Observation Tools

Cohort 10 Snapshot

Cohort 10 was comprised of 240 DPP students who participated in the DPP child outcomes study. Sample children averaged 4.5 years of age in the fall of the last year before kindergarten, October 2017, and were 53% female. Full sample demographics are provided in Appendix A. Forty-two percent of the students (n=101) attended community preschools and the remaining (n=139) attended Denver Public Schools (DPS) pre-kindergarten programs.

The majority of DPP students were enrolled in full-day preschool at 68%. This is akin to a child attending at least five hours for at least 25 hours a week. The second-highest participation rate was extended day at 20%. Extended day preschool in DPP is categorized similarly to a full work day, at least eight hours a day and 33 hours per week. Lastly, 12% were enrolled in part-day preschool, which is typically a child attending for at least 2.5 hours for a minimum of five hours per week.

The sample was distributed across 111 DPP preschool providers and 175 teachers. Data were collected for 234 children in the fall and 231 in the spring. Nine students (4%) who were assessed in the fall dropped out before the spring data collection, typically because they moved out of the DPP service area. Data were collected for the 225 returning from fall, and an additional six (6) new participants in the spring to make up for some of the attrition (see Table 6.)

Table 6. Sample Sizes by Data Collection Type, Fall 2017 and Spring 2018

To maximize the conclusions that can be drawn about both community DPP sites and those sites in Denver Public Schools (DPS), the researchers stratified the sample by type of provider. The result was two samples: a sample of students in community preschools and a sample of students in DPS preschools. Both of these samples were representative of the population of children in each type of preschool at the time of sampling. For all analyses on the sample as a whole, sampling weights were applied so that the results would be representative of the population of students enrolled in DPP over the course of the full school year. For analyses comparing DPS and community preschools, weights were not applied.

Preschool Progress – Cohort 10 Full Sample

In general, the majority of students scored 85 or above (approaching to excelling expectations categories) on English child assessments in both the fall and spring. The same was true for Spanish child assessments except for the Spanish vocabulary. The majority of students are in the normative range on the social emotional and executive function assessments. It is noteworthy that for all child assessments, there is considerable variability in student’s scores, with some individual students scoring quite low and some scoring rather high. Weighted descriptive statistics and ranges are displayed in Table 7 and Table 8.

Table 7. 2017-2018 Weighted Descriptive Statistics for the Directly Assessed Child Outcome Measures

Table 8. 2017-2018 Weighted Descriptive Statistics for Parent and Teacher Surveys

Do students make progress in their development while participating in DPP (i.e., vocabulary, early literacy, mathematics, social-emotional development and executive function)?

Paired t-test analyses revealed that students had steady scores across the year for English early literacy. This indicated that students were developing at a steady rate that is typical for their age. A statistically significant increase was found for scores for vocabulary and math. This increase represents almost four points in vocabulary and about two points for math skills assessed in English (small effects).7 This suggests that children learned vocabulary and math skills in English, at a faster rate than what is expected with typical maturation.

Analyses for Spanish assessments showed steady scores for vocabulary, early literacy, and math. This means that Spanish-English dual language learners learned at a rate expected due to naturally growing older. The results are displayed in Table 9.

Table 9. 2017-2018 Preschool Gains in Vocabulary, Early Literacy, and Math (Weighted)

Tables 10 and 11 display results for the examination of changes from fall to spring in the social-emotional and executive function ratings by parents and teachers. Paired t-tests on composite ratings of student social-emotional development (protective factors and behavioral concerns) and for the CHEXI composites of executive function (working memory and inhibition).

Social-Emotional Development. When rated by parents, no statistically significant changes were observed for children’s Total Protective Factors nor for Behavior Concerns. One subscale of the Total Protective Factors composite, Self-Regulation, did show a statistically significant improvement (2.6 point increase, p < .001). A statistically significant improvement was observed among the teacher ratings of student Total Protective Factors from fall to spring 8, while no significant improvement in scores was found for Behavioral Concerns as rated by teachers (Table 10).

Table 10. 2017-2018 Preschool Gains in Social Emotional Development9

Executive Function. Table 11 shows the results of fall to spring comparisons for both teacher and parent ratings of children’s executive function. Parent ratings significantly improved from fall to spring for students’ Working Memory.10 However, parent scores did not change over time on the Inhibition scale. Significant improvements for children were observed on both teacher-rated Working Memory and Inhibition over the course of the school year.11

Table 11. 2017-2018 Preschool Gains in Executive Function12

Preschool Progress – Group Comparisons

Subgroup comparisons on student progress on English standardized assessment scores are of great interest because they support an understanding of how DPP participation may be more or less effective for different students. Group comparisons related to Community or DPS program, gender, race/ethnicity, primary language, and income tier have been examined over the years. This year, comparisons of students by DPS and Community sites reveal no differences in progress on vocabulary, early literacy and math.13 Comparisons of students by gender also showed no differences in progress on vocabulary, early literacy, and math.14 Examining the assessments by ethnicity yielded no differences in the rate of progress for vocabulary, early literacy, and math. Historically, it has been found that students’ primary language and income have shown the largest progress differences among the group comparison, thus these analyses have been highlighted below.

Do students from different income levels and with different primary languages make similar progress in their development while participating in DPP?

Overall, in this sample of students, those from different income levels progressed at similar rates across the year on English and Spanish direct assessments. That is, this year, there were no differences in students’ rate of progress by income level. A different rate of progress was found, however, between English-monolingual and dual language learners. Dual language learners outpaced English-monolingual learners on the Math assessment, developing skills at a faster rate from fall to spring (see Chart 2).

Prior to conducting these analyses, the degree to which income tier and students’ primary language were associated was assessed to support understanding and interpretation of the comparative findings. For income tier, some data reduction/combination was necessary since the number of participants from some of the income tiers was rather small. DPP’s defined income tiers were collapsed into a new income tier group variable with four categories: Tier 1, Tier 2, Tiers 3-5 and Tier 6 (i.e., parents who opted out of reporting income and were automatically assigned the lowest tuition credit amount).15

It is important to note that these two background characteristics, income tier and student’s primary language, were strongly associated (see Chart 1).16 This year, almost 90% of dual language learners were from Tiers 1 or 2, whereas about 37% of the students whose primary language was English were from Tiers 3-6. As a result, in this sample, it was difficult to disentangle the effects of income and primary language and any effects observed are possibly the result of the combination of these two factors.

Chart 1. Income Tier Groups by Student Primary Language

Change over Time by Income Tier

Analyses were conducted to examine gains over time on assessments administered in English and Spanish for each income tier group. For Cohort 10, no significant income tier by time interactions were found for any of the assessments administered in English nor in Spanish. The non-significant income tier by time interactions indicates that children progressed similarly in these areas over the course of their preschool year, regardless of their income tier. The proportions of Cohort 10 children across income are reported in Table 12.

Table 12. 2017-2018 Proportion of Sample by Income Tier

Change from Fall to Spring by Student Primary Language

Since all students were assessed in English, regardless of their primary language, it is useful to consider whether students’ scores on the English assessments differed based on whether students spoke English as their primary language or whether they were dual language learners. The Clayton research team performed t-tests to examine whether there were differences in mean scores on the English direct child assessments by primary language group (i.e., English-monolingual versus any other language or language combination).17 Results for Fall 2017 are presented in Table 13. In the fall and spring, there were statistically significant group differences in vocabulary18, early literacy19 and math20 assessment scores between English-monolingual and dual language learners.

As a whole, dual language learners consistently scored lower on the English vocabulary, early literacy, and math assessments than English-monolingual students. In the fall, dual language learners, on average, scored about 28 points lower than English-monolingual students on the English receptive vocabulary assessment. English-monolingual students score 7.8 points higher on average than dual language learners on early literacy, and 15.3 points higher on math (note that one standard deviation is 15 points on the assessment). A similar pattern of findings was observed in the spring round (Table 14), though the difference is not as pronounced, indicating the dual language learners are catching up somewhat.

Table 13. Weighted English Standardized Assessment Scores by Student Primary Language, Fall 2017 Round

Table 14. Weighted English Standardized Assessment Scores by Student Primary Language, Spring 2018 Round

Analyses were conducted with primary language predicting scores over time on assessments administered in English.21 No significant interactions were found between primary language group and time for vocabulary nor early literacy, meaning that English-monolingual and dual language learners progressed at the same rate across the year in vocabulary and early literacy. However, for math, there was a significant language by time interaction.22 In this case, dual language learners were “catching up” or learning at a faster rate than English-only peers during the school year.

Chart 2. Change in Math Assessment Scores over Time by Student Primary Language (Assessed in English)

Cohort 10 Kindergarten Readiness

Analyses were conducted to determine how ready for kindergarten DPP students in Cohort 10 appeared to be at the end of their last preschool year before kindergarten. Readiness was examined in several ways. First, analyses were conducted to examine whether students scored in the typical range as defined by the test publishers, namely a standard score of 85 or above. For reference, a standard score below 85 indicates being in the risk range for the assessment. While not being at risk when entering kindergarten is important, it is also useful to examine whether students meet a higher standard, defined as scoring at or above 100, the national population mean, on the direct assessments used in the study. Chart 3 presents the percentage of students scoring 85 or above and, then, of that group, those who score 100 or above, on each of the assessments in English and in Spanish at the spring time point. In the general population, one would expect about 84% of children to score above 85 and 50% of children to score above 100.

Chart 3. Weighted Percentage of Students Scoring in the Average Range or Above on All Spring Direct Standardized Assessments 2017-2018*

Further analyses were conducted to examine proportions of students ready by spring. Using the same national standards on the English-administered assessments (i.e., first three bar sets in Chart 3), but additionally took into account whether their primary language was English only or whether they were dual language learners. Students in the dual language learner group included those additionally learning Spanish, Mandarin, Khmer, Arabic, Amharic, Bulgarian, and other languages. These findings are presented in Chart 4 (English-monolingual learners) and Chart 5 (dual language learners).

Chart 4. Weighted Percentage of Students Scoring in the Average Range or Above on Spring Standardized Assessments Administered in English for English-monolingual Learners 2017-2018

Chart 5. Weighted Percentage of Children Scoring in the Average Range or Above on Spring Standardized Assessments Administered in English for Dual Language Learners 2017-2018

It is hypothesized that the study’s dual language learners have a different developmental trajectory than children who are learning only one language. Analyses revealed that the likelihood of scoring 85 or above on these spring assessments was strongly associated with children’s primary language. About 96% of students whose primary language was English scored 85 or above on the English vocabulary assessment as compared with 50% of children whose primary language was not English.23 Over 80% of students whose primary language was English scored 85 or above on English literacy as compared to 54% of students whose primary language was not English.24 In Cohort 10, the proportion of students scoring 85 or above on math assessed in English was similar for English-monolingual (88%) and dual language learners (83%).25

A more pronounced pattern of differentiated results emerged when a score of 100 was used as the cutoff. On English vocabulary, for instance, 81% of English-monolingual students earned a score of 100 or greater as compared to 26% of dual language learners.26 For English early literacy, 37% of English-monolingual students scored 100 or greater as compared with 15% of dual language learners.27 Finally, for math assessed in English, 57% of English-monolingual learners earned scores of 100 or above compared with 21% of students who were learning more than one language.28

Kindergarten Readiness on Standardized Assessments in Either Language

All of these assessments were normed with monolingual children who were learning only one language. Language development for children learning two or more languages is expected to progress at a different pace than for children learning one language. One way to address the instrumentation limitations in understanding the progress of dual language learners is to look at dual language learner’s scores in both languages.

In Chart 6, the students’ highest scores in either English or Spanish were used as the measure of kindergarten readiness for each standardized assessment. Students who were Spanish-English dual language learners contributed their highest score in either the Spanish or English assessment. Students who were only assessed in English had only one score for each domain to use for their highest score.

Thus, in at least one language of administration, 82% of Cohort 10 students met or exceeded expectations in vocabulary. Seventy-one percent of children met or exceeded expectations in early literacy and 74% met or exceeded expectations in math. Showing all of these benchmarks and measures all together provides an overview of students’ school readiness at different levels. As discussed in the data collection and measurement section of this report, these categories have practical implications in terms of children’s readiness. Students with scores in lagging or approaching categories may need additional support in kindergarten to “catch up and keep up” to their peers with higher categories of scores. Students who are meeting, exceeding, or excelling in expectations are primed for learning and need generally less support to engage in academic instruction.

Chart 6. Proportion by Spring Proficiency Level Best Score Assessed in either English or Spanish 2017-2018 (Cohort 10)

Kindergarten Readiness – Social Emotional Development

The DECA parent and teacher survey was used to examine social-emotional development. Readiness is defined as scoring in the “Typical” or “Strength” categories as indicated by the publisher. As stated earlier, the DECA is scaled using T-scores, which have a mean of 50 and a standard deviation of 10. For Protective Factors, children with T-scores greater than 40 fall into these categories. For Behavioral Concerns, higher scores indicate greater levels of behavioral concerns, so children with T-scores below 60 are considered in the “Typical” range. In the general population, one would expect about 84% of children to fall within these ranges.29

Fall parent ratings of students’ Total Protective Factors (a combination of the Initiative, Self-Regulation and Attachment subscales) showed that 84% were within or exceeded the expected range and this proportion increased by spring. Slightly fewer students than expected in fall were rated in the typical range for Behavioral Concerns rated by parents (82% versus the 84% benchmark). By spring, however, parents rated more students than expected in the typical or strength range for Initiative (89%), Self-regulation (94%), Attachment (88%), and Total Protective Factors (91%; see Chart 7). Additionally, more students than expected were in the typical range for Behavioral Concerns (87%). Refer to detailed findings in Table 10 for change from fall to spring.

Teachers also rated a majority of students in the typical or strength range in both time points for Total Protective Factors and on Behavioral Concerns (i.e., over the expected 84%). In the spring, teachers rated students (as a group) slightly higher than the national average of 50 on all of the subscales, with the exception behavioral concerns (indicating less concern than expected); however, there was substantial variability in all of the scores (see ranges listed in Table 10).

Chart 7. Weighted Percentage of Students Scoring in the Average Range or Above on Spring Parent and Teacher Social Emotional Development Surveys 2017-2018

Kindergarten Readiness – Executive Function

The Childhood Executive Function Inventory (CHEXI; appropriate for ages 4-12), was used to measure an important kindergarten readiness indicator related to students’ thinking and memory skills. This was measured, like the other assessment, at two time points this year: fall and spring of Pre-K. The CHEXI yields four subscale scores (see Table 4) that combine into two executive function factor scores: Working Memory and Inhibition. This is a relatively new instrument, and it has been validated with a number of cross-cultural samples.

On the CHEXI, possible scores can range from 13 to 65 on the Working Memory factor and from 11 to 55 on the Inhibition factor. Lower scores indicate greater levels of executive function (desired levels) in each of these areas. Cutoff scores higher than 34 for Working Memory and higher than 32 for Inhibition indicate more concerning levels of executive functioning.

Chart 8 shows the proportion of students rated by parents and teachers, respectively, who were categorized into the adaptive executive function range at both time points. The proportions increased from fall to spring, and reflected the findings in Table 11. High levels of students are in the adaptively ready range with regard to their working memory and inhibition.

Chart 8. Percentage of Students in the Adaptive Executive Function Range on Spring Teacher and Parent Executive Function Surveys, Weighted, 2017-2018

Kindergarten Readiness – Effects of Participation Type

DPP students take part in one of three options of participation for the school year: part-day, full-day, and extended day. A comparative analysis of all student kindergarten readiness outcomes (an ANCOVA, controlling for income tier) revealed no statistically significant differences in pre-academic, socialemotional, nor executive function outcomes based on students’ participation type. The mean differences and effect sizes were not meaningfully different (ranged from .003-very small to .042-small).

Cohort 10 Preschool Quality in 2017-2018

Preschool Quality

The 240 students in the sample were enrolled in 111 different preschools. Information regarding quality of these preschools was gleaned from two sources: a) the Colorado Shines Quality Rating Improvement System (adopted in January 2015; Colorado Shines QRIS) and b) classroom observations using the Pre-K CLASS® (Classroom Assessment Scoring System) tool that were conducted specifically for this evaluation project combined with observations that were conducted during the same program year for the ongoing DPP CLASS® ratings requirements.

Colorado Shines Rating

Within the sample, data were available from the Colorado Department of Education for all program sites except two (n=109). These data represent the level quality program from classrooms for 236 of 240 of this DPP student sample. Chart 9 presents the array of programs by quality level. Nearly 10% of programs were rated at a Level 3. Nearly 74% of programs were rated Level 4. About 11% of programs were rated as Level 5. Only two preschools were rated at a Level 1 designation (licensed) and 10 preschools earned a Level 2, indicating that very few programs were of the lowest quality.

Early learning programs are rated through Colorado Shines on a scale of 1 to 5: 30

  • Level 1: Program currently licensed with the State of Colorado.
  • Level 2: Program is licensed and in good standing, plus:
    • has a quality improvement plan in place
    • has conducted the Level 2 Quality Indicator Program Assessment
    • has registered staff in the Colorado Shines Professional Development Information System (PDIS)
    • has completed Colorado Shines Level 2 E-learning Courses
  • Levels 3-5: Program is licensed and in good standing, plus:
    • has completed the Level 2 requirements
    • has been assessed and rated by a Colorado Shines Quality Ratings Assessor based on points in five categories (workforce qualifications, family partnerships, administration, learning environment, child health)

Chart 9. Colorado Shines Ratings for Classrooms with DPP Study Participants

Analyses were conducted to test whether the type of provider (DPS vs. Community) was associated with the level of Colorado Shines rating. The two types of programs differed with mean rating levels slightly higher for DPS programs (4.07 versus 3.67).31

Class Observations

Chart 10 displays the mean scores for the 80 classrooms (representing the 109 students) that were observed using the CLASS® Observation. The Pre-K CLASS® tool examines three domains of teacher-child interaction quality, each with a total possible score of 7.

Domain Descriptions:

  1. Emotional Support evaluates relationship aspects including positive climate, negative climate, teacher sensitivity, and regard for student perspectives.
  2. Classroom Organization assesses the interactions in terms of behavior management, productivity, and instructional learning formats used by the teacher.
  3. Instructional Support examines concept development, quality of feedback, and language modeling.

Studies using the Pre-K CLASS® conducted by the Office of Head Start (OHS) have consistently shown average preschool classroom scores are higher in the domains of Emotional Support and Classroom Organization than in the domain of Instructional Support.32 During the 2016-2017 program year, on-site reviews of a national sample of Head Start classrooms found mean scores of 6.07 for Emotional Support, 5.83 for Classroom Organization, and 3.00 for Instructional Support.33

On average, scores for DPP classrooms in the domains of Emotional Support and Classroom Organization were high, while scores for Instructional Support were near the bottom of the middle-range of the total possible score of 7. Average scores for Emotional Support and Classroom Organization were slightly higher than average scores from previous large studies (and slightly higher for this Cohort than for the previous year’s classrooms). As described above, in previous large studies using this observation tool, average scores for Emotional Support tended to be in the 5.55-6.5 range and average scores for Classroom Organization tended to be in the 4.5-6.0 point range. Scores for Instructional Support were similar to what has been observed in previous large studies, which have been in the 2-3 range.

Chart 10. Average CLASS® Domain Scores (n=80 Classrooms)

Charts 11, 12, and 13 provide information about the variability in these domain scores. For Emotional Support, all classrooms scored in the high range (scores above 5). For Classroom Organization, too, all classrooms scored in the high range. For Instructional Support, a little over half of the classrooms scored in the low range, and 46% scored in the middle range, and no classrooms scored in the high range.

Chart 11. Distribution of Scores for Emotional Support (n=80 Classrooms)

Chart 12. Distribution of Scores for Classroom Organization (n=80 Classrooms)

Chart 13. Distribution of Scores for Instructional Support (n=80 Classrooms)

Analyses were conducted to test for differences in CLASS ® domain scores by provider type. The results of these analyses are presented in Chart 12. Scores for all CLASS® subscales were not statistically different, on average, for DPS classrooms compared with community-based preschool classrooms.34

Chart 14. CLASS® Domain Scores by Provider Type

Does Quality Impact Child Outcomes?

For this analysis, the association between CLASS® Observation and Colorado Shines rating data (proxies for preschool quality) and child outcomes was examined. As a preliminary analysis, Clayton Researchers looked at all of the CLASS® dimension-level indicators and did not find any correlations with student outcomes. Because DPP strives for high quality, there is little variation in the quality levels observable with these tools, as is evident from the section above. This makes it difficult to address this important question. Since there was no variability in the Emotional Support domain (see Chart 9), and Classroom Organization (Chart 10) for the CLASS®, only the Instructional Support domain was used in the main analysis (acknowledging that the range of scores was also limited for this domain; on the seven-point scale the full sample scores ranged from 1.33 to 4.58 and the range was further restricted for the subsample of Spanish assessments – 2.00 to 4.41). To examine this association between Instructional Support and child outcomes, partial correlations were computed between all spring assessment scores and CLASS® Instructional Support scores (controlling for the fall assessment scores). No significant associations were observed for any of the spring assessments in English or in Spanish, nor for teacher or parent ratings of the DECA and CHEXI and the CLASS Instructional Support Domain.

An analysis of the Colorado Shines rating levels with spring assessment scores was conducted in the same fashion. No significant associations were observed between this estimation of quality and child assessment results in English. Since analyses revealed that variability in Shines ratings was limited among dual language learners (88% of the sample attended schools with a rating of 4), the associations between the ratings and children’s outcomes on the Spanish measures were not examined.

Conclusion

Overall, Denver Preschool Program students in Cohort 10 were ready for kindergarten in pre-academic, social-emotional development, and executive function domains. In at least one language of administration, 82% of Cohort 10 students met or exceeded expectations in vocabulary. Seventy-one percent of children met or exceeded expectations in early literacy and 74% met or exceeded expectations in math. The evaluation findings consistently show year-to-year that students benefit from participation in high-quality preschool and that dual language learners experience growth albeit on a slightly different developmental trajectory that monolingual English language learners. The majority of dual language learners demonstrated kindergarten readiness in at least one language and showed progress over the school year in both English and Spanish.

Social-emotional development and executive function are valuable components of school readiness and promote a greater understanding of the whole child and the learning approaches DPP students take into kindergarten. The results indicated that students fare well with regard to initiative, attachment, self-regulation, behavior concerns, working memory, and inhibition – all of which are critical elements in growth and learning.

An opportunity exists for the Denver Preschool Program with regard to communicating and partnering with school districts around the aggregate findings and descriptions of students who may be likely to benefit from additional support. It is recommended that findings from the many years of study are used to forge new ways of working with kindergarten teachers.

Appendix A. Sample Characteristics – Spring 2018

Supporting Documentation

  1. Dunn, L. M., & Dunn, D. M. (2007). PPVT-4: Peabody picture vocabulary test. London, UK: Pearson Assessments
  2. Schrank, F. A., McGrew, K. S., Mather, N., Wendling, B. J., & LaForte, E. M. (2014). Woodcock-Johnson IV tests of achievement: Form A. Itasca, IL: Riverside Publishing.
  3. Muñoz-Sandoval, A. F., Woodcock, R. W., McGrew, K. S., & Mather, N. (2005). Batería III Woodcock-Muñoz: Pruebas de aprovechamiento. Itasca, IL: Riverside Publishing.
  4. LeBuffe, P.A. & Naglieri, J.A. (2012). Devereux early childhood assessment for preschoolers, second edition. Lewisville, NC: Kaplan Early Learning Company.
  5. CHEXI – Thorell, L. B., & Nyberg, L. (2008). The childhood executive functioning inventory (CHEXI): A new rating instrument for parents and teachers. Developmental Neuropsychology, 33(4), 536-552.
  6. Pianta, R. C., La Paro, K. M., & Hamre, B. K. (2008). Classroom assessment scoring system (CLASS®) manual, pre-K. Baltimore, MD: Paul H. Brookes Pub. Co.
  7. Cohen’s d effect size guidelines: .20-.40 small effect; .41-.70 medium effect; >0.7 large effect. Vocabulary: Cohen’s d = .19 (small effect); Math: Cohen’s d = .13 (small effect)
  8. Teacher rating of Total Protective Factors: Cohen’s d = .43 (medium effect)
  9. Some teachers and parents left items blank on the DECA. Scores were calculated if at least 75% of the items were present. This resulted in some missing data for the DECA.
  10. Parent rating of Working Memory: Cohen’s d = .20 (small effect)
  11. Teacher rating of Working Memory: Cohen’s d = .27 (small effect) and Inhibition: Cohen’s d = .15 (small effect)
  12. Some teachers and parents left items blank on the CHEXI. Scores were calculated if at least 75% of the items were present. This resulted in some missing data for the CHEXI
  13. F(1, 223)=.04, p=.84; F(1, 224)=2.56, p=.11; F(1, 221)=.601, p=.43.
  14. F(1, 223)=.006, p=.93; F(1, 224)=1.16, p=.28; F(1, 221)=.124, p=.73.
    F(1, 220)=.26, p=.91; F(1, 221)=.33, p=.86; F(1, 218)=.1.84, p=.12.
  15. For analyses of assessments administered in Spanish, a two-level income tier group variable was used omitting the categories ‘tiers 3-5’ and ‘tier 6’ because only three child assessed in Spanish fell into tiers 3-5 and one children assessed in Spanish fell into tier 6.
  16. x25 = 25.43, p.<.0001
  17. It is important to carefully examine sample sizes when considering group differences by language groups. In these analyses, we have used Hedges’ g as the appropriate effect size measure. Hedges’ g provides an appropriate alternative to Cohen’s d has a measure of effect size weighted according to the relative size of each sample. Hedge’s g and Cohen’s d effect sizes are interpreted in a similar way with the following guidelines: .20-.40 small effect; .41-.70 medium effect; >0.7 large effect.
  18. Vocabulary: Hedges’ g = 1.67 (large effect)
  19. Early Literacy: Hedges’ g = .60 (medium effect)
  20. Math: Hedges’ g = 1.09 (large effect)
  21. It does not make sense to conduct this set of analyses for assessments administered in Spanish, since there is not adequate variability in children’s primary language among children assessed in Spanish.
  22. F(1, 221)=8.37, p=.004.
  23. x21 = 69.62, p<.0001
  24. x21 = 20.40, p<.0001
  25. x21 = .792, p = n.s.
  26. x21 = 61.05, p<.0001
  27. x21 = 10.84, p=001
  28. x21 = 23.02, p<.0001
  29. LeBuffe, P.A., & Naglieri, J.A. (1999). Technical manual for the Devereux Early Childhood Assessment (DECA). Villanova, PA: Devereux Foundation
  30. Colorado Departments of Human Services and Education. Retrieved from http://coloradoshines.force.com/ColoradoShines/programs?p=Your-Program-Colorado-Shines
  31. t=4.72, df=234, p<.001
  32. Office of Head Start, National Center on Early Childhood Development, Teaching and Learning. CLASS® brief: Understanding and using CLASS® for program improvement. https://eclkc.ohs.acf.hhs.gov/sites/default/files/pdf/class-brief-understanding-using-class-program-improvement.pdf
  33. 3 Office of Head Start, National Center on Early Childhood Development, Teaching and Learning. A national overview of grantee CLASS® Scores in 2017. Retrieved from https://eclkc.ohs.acf.hhs.gov/data-ongoingmonitoring/article/national-overview-grantee-class-scores-2017
  34. Emotional Support—t=.016, df=78, p=n.s.; Classroom Organization—t=.852, df=78, p=n.s.; Instructional Support— t=.852, df=78, p=n.s.