- REL northwest
- ask REL
Question
"What Does the Study Say About the Usefulness of Renaissance Star Assessments and Their Use in Guiding Teachers' Educational Practices?"
Ask REL
Thank you for submitting a request to our Regional Educational Laboratory (REL). Ask A REL is a collaborative reference point service provided by 10 REL, designed to function similarly to a technical reference library. Ask A REL provides references, references, and short answer citations in response to questions about available educational research.
Following the research protocol established by REL Northwest, we conducted a search for evidence-based studies. Sources included ERIC and other federally funded databases and organizations, research institutions, academic research databases, Google Scholar, and general Internet search engines. For more information, see the methods section at the end of this document.
The research team did not assess the quality of references and sources in this response; we offer them for informational purposes only. The search included the most commonly used research databases and search engines to generate the literature presented here. References are listed alphabetically, not necessarily in order of importance. Test references are not necessarily exhaustive and there may be other relevant test references. In addition to peer-reviewed, evidence-based scientific resources, we've included other resources you may find useful. We only provide publicly available sources unless such sources are lacking or the article is considered groundbreaking in a particular field.
Bibliography
Burns, MK, Kanive, R., & DeGrande, M. (2012). The impact of computer-assisted mathematics intervention as an additional mathematics intervention in the third and fourth grades. Compensatory and special education, 33(3), 184-191. downloaded fromhttps://www.researchgate.net
From the summary:
"This study reviewed a computer-based math fluency intervention conducted among 216 third- and fourth-grade students who were at risk for math difficulties. The intervention used a computer program to practice math facts three times a week, on average, for 8 to 8 years. The data were compared with the data obtained from 226 students of the control group. The results showed that students who participated in the intervention made significantly greater progress in math scores than those in the control group, and students with severe math problems (at the 15th percentile or lower) grew at the same rate as students whose scores were earlier tests were between the 15th and 25th percentile.Furthermore, after participating in the intervention, significantly fewer students in the intervention group were at risk of failing mathematics. the data suggest that the numeracy intervention was a useful additional math intervention. Suggestions for future research are given.
Bulut, O. and Cormier, DC (2018). Evidence of the importance of tracking your progress with Star Reading: slope estimates, feeding frequency, and number of data points. Frontiers in Education, 3(68), 1–12. downloaded fromhttps://www.frontiersin.org
From the summary:
"The increasing use of computerized adaptive testing (CAT) to gather information about students' academic progress or responses to academic interventions has led to many questions about the use of these measures to monitor progress. Star Reading is an example of a CAT-based assessment with substantial evidence of validity for its use in monitoring progress. However, additional validity evidence can be gathered to improve the use and interpretation of star reading data for progress monitoring. Therefore, the aim of this study was to focus on three aspects of progress monitoring that will benefit Star Reading users. Specific research questions to be answered are: (a) how robust estimation methods are in producing significant slopes for tracking progress in the presence of outliers; (b) how long should I use Star Reading for progress monitoring purposes and (c) how many data points should I use Star Reading for progress monitoring purposes? The first research question was addressed by a Monte Carlo simulation study. To answer the second and third research questions, the real data of 6,396,145 students who took the Reading the Stars test in the school year 2014-2015 were used. The results suggest that the Theil-Sen estimator is the most reliable estimator of student development when using star reading. Additionally, five data points and a progress monitoring period of approximately 20 weeks appear to be the minimum star reading parameters used for progress monitoring purposes. Implications for practice include adapting progress monitoring parameters to students' current grade-level reading performance.
Clemens, NH, Hagan-Burke, S., Luo, W., Cerda, C., Blakely, A., Frosch, J., ... and Jones, M. (2015). Predictive validity of computer-assessed reading skills in kindergarten and first grade. Journal of School Psychology, 44(1), 76-97. taken fromhttps://www.researchgate.net
From the summary:
"This study investigated the predictive validity of a computer-based adaptive assessment for measuring preschool reading skills using the STAR Test of Early Literacy (SEL). Results indicated that fall, winter, and spring preschool SEL scores were moderate and statistically significant predictors of end-of-year reading and literacy and explained 35% to 38% of the variance in the latent word reading ability variable. Similar results were observed in a subsample of 71 participants who underwent the following studies: End-of-kindergarten analyzes showed that, when added as predictors of SEL, paper-based measures of letter naming, pronunciation fluency, and word reading fluency improved the amount of explained variance in preschool and elementary school. Word literacy result at the end of the year. Analysis of scoring accuracy indicated that SEL literacy scores were consistent with written word description for students with higher SEL scores, but less so for students with lower SEL scores. Furthermore, shortened SEL scores showed problematic accuracy, particularly in predicting scores at the end of first grade. Adding paper notes tended to improve accuracy compared to using the SEL alone. Overall, SEL shows promise as a complete tool for preschool reading, although it may not yet be able to fully replace paper-based early reading assessments."
Clemens, NH, Hsiao, YY, Simmons, LE, Kwok, OM, Greene, EA, Soohoo, MM, ... and Otaiba, SA (2019). Predictive validity of kindergarten progress monitoring measures during the school year: Application of dominance analysis. Effective Intervention Evaluation, 44(4), 241-255.https://eric.ed.gov
From the summary:
"Although several measures are available to monitor preschool reading progress, few studies have directly compared them to determine which are better at predicting end-of-year reading ability compared to other measures and how validity may vary over time during the school year. as reading skills develop. A sample of 426 preschool children considered to be at risk for reading difficulties at the beginning of kindergarten was followed throughout the year using a series of reading monitoring measures. Paper-based progress and a computer-based adaptive test. was used to determine the degree to which each measure clearly predicted end-of-year reading ability compared to other measures.Although the computer adaptive test was the most dominant predictor of letter sound fluency, letter naming fluency, and phoneme segmentation fluency at the beginning of the year, letter fluency was the most dominant in Dec. Real word reading fluency measures recorded in the second half of the year dominated all other assessments. The consequences of the choice of measure are discussed.
Lambert, R., Algozzine, B., & McGee, J. (2014). The impact of progress monitoring on the mathematics achievement of at-risk students. Journal of Education, Society and Behavioral Sciences, 527-540. taken fromhttps://www.journaljesbs.com
From the summary:
“Objectives: In this study, we evaluated the effects of progress monitoring using a commercially available task adaptation and mathematics progress monitoring tool for elementary school students. Study Design: We used a randomized clinical trial and multilevel analysis to examine treatment effects on outcome measures when assigning students to their classrooms. Study Setting and Duration: The study, which spanned an entire school year, enrolled students from three elementary schools in the Midwest region of the United States. Methodology: We used two-level hierarchical linear models for our analysis due to the nested nature of our data. We compared results in high and low implementation fidelity treatment classes, as well as in treatment and control classes. Results: We found that the treatment was statistically significant. Differences in the monthly growth rate and fidelity to the effects of implementation in primary school were documented. Conclusion: Advancement professionals use a variety of metrics to track student performance and help make instructional decisions when data indicates a need for change. We found that using a self-paced computer-based math program and monitoring progress led to improvements in both curriculum-based and standardized assessments. The effects of system use were greater the higher the level of implementation (ie, intervention fidelity). The value of monitoring progress and the importance of measuring the relationship between implementation fidelity and achievement outcomes, which we believe is supported by previous research.”
McBride, JR, Ysseldyke, J., Milone, M., & Stickney, E. (2010). Technical adequacy and cost-effectiveness of four measures of early literacy. Canadian Journal of School Psychology, 25(2), 189-204. downloaded fromhttps://www.academia.edu
From the summary:
"Technical adequacy and information retrieval/cost were tested against four measures of early reading: Dynamic Indicators of Basic Early Reading Skills (DIBELS), Early Literacy STAR (SEL), Group Reading Assessment and Diagnostic Assessment (GRADE), and the Texas Elementary Reading Inventory ( TPRI). All four assessments were administered to the same students in each grade K through 2 over a 5-week period. The tests included 200 students from each grade in 7 countries. Both SEL and DIBELS were run twice to determine test-retest reliability in each class. We focused on the convergent validity of each test to measure the five key elements of reading development identified by the US National Research Council: phonemic awareness, phonics, vocabulary, comprehension, and fluency. Both DIBELS and TPRI are believed to assess all five components: GRADE and STAR Early Literacy explicitly measure all but fluency. For all components, correlations between the respective subtests were high and comparable. The cross-correlation pattern of fluency measures with fluency suggests that tests of fluency, vocabulary, word comprehension, and word reading measure the same underlying construct. A separate cost-effectiveness study was conducted, which concluded that STAR early literacy was the most cost-effective measure among respondents. In terms of the amount of time per unit of test administration or teacher time, CAT (computer-based adaptive testing) in general, and STAR Early Literacy in particular, are attractive options for early literacy assessment.
Monpas-Huber, JB and Marysville Public Schools (2015). Do you just push buttons? Validity evidence for the STAR and Smarter Balanced summative assessments. Educational Journal WERA, 8(1), 39-44. form downloadedhttps://jackbhuber.files.wordpress.com
From the summary:
“As students in America's public schools continue to take tests, educational evaluators who develop and use the tests will be encouraged to adhere to high standards of practice. For testing professionals working in public school districts, this means being very clear about the purpose of various assessments and then being able to describe and provide evidence of validity for those purposes. The article discusses challenges that undermine the validity of the STAR and Smarter Balanced ratings based on content and administrative irregularities. Correlations between test results were then examined for evidence of validity. The results show a strong correlation.”
Nelson, PM, Van Norman, ER, Klingbeil, DA, & Parker, DC (2017). Monitoring progress with computer-adjusted assessments: the impact of data collection time on growth estimates. School Psychology, 54(5), 463-471. downloaded fromhttps://www.researchgate.net
From the summary:
“While there is extensive research on the use of curriculum-based measures to monitor progress, little is known about the use of computer adaptive testing (CAT) for progress monitoring purposes. The aim of this study was to assess the effect of frequency of data collection on individual and group estimates of height using the CAT. Data are available for 278 fourth and fifth grade students. Growth estimates were obtained when five, three, and two sets of data were available over 18 weeks. Data were analyzed by grade to assess any observed differences in height. Mean squared error values were then obtained to assess differences in estimates of individual student development across data collection schedules. Group-level estimates of height did not differ between data collection schedules; however, individual student height estimates varied depending on the data collection schedule. The implications of using CAT to monitor student progress at the individual or group level are discussed.
Ochs S, Keller-Margulis MA, Santi KL and Jones JH (2020). Long-term validity and diagnostic validity of a computer-based adaptive reading test. Effective Intervention Evaluation, 45(3), 210-225. downloaded fromhttps://www.researchgate.net
From the summary:
"Universal screening is the primary mechanism by which students are identified as being at risk of failure in the context of a multi-level support system. This study investigated the diagnostic validity and accuracy of the Computerized Reading Adaptation Test as a screening tool to determine the achievement test scores of students in grades 3 through 5 (N = 1696). because the slope and condition test was poor or insignificant. Diagnostic accuracy cutoffs that maximized sensitivity and specificity resulted in high precision for single points, while sensitivity was poor for slopes. Practical implications and future directions are given.
method
Keywords and search strings:The following keywords, subject headings, and search strings were used to search reference databases and other sources: Star, Renaissance, (Rating? OR Screener?), (Manual OR How-to)
Databases and resources:ERIC we searched for relevant sources. ERIC is a free online library with more than 1.6 million citations of educational research, sponsored by the Instituto de Ciências da Educação (IES). Additionally, searches were conducted in Google Scholar and EBSCO (Academic Search Premier, Education Research Complete, and Professional Development Collection) databases.
Search criteria and reference selection
In the search and review of resources, we took into account the following criteria:
Date of publication:This research and review included references and sources published in the last 10 years.
Priorities for searching reference sources:Research priority is given to research reports, abstracts, and other papers published and/or peer-reviewed by IES and other federal or federally funded organizations, as well as academic databases, including ERIC databases, EBSCO, and Google Scholar.
Methodology:The following priorities/methodological considerations were taken into account in the literature review and selection:
- Types of research: randomized clinical trials, quasi-experiments, surveys, descriptive data analysis, literature reviews and policy guidelines, usually in that order
- Target population and samples: representativeness of the target population, sample size and whether participants volunteered or were randomly selected
- duration of studies
- Limitations and generalization of results and conclusions
This memo is part of a series of rapid responses to specific questions from stakeholders in Alaska, Idaho, Montana, Oregon, and Washington supported by the Northwest Regional Education Laboratory (REL). It was prepared under contract ED-IES-17-C-0009 REL Northwest, administered by Education Northwest. The content does not necessarily reflect the views or policies of IES or the US Department of Education, and mention of trade names, commercial products, or organizations does not imply endorsement by the US government.