EXTERNAL VALIDITY OF A READING INTERVENTION FOR PRIMARY EDUCATION AS SHOWN IN DOMINICAN SIXTH GRADE STUDENTS PERFORMANCE ON THE NATIONAL DIAGNOSTIC EVALUATION

1. INTRODUCTION

According to the World Bank (2019), approximately half of students from low- and middle-income countries (LMIC) are not proficient in basic or foundational learning skills. This situation has both country specific and regional impacts since education, a fundamental human right, is one of the most effective ways to reduce poverty (UNESCO, 2017). The World Bank posits that, although ¨foundational skills¨ is an umbrella term for multiple skills –such as reading, numeracy, and socioemotional regulation– reading skills are more important given that they facilitate learning in all other areas, becoming a relevant proxy for learning. However, for reading skills to facilitate learning, they must be automatized by the time children are ten years old (World Bank, 2019).

There is evidence that reading skills in first grade predict academic success in later grades. For example, a longitudinal study monitoring US American student from first grade through tenth grade found that children who learned to read by first grade show high scores in reading comprehension, language comprehension, and overall declarative memory in tenth grade (Sparks et al., 2014), and this is especially true for children whose print exposure and reading volume is large.

Another longitudinal study on US American students also showed the importance of early reading in academic success (Herbers et al., 2012). Specifically, the study found that early reading fluency predicted reading comprehension and math scores from third to eighth grade, and therefore is a protective factor for future achievement, in children from low resource areas.

Multiple studies have shown that there is a relationship between solid reading comprehension abilities and maths problem-solving. This relationship has been found in elementary school in US American students (Fuchs et al., 2018); in middle school Dutch students (Boonen et al., 2014, 2016), and Finnish students (Kyttälä & Björn, 2014; Vilenius-Tuohimaa et al., 2008).

The Dominican Republic has increased its investment in public education in recent years. However, such investment while boosting school infrastructure and access, has yet to show improvements in learning outcomes and quality of education. International evaluations consistently show that Dominican students in both third and sixth grade have the lowest scores in the region in reading, math, and science (MINERD, 2016; UNESCO, 2008, 2015). The Dominican Republic’s national diagnostic evaluation reports that 69% of third graders (MINERD, 2017) and 19.6% of sixth graders (MINERD, 2018) cannot comprehend a simple text and answer literal reading comprehension questions.

In order to improve literacy outcomes, the United States Agency for International Development (USAID) funded an educational initiative implemented by Universidad Iberoamericana (UNIBE) from 2015-2022. The objective of the project was to improve the reading abilities of public school students in the Dominican Republic from first grade to sixth grade. The theory of change consisted of a multilevel teacher training model in which teacher training and teacher mentoring would improve teaching quality and, therefore, would improve student outcomes (Wood et al., 2016). The initial improvement of students was confirmed in the midline evaluation when results were compared with baseline findings (Mencía-Ripley et al., 2016; Sánchez-Vincitore et al., 2020). Overall, the study found low scores in reading comprehension and reading comprehension precursors (working memory, phonological awareness, letter knowledge, words per minute, pseudowords per minute, and fluency) for both the group of students who received the intervention (experimental group) and a control group. No significant differences were found between the control and the experimental groups at baseline, which means that the control group was a suitable counterfactual. In 2017, a representative sample of the same cohort (in fourth grade) was evaluated. The study found statistically significant differences between the control and experimental group in most reading comprehension precursors (except for fluency) but not in reading comprehension. This means that the project produced a meaningful change, but the gains in reading comprehension precursors were insufficient to induce changes in reading comprehension by fourth grade. One of the hypothesized reasons for this result was the lack of practice reading materials young learners need in order to attain reading automaticity.

The intervention received a significant upgrade in 2017. Responding to project data, which showed a significant lack of reading materials that were age and grade level appropriate, the project developed and implemented the use of grade-leveled reading practice materials that included decodable and leveled books (Sánchez-Vincitore, 2018). Decodable and leveled texts are a series of books that allow students to have successful independent reading practice at all reading levels. Decodable books are organized according to the frequency of letter combinations in Spanish and how difficult it is to learn each letter. The series has six stages, in which more frequent and easier-to-learn letters are introduced in the first stages, and less frequent and harder-to-learn letters are introduced in the later stages. Since the content of the books restricts the number of grapheme-phoneme conversions the students must transform when they are starting to learn to read, their working memory capacity does not overload. Therefore, the students were more likely to successfully attempt to read the text in context, which is required by the Dominican curriculum (MINERD, 2014). Leveled books are a series of reading practice materials graded according to syntactic and lexical difficulties that go from easier to more complex.

The Dominican Ministry of Education’s 1-2016 ordinance made national diagnostic evaluations mandatory at the end of primary and secondary school. The first evaluations were implemented in 2017 to evaluate third graders (MINERD, 2017) and in 2018 to assess sixth graders (MINERD, 2018). Given the availability of 2018 national data from the Ministry of Education and that project USAID-Read's intervention was updated after the midline evaluation in 2017, we conducted secondary data analysis on the nationwide data as a triangulation strategy to supplement our monitoring and evaluation system.

1.1. THIS STUDY

This study's first aim was to determine if project USAID-Read’s effects could be detected by a national diagnostic educational evaluation two and a half years after the implementation had started. For this purpose, we compared students’ performance on the 2018 sixth grade national diagnostic evaluation scores (language arts, math, social studies, and science) between the experimental and the control groups assigned at the beginning of the project. We hypothesized that students from project USAID-Read’s schools would outperform students from the comparison group at least in language scores, which consisted of a reading comprehension evaluation.

The second purpose of the study was to evaluate if the student’s categorical scores on the evaluations that tap into procedural knowledge (language and mathematics) predicted the likelihood of belonging to project USAID-Read. We expected that participants with higher categorical scores on either language or math evaluations would be more likely to belong to project USAID-Read.

2. METHODOLOGY

This is a quantitative archival design study and secondary data analysis from the Sixth Grade National Diagnostic Evaluation conducted in 2018 by the Dominican Republic’s Ministry of Education to compare the performance of students from schools that received project USAID-Read’s intervention with the performance of students from schools from a control group.

The original methodological design for project USAID-Read consisted of an experimental design with random group assignment and estimation of the counterfactual from a baseline study (Mencía-Ripley et al., 2016; Sánchez-Vincitore et al., 2020). Before project USAID-Read started in 2015, the Ministry of Education provided the project with a list of 400 schools that fulfilled the inclusion criteria, in which 200 schools were randomly assigned to the experimental group and 200 to the control group. The inclusion criteria consisted of: (1) Schools should not have received any literacy interventions; (2) Schools must offer primary education (from first to sixth grade); (3) As requested by USAID, schools must belong to the following regionals 4, 6, 8, 9, 10, 11, 15 y 16.

For the present study, national data pertaining to the sixth-grade diagnostic evaluation were obtained from the experimental and control groups.

2.1. PARTICIPANTS

The total sample consisted of 21,871 sixth-grade students (Project USAID-Read n = 10,736; Control group n = 11,135). There were 10,777 girls and 11,094 boys. There were no statistically significant differences in the distribution of boys and girls among the total sample. Students evaluated on the sixth-grade national diagnostic evaluation were in third grade when project USAID-Read started and had never been tested before this diagnostic evaluation (two years and six months after the beginning of the project). Students from the experimental group received one and a half years of initial intervention programming (teacher training and teacher mentoring) and a full year of the updated intervention (teacher training, teacher mentoring, and the availability of practice reading materials); while students from the control group received traditional schooling.

Since this is a secondary data analysis of national, open, and anonymized data, this study is exempt from approval from an ethics committee. In addition, educational data excludes medical or other types of information that may be considered protected or sensitive. However, the original project USAID-Read intervention was approved by Universidad Iberoamericana’s ethics committee to ensure the protection of human participants.

2.2. INSTRUMENTS

2.2.1. SOCIODEMOGRAPHIC QUESTIONNAIRE

This questionnaire gathered general information by asking overall demographic questions of the participants.

2.2.2. LANGUAGE

This multiple-choice language test assesses reading comprehension of different text types, including narrative, expository, persuasive, instructive, and conversational texts. The test asks questions that evoke literal, inferential, and critical comprehension processes (30%, 40%, and 30% of the questions, respectively).

The language test provides two scores per participant: (1) A numeric score, in which the data are presented on a scale of an established mean of 300 and an established standard deviation of 50; (2) A categorical performance level score that classifies students' numeric scores into three levels: (1) elementary, (2) acceptable, and (3) satisfactory.

For the language test, an elementary level consists of being able to localize easily identifiable explicit information in the text (MINERD, 2018). It reflects literal reading comprehension but without the ability to differentiate information within the test. An acceptable level consists of being able to localize explicit details in different parts of the text; being able to make clear inferences by specific textual cues; being able to understand causal relationships from evident textual cues, and able to determine the purpose of a text. A satisfactory level consists of being able to comprehend text, make inferences about the character's motivations and attitudes in narrative texts, make causal inferences from multiple text information, determine the meaning of a word from contextual cues, and able to determine the text structure.

2.2.3. MATHEMATICS

This multiple-choice test evaluates learning of mathematical concepts, processes, word problem solving, and reasoning. The test is organized into four domains: numeracy, geometry, measurement, and statistics (30%, 20%, 30%, and 20% of the questions, respectively).

The mathematics test provides two scores per participant: (1) A numeric score, in which the data are presented on a scale of an established mean of 300 and an established standard deviation of 50; (2) A categorical performance level that classifies students' scores into (1) elementary, (2) acceptable, and (3) satisfactory.

For the math test, the elementary level consists of only knowing general concepts of numbers and operations, geometry, measurement, probability, and statistics (MINERD, 2018). Students can solve only very simple word problems from each content area at this level. The acceptable level consists of knowing rational numbers (such as whole numbers, fractions, and decimals), conversion from different units of measurement, interpretation of pie graphs, and the ability to solve different types of math word problems. The satisfactory level consists of being able to use concepts and procedures, mathematical reasoning, and problem-solving strategies for word problems that portray different everyday life situations.

2.2.4. SOCIAL STUDIES

This multiple-choice test assesses knowledge about history, geography, and civic education (for 30%, 40%, and 30% of the questions, respectively).

The social studies test provides two scores per participant: (1) A numeric score, in which the data are presented on a scale of an established mean of 300 and an established standard deviation of 50; (2) A categorical performance level that classifies students' scores into (1) elementary, (2) acceptable, and (3) satisfactory.

For social studies, the elementary level consists of knowing concrete geographical and historical concepts and superficially recognizing their importance for social interaction (MINERD, 2018). The acceptable level consists of understanding the cultural, economic, political, social, and geographical characteristics of ancient civilizations and historical processes. Students at this level can identify norms and laws that regulate citizen interaction within democracy in everyday life. In addition, they can apply social science concepts to everyday life. The satisfactory level involves linking complex social science concepts with historical processes to determine their importance in social, political, and economic development. Students at this level can infer cultural inferences to assess the consequences of different contextual historical scenarios.

2.2.5. SCIENCE

This is a multiple-choice test that assesses knowledge about life sciences, physical sciences, and scientific knowledge about the universe (40%, 30%, and 30 of the questions, respectively).

The science test provides two scores per participant: (1) A numeric score, in which the data are presented on a scale of an established mean of 300 and an established standard deviation of 50; (2) A categorical performance level that classifies students' scores into (1) elementary, (2) acceptable, and (3) satisfactory.

For the science test, the elementary level consists of identifying simple concepts of health, energy, matter, and the environment (MINERD, 2018). The acceptable level consists of having basic knowledge about living beings, health, matter, and the environment. Students at this level can identify basic characteristics of systems. The satisfactory level consists of comprehending complex concepts and processes regarding living beings and the universe. Students at this level can compare systems' structures. They can solve problems that explore physical magnitudes, such as distance, temperature, mass, volume, and density.

2.3. PROCEDURES

The Ministry of Education database was obtained in February of 2020. The database contained anonymous information from participants from every school (public and private) in the Dominican Republic. The research team created the study database using student data from schools assigned to the control and experimental groups.

2.4. DATA ANALYSIS PLAN

Before conducting the statistical analyses, we present descriptive statistics of categorical scores for both groups.

To accomplish the first purpose of the study, we conducted an independent samples Student t-test to compare the means of the two groups (experimental vs. control) in numeric scores for language, maths, social studies, and science. Effect sizes were interpreted using cutoff points proposed by Kraft (2018) for interpreting effect sizes in educational research.

To accomplish the second purpose of the study, we applied a null model binomial logistic regression model using forward selection. The criterion variable was group (experimental vs. control group), and the predictors were the categorical performance levels for language and maths (elementary, acceptable, and satisfactory).

3.3. ANALYSIS AND RESULTS

Before answering the research questions, a frequency analysis of the measurements' categorical scores will be presented. Figures 1-4 compare the distribution of students within the different levels in language, math, social studies, and sciences across groups. As can be noticed, only in math and language the performance registered on the database were higher in the satisfactory category and lower in the elementary.

FIGURE 1.
DISTRIBUTION OF SAMPLE BY LANGUAGE LEVELS AND GROUPS

FIGURE 2.
DISTRIBUTION OF SAMPLE BY MATH LEVELS AND GROUPS

FIGURE 3.
DISTRIBUTION OF SAMPLE BY SOCIAL STUDIES LEVELS AND GROUPS

FIGURE 4.
DISTRIBUTION OF THE SAMPLE BY SCIENCE LEVELS AND GROUPSS

3.1. RESEARCH QUESTION 1: ARE THERE ANY STATISTICALLY SIGNIFICANT DIFFERENCES BETWEEN THE EXPERIMENTAL AND CONTROL GROUPS IN THE EVALUATED CONTENT AREAS?

The experimental group significantly outperformed the control group in language (t(21614) = 2.847, p = 0.002, d = 0.039) and math (t(21645) = 5.557, p < 0.001; d = 0.076). There were no statistically significant differences between the groups in science and social studies content areas. Table 1 contains descriptive statistics and the remaining inferential statistics.

TABLA 1.

DESCRIPTIVE AND INFERENTIAL STATISTICS COMPARING EXPERIMENTAL AND CONTROL GROUPS

Note. M = Mean; SD = Standard Deviation; t = Independent groups t of Student; df = Degrees of freedom; p = Significance; d = Cohen’s d; H0: Experimental = Control. Significant results are marked in bold.

3.2. RESEARCH QUESTION 2: DO LANGUAGE AND MATH CATEGORICAL SCORES PREDICT THE LIKELIHOOD OF THE STUDENTS BELONGING TO PROJECT USAID-READ?

A hierarchical binary logistic regression analysis was performed to ascertain the likelihood that participants belong to project USAID-Read depending on the levels detected in language and math areas (see Table 3). The logistic regression model was statistically significant, χ2 (37) = 27.1, p < .001. The model correctly classified 52% of cases. All pseudo R2 increase (although modestly) compared to the null model. The performance metrics indicate a sensitivity of 19% and a specificity of 84%.

Being in the 3rd language level was associated with an increased likelihood of belonging to the experimental group at 13% (p = .003) due to the odds ratio is 1.13, and being in the second and third levels on math were associated with belonging to project Project USAID-Read. These probabilities increase by 10% (OR = 1.10, p = .003) for the second categorical level, and 56% for third categorical level (OR = 1.56, p < .001).

TABLA 2.

HIERARCHICAL BINARY LOGISTIC REGRESSION ANALYSIS FOR LANGUAGE AND MATH LEVELS PREDICTING BEING IN A READING PROGRAM

Note: Group-level “Experimental” coded as class 1. Note. The null model contains nuisance parameters: Language level (2), Language level (3)

4. DISCUSSION AND CONCLUSIONS

The first purpose of the present study was to explore project USAID-Read’s literacy intervention by using data from the Sixth-Grade National Diagnostic Evaluation. Specifically, we compared the language, math, science, and social studies scores of sixth-grade students from project USAID-Read schools vs. students from the control group. This study found that sixth-grade students whose schools participated in project USAID-Read programming for two years and six months obtained significantly better results than sixth-grade students from the control schools in reading comprehension (as measured by the language evaluation) and math. This study contributes to the literature supporting that a combination of teacher training and teacher mentoring in science-based literacy instruction (Stone et al., 2019), in addition to providing adequate reading practice materials (Sparks et al., 2014), positively impacts reading comprehension. Such improvement in reading comprehension was transferred to math scores, as students from the experimental group outperformed students from the control group in math. This finding supports the cited studies (Kyttälä & Björn, 2014; Vilenius-Tuohimaa et al., 2008).

Booen et al. (2014) also found a significant but weak relationship between reading comprehension and math word problem-solving in sixth grade. As they mentioned, their word-solving problems were easy, and therefore, someone with minimal reading comprehension could perform well. In our case, most of our students are struggling readers, and therefore, the actual math questions were a challenge. Even for simple word problems, there is a minimum reading comprehension threshold that should be attained to get enough information to perform well on a mathematical word problem.

The second purpose of the study was to determine if language and math categorical scores could predict if students belonged to project USAID-Read. The results show that students at the highest categorical level (satisfactory level) were more likely to belong to project USAID-Read for language and math scores. It is interesting that a literacy intervention had a stronger effect on math scores than language scores (a difference of 43 percentage points). This could be due to instrumentation. The language evaluation measured reading comprehension on three dimensions: literal, inferential, and critical comprehension. The inferential and critical comprehension dimensions involve many additional processes that are related to cultural competence, previous knowledge, and overall exposure to content, in addition to understanding literal information provided by the text (Elbro & Buch-Iversen, 2013). On the other hand, the math evaluation consisted of concrete word problems in which the students had to extract the necessary information from the text to perform math operations. In this sense, the ability to obtain essential information from the text could have been better captured by the math evaluation, given the complexities of the language evaluation.

Both the control and experimental groups performed similarly in science and social studies. Since the project was not designed to intervene in those areas, and although the intervention was successful at impacting subject areas that require more procedural knowledge (reading and math), the reading comprehension gains were not sufficient to be transferred to content knowledge (science and social studies). It is important to note that, even though the language and math results were significant, they had small and medium effect sizes, respectively. This could probably be the reason for not finding differences between the groups in content areas that demand stronger reading comprehension skills and more declarative knowledge.

The change in reading comprehension that was elicited by project USAID-Read after two years and six months is positive but still modest. There are many reasons for the modest effect of this intervention. As Sánchez-Vincitore et al. (2020) noted, the intervention was not implemented at full capacity from the beginning of the project due to lack of practice reading materials, which means that related data collection activity would have to account for timing issues. In this case, the first two years of project USAID-Read were dedicated to teacher mentoring and teacher training. Although having qualified teachers is essential for students to learn to read, reading automaticity –that is, the ability to read effortlessly and therefore have more cognitive resources available for comprehension (LaBerge & Samuels, 1974)– requires vast successful reading practice. The students evaluated in this data analysis have only received one academic year of the updated intervention, which included teacher training, teacher coaching, and reading practice materials.

The present study did not follow a longitudinal, pre-post design. Therefore, any evidence that is obtained by comparing scores between the experimental and control groups would show a smaller effect size because of the cross-sectional methodological design (Kraft, 2018). Another limitation is that, although project USAID-Read had a baseline study in which the researchers found that students from both groups showed no differences before the intervention started, the sixth-grade diagnostic evaluation students were not part of that initial cohort, and therefore we cannot assume group equivalency at baseline.

Strong monitoring and evaluation systems are important to track any project’s progress. For this reason, project USAID-Read developed highly specialized monitoring and evaluation protocols to assess the project’s trajectory (Mencía-Ripley et al., 2017). However, it is also important to validate these findings with data sets that are not part of the project in order to confirm that the evaluations conducted within the project are free of bias. One of the strengths of the present study is that we used available administrative and national data to compare the scores of the control and experimental groups established at the beginning of the project under the original methodology. Taking advantage of available data is a sustainable practice that not only advances science but also contributes to providing high-quality local evidence for decision-making. Herbes et al. (2012) propose to use administrative data linked to data that is not usually obtained routinely to understand the reasons behind a phenomenon. In Herbes et al.'s case, they were looking for reasons behind the resilience and maladaptation of homeless children in terms of reading. In the case of project USAID-Read, the data merge has allowed us to better understand the subjacent reasons for the low reading scores of Dominican children reported by national and international evaluations (MINERD, 2016, 2018; UNESCO, 2015). The national diagnostic evaluations and the international evaluations measure reading comprehension as a unified skill. However, reading comprehension is the outcome of a demanding cognitive process that takes time and practice to master. Project USAID-Read provides additional information on how this cognitive process takes place in this context by having conducted multiple studies on reading comprehension precursors. These studies have not only validated the project's efficacy (Sánchez-Vincitore et al., 2020) but have also provided support for the simple view theory of reading and described the predictive weights of each subcomponent of reading comprehension (Sánchez-Vincitore et al., 2022).

The present study provides important information about a scientifically sound intervention for literacy instruction in the Dominican Republic. It provides additional data on the intervention’s effectiveness that could be considered for present and future policy both nationally and internationally. Based on the results of this study, more resources should be allocated towards strengthening both early-grade reading skills and, later, more complex content-dependent reading comprehension skills in primary education students. This is especially relevant at the time of this publication, as the international community witnesses significant learning loss due to school lockdowns during the COVID-19 pandemic and the deepening levels of inequality that the current economic crisis is causing (Donnelly & Patrinos, 2021). Agile solutions must be provided from multi-sectoral efforts that include decision-makers, policymakers, educators, scientists, economists, and members of civil society to not only recover the learning loss but surpass expectations of gains in the educational system.

FINANCING

Project USAID-Read was funded by the United States Agency for International Development, agreement number AID-517-A-15-00005. The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the Government of the United States.