Resource Center

• by María Robledo Montecel, Ph.D., Josie Danini Cortez, M.A., and Albert Cortez, Ph.D. • IDRA Newsletter • October 2001

Editor’s Note: Last year, the Intercultural Development Research Association (IDRA) conducted a research study with funding by the U.S. Department of Education, Office of Bilingual Education and Minority Languages Affairs (OBEMLA) to identify characteristics that contribute to the high academic performance of students served by bilingual education programs. The August 2001 issue of the IDRA Newsletter began a series of six articles describing this research study’s significant findings. The first installment provided an overview of the research design and methods. In the September 2001 issue we featured an overview of the schools’ demographics and the major findings pertaining to school indicators. This third installment features the major findings in student outcomes.

More than two decades ago, IDRA’s founder and director emeritus, Dr. José A. Cárdenas, wrote: “The evaluation design, materials, and techniques commonly used by the school are frequently most inappropriate for use with minority populations. Not only are the tools inadequate, but conclusions based on cultural and language biases can be extremely erroneous and detrimental to the student” (Cárdenas and Cárdenas, 1973).

While much has improved in the area of accountability and assessment, much remains to be done. Nowhere is this more evident than in the assessment of bilingual education programs where assessment tools and their appropriate use with limited-English-proficient (LEP) students are often found lacking. Without appropriate and meaningful assessment tools that hold the teachers and administrators accountable for student academic achievement, it is impossible to determine a program’s effectiveness or impact on the students it is serving.

IDRA’s research study of bilingual education programs was grounded in the premise that a “successful” bilingual education program must have evidence of student academic achievement as determined by appropriate assessment measures. Each of the 10 programs selected for this research study provided data for students in their bilingual education programs. Given that IDRA had operationally defined “student success” as evidence of academic achievement, it was imperative that programs provide relevant and appropriate data for review. This data included student outcome indicators, such as oral and written language proficiency and content area mastery in English and the native language.

Prior to IDRA’s site visits, each school submitted for review its most recent achievement data (1997-98) disaggregated by LEP and non-LEP status. Longitudinal data (three years or more), if available, were also provided. Assessment measures, as expected, varied among the 10 programs. These programs were located in schools in California, Florida, Illinois, Massachusetts, Oregon, Texas, Utah, and Washington, D.C.

Assessment Measures

In reviewing the data provided by the schools, it is important to note some caveats regarding LEP student assessment – namely, that variations in assessment instruments across states, the exemption of LEP students from testing and data not reported by the category of “LEP student” – makes comparisons of achievement data across sites next to impossible. Exemptions for LEP students at the schools we studied were uncommon, with only one school in Texas reporting a 2 percent exemption rate. All of the other schools reported no exemptions.

In compliance with the U.S. Department of Education, Office of Bilingual Education and Minority Languages Affairs (OBEMLA) regulations governing the implementation of Title VII-funded programs, there is a range of student assessment instruments across the country. Title VII grantees are required to collect common types of data, including achievement data, language proficiency data, and teacher credentialing and certification data. While these data are included in many of the evaluations submitted to OBEMLA, there is no current requirement that the grantees use any specific assessment instruments. The rationales for this usually include variations in program focus, differing state assessment requirements that may parallel or take precedence over local assessment decisions, and the belief that certain nationally-normed standardized test instruments may be better aligned with the local program curricula.
All of these are considered legitimate reasons for non-standardization. However, without a uniform standard of assessment involving common instruments, a comparative analysis across sites would be inappropriate. Any macro- or meta-analysis can only attempt to paint broad brush strokes of common assessment and evaluation practices at schools implementing bilingual education programs.

In IDRA’s review of the evaluation data submitted by the schools, two things became evident: all of the schools tested their students and were committed to accountability for all students, and there was a wide range of assessment instruments used by schools.

Keep in mind that part of the selection process for this study required all of the schools to have data reflecting high student performance on locally-selected achievement measures. Given the known variability across sites, specific types of data requested were not prescribed. Nevertheless, the instruments used by schools tend to cluster into three major types:

• state-mandated assessments that are part of a state assessment or accountability system;
• locally-selected instruments in English and/or the students’ native language, that are nationally-normed and considered appropriate for evaluation of the program being implemented; and
• locally-developed instruments that yield data considered useful by the local project in assessing its effectiveness.

Of the schools studied, one in California, one in Illinois, one in Oregon, and two in Texas use data collected from state-required assessment programs as part of their local program evaluation. In California, the school used the Stanford Achievement Tests – required under the California Standardized Testing and Reporting (STAR) system. In Illinois, the school incorporated data collected as part of the Illinois G Achievement Program (IGAP). In Texas, the schools used the Texas Assessment of Academic Skills (TAAS), which is the state-developed criterion referenced measure used to evaluate Texas schools’ performance.

Other sites used non-mandated standardized tests, including the Brigance, the California Test of Basic Skills, and the Woodcock Muñoz Battery, to obtain objective measures of student progress. Some of the sites also reported student achievement using standardized tests in the student’s native languages such as the APRENDA.

In addition to achievement test data, some of the schools track attendance rates, retention, and student graduation rates for all of their students.

Assessment Practices

The assessment and evaluation practices also varied across schools. We found that the schools we studied tended to do the following:

• compile data on year-to-year progress of students enrolled in the bilingual programs and simply assess the extent and/or statistical significance of those changes (Oregon);
• compare the performance of program students against a state passing standard (Illinois, Texas);
• compare the percentage of program pupils scoring at or above a set percentile – usually the 50th percentile (California, Illinois, Oregon); and/or
• compare the program students’ performance against some other standard such as “expected scores” developed by the test publishers (Oregon: Pre-LAS).

The number of years of achievement data available ranged from two to five years. The schools had compiled at least two years of data – allowing for pre-post comparisons (Oregon) or multiple year trend analyses (Illinois, Texas). Some of the data provided was longitudinal – spanning several years, while other data focused on a single year, comparing program performance levels against some local or state-selected standard.

Almost all of the sites measured students’ progress in English, assessing proficiency (LAS: Brigance) and/or English reading (TAAS, CTB, SAT, IGAP, Oregon Plus). The majority of schools also assessed student achievement in mathematics (California, Illinois, Oregon, Texas).

Programs tended to compare their students’ performance either against an external performance standard (state passing scores or percentiles – California, Illinois, Oregon, Texas) or against the average score for non-LEP pupils or sub-groups of other pupils, such as Title I and special education.

Unique Student Assessment Features

Schools reflected the different contextual features in their assessment measures. Some states, in addition to assessing reading and mathematics, measured student achievement in language (Illinois), social studies (Illinois), spelling (California), and writing (Texas).

Common Achievement Level Findings

In analyzing student achievement data, there are significant observations that are common to all of the schools:

• They collected and analyzed one or more types of student achievement data, using multiple measures.
• They had procedures for assessing all of their students and for compiling, organizing, and analyzing their student data.
• They engaged in some tabulation and analysis of the data. Some had external support from external evaluators; others involved teachers in the collection and the analysis of the data to help school teams craft improvement plans.

IDRA also observed the use of multiple measures, which were culturally and contextually appropriate for the students. In addition to the yearly progress measures, there were ongoing or interim measures that were used as benchmarks and indicators of progress throughout the year. Schools used data to inform and drive their curricular and instructional practices, with administrators and teachers accepting accountability for the academic performance of their students.

Student Outcome Indicators

All of the 10 schools that IDRA studied reflected significant progress (statistically and educationally) for the students served by their bilingual education programs during the program year (1997-98). While, in some cases, there was a notable gap in the achievement of students served by the program and the regular students, especially when they were compared to the state’s standards, the majority of students reflected a narrowing of the achievement gap over time.

In fact, in many cases, the growth rates for the students served in the program sites exceeded the rates of improvement for the comparison groups included in the reports. In a few instances, the growth rates were extraordinary, reflecting accelerated improvement rates over relatively short time frames.

Example of a Successful Bilingual Education Program

Each bilingual education program is part of a school with its own unique context and special characteristics that are clearly evident. These characteristics or “indicators of success” are described in the following profile of one school, providing a firsthand look at the inner workings of a successful program and school.

James Bowie Elementary School, Alamo, Texas

Part of the Pharr-San Juan-Alamo Independent School District, James Bowie Elementary School is located in Alamo, Texas. The district is in the southern tip of Texas, known as the “Valley” (even though, geographically, the area comprises the Rio Grande river delta). The three cities that make up the Pharr-San Juan-Alamo Independent School District are small, the largest being Pharr.

The entire area’s population is approximately 40,000 permanent residents. The population swells during the winter months because of an influx of retired individuals who migrate to the area in the winter in and return to their northern homes in the spring. Access to Mexico is via the McAllen-Hidalgo-Reynosa International Bridge, only 11 miles away.

The area is a center for winter vegetables, citrus and cotton; most of the students come from the agricultural community. Because there are three growing seasons in the Valley, most agricultural workers are not migrants but permanent residents of the area.
James Bowie Elementary School is a clean, well-lit school, very functional and conducive to learning. The classrooms are decorated with student work. Each classroom also displays some kind of cultural artwork, such as prints by Diego Rivera, Pablo Picasso and Vincent Van Gogh.

Students at the school represent three ethnic groups, but the overwhelming majority – more than 95 percent – are Mexican Americans who speak Spanish as their native language. Anglo students represent less than 5 percent of the student body, and less than 1 percent are African-American.

Classrooms are highly student-centered at James Bowie Elementary School. Rules are posted on one wall, while student work, examples of recently introduced lessons (e.g., vocabulary words) and other educational materials are displayed prominently on other walls. Rooms are divided into “work centers” that enable the students to take advantage of a variety of learning tools.

A computer classroom is one work center, and children as young as pre-kindergarten make use of the area. In another area, the writing center introduces even the youngest students to the elements of proper writing, beginning with holding a pencil correctly and tracing various shapes. The music center allows the students to identify different instruments by sight and sound and to listen to and learn about different types of music. Pre-kindergarten classrooms always have music playing softly in the background – from classical to folk.

Impressively, 20 or more students in each classroom, working in groups, did not result in an overwhelming noise level. One IDRA researcher noted, “In every room, in the hallways, even in the music room, the sounds being produced were sounds of learning.”

Teachers at James Bowie Elementary School use every teaching opportunity to relate to the prevailing culture of the area, touching on the everyday life of the children. Examples in math classes may utilize the exchange rates between dollars and pesos. Writing assignments and class discussions encourage examples from students’ homes and community. Such topical basis in the everyday lives of students reinforces the importance of the area’s culture.

The teachers are cognizant of the need to provide a learning environment attentive to the needs of English-learning students: classes are conducted in Spanish even though every student is bilingual. As a lesson moves along, the teacher may teach a concept in Spanish, but the students may answer in English. English-learners pick up vocabulary from Spanish-learners, and vice-versa.

In addition, the school has hired several Title I-support teachers who provide intense Spanish instruction in a pull-out program. These teachers work with those students who will be tested in Spanish.

Another IDRA researcher noted, “I had the opportunity to observe these classes and found the students completely engaged in discussion and hands-on activities before they began writing their compositions.” The climate in these classes mirrors that of the regular classroom.

Much of the uniformity in class structure and equality of lesson plans is the result of collaborative planning. Teacher conference and planning periods are scheduled at the same time each day by grade level. This allows time for development and sharing of ideas on how to use curriculum and materials to augment effectiveness.

Test results are reviewed at this time, as teachers are held accountable for student learning in six-week assessments. To maximize test scores, teachers provide after-school and Saturday tutoring sessions. When they see that a student is experiencing academic difficulties, they provide one-on-one tutoring sessions for that student.

Writing assignments tend to reflect the cultural background of the students and always begin with a classwide discussion of the topic. Sometimes, the teacher assigns students to write a group story. For example, one class was prompted: “Te he premiado $2,500. ¡Como vas a compartir este dinero? [You have been awarded $2,500. How are you going to divide up this money?].” One student began discussing how his uncle had won some money, and that if this had happened to the student, he would give the money to certain groups of people. Other students joined in the discussion. After 10 to 15 minutes, the teacher asked them to come to a consensus. The students decided that they would help out their families, their church and the poor children of Mexico. The teacher then proceeded to model the writing process, and wrote a group story as a class.

Again, an IDRA researcher commented: “It is discussion like this that leads me to believe that the English-learning students are being served, not only academically but also culturally. Every lesson I observed touched on the everyday life of the children.”

One unique aspect of the bilingual program at James Bowie Elementary School is the “One World, One Culture” class. This is an enrichment class that all students attend once a week as part of a Title-I schoolwide project. In this class the lessons are structured to teach self-respect and pride in Hispanic cultures as well as a diverse array of other cultures.

The teacher of this class is specially trained in diversity and very knowledgeable in the areas of history and geography. Because she immediately captures the students’ attention, everyone looks forward to the class. There is also a hands-on component that usually takes the form of a writing assignment.

One observed lesson focused on the importance of older family members in various cultures. The teacher began a discussion of grandparents, asking specifically about the children’s grandparents and how they were regarded in their own families and culture. The final assignment had each student design and create a card for his or her grandparents to be presented to them on Grandparents’ Day.

Another unique program at the school is the music (Estudiantina) program. During music class, students in third, fourth and fifth grades begin learning how to play various instruments, such as the violin, guitar, mandolin, or piano. Each student is provided an instrument. They play Spanish songs with which they are all familiar, songs they hear on the radio, in the community, at weddings and other special occasions. At the same time, they learn the foundations of music and theory. The Estudiantina requires after-school practice sessions, and the group performs often throughout the Rio Grande Valley.

Students participate year-around in cultural events in the community such as parades, social functions and holiday celebrations where the choir, drill team, folkloric dance troupe, Estudiantina and other groups perform. As one teacher put it, “The school has set up various committees for the sole purpose of promoting unity among staff, students and their families in community involvement.”

Lessons in all classes tend to be interconnected across disciplines, which is accomplished by using literature-based lessons. When a teacher introduces a story, a discussion is held and then a semantic map is created. The students’ work then reflects the connections across the curriculum. Discussions and research on topics under study are facilitated by access to computers – at least two in each classroom and separate computer labs available to all children.

Students with special needs are served in the regular classrooms. They are not singled out, rather they mix with the rest of the class. The teachers may afford them more individualized attention, but their inclusion in discussions and group assignments is the same as other class members.

Children at James Bowie Elementary School are fortunate that bilingualism is inherent in the culture of the Valley. Spanish and English are both spoken in conversations throughout the area, often blended together in the same sentence. Children in bilingual classrooms receive instruction in Spanish, but they carry on regular conversations with their friends in English. Bilingualism in the area, coupled with the school’s comprehensive bilingual program, is a main reason the school has earned national recognition for the performance of bilingual students.

Teachers keep a closely monitored portfolio of each student’s work that is shared with the student’s parents on a weekly basis. Most of the work included is work created by the student, not worksheets that he or she has completed. This portfolio is also reviewed by the school administration. Together, teachers and staff monitor students’ progress so that any needed modifications are made as soon as possible and instructional time is not lost. Parents must sign and return the portfolio so that the teacher may document that they are aware of their child’s progress. Communication with parents is in both Spanish and English.

It is evident that all the teachers at James Bowie Elementary School believe their students are important. The school’s vision and goal is the success of all its students as reflected in the school motto, “All students can learn.” Students are treated with respect and dignity, and the students treat their teachers in the same manner.

Throughout the entire school, banners in both Spanish and English are displayed, reinforcing that each student is important. A sense of pride is evident everywhere in the school. This sense is engendered and reinforced by the teachers and staff and creates a bond among the students, parents and school personnel. This bond perpetuates the high standards expected from each student and gives families as well as the entire community a stake in the school’s success.

Resources

Cárdenas, J.A., and B. Cárdenas. The Theory of Incompatibilities: A Conceptional Framework for Responding to the Educational Needs of Hispanic Americans (San Antonio, Texas: Intercultural Development Research Association, 1977).

Robledo Montecel, M., and J.D. Cortez. “Successful Bilingual Education Programs: 10 Schools Serve as Models,” IDRA Newsletter (San Antonio, Texas: Intercultural Development Research Association, September 2001).

Robledo Montecel, M., and J.D. Cortez. “Successful Bilingual Education Programs: Criteria for Exemplary Practices in Bilingual Education,” IDRA Newsletter (San Antonio, Texas: Intercultural Development Research Association, August 2001).

María Robledo Montecel, Ph.D., is the IDRA executive director. Josie Danini Cortez, M.A., is the production development coordinator. Albert Cortez, Ph.D., is the director of the IDRA Institute for Policy and Leadership. Comments and questions may be directed to them via e-mail at feedback@idra.org.

[©2001, IDRA. This article originally appeared in the October 2001 IDRA Newsletter by the Intercultural Development Research Association. Permission to reproduce this article is granted provided the article is reprinted in its entirety and proper credit is given to IDRA and the author.]