STATS ARTICLES 2008
Test scores, graduation rates, and school vouchers
Patrick J. McCloskey, October 20, 2008
Barack Obama says he's keeping an open mind on school vouchers because we need to do whatever works for the kids. STATS asked Patrick McCloskey, author of the forthcoming "The Street Stops Here," a study of Catholic education in Harlem, to look at how the voucher system is being assessed and whether test scores should be seen as the only measure of their success
One of the most surprising policy statements of this presidential election cycle was made by Barack Obama last February during the primaries, when he met with the Milwaukee Journal Sentinel’s editorial board and said he’d keep an open mind about school vouchers. The pledge was surprising because Democratic candidates, traditionally, have allied themselves with the teachers’ unions, which vehemently oppose school choice initiatives, especially vouchers.
Perhaps Obama was just pandering last February, since Milwaukee is home to the nation’s first and largest publicly-funded voucher program. Established in 1990, the Milwaukee Parental Choice Program (MPCP) now enrolls 18,000 disadvantaged and mostly minority students (14 percent of the city’s school-aged children) in 122 private secular and religious schools.
Nationwide, there are twelve publicly funded voucher programs enrolling 56,285 students during the 2006-07 school year. All these programs are designed to serve students with various disadvantages. Five statewide voucher programs are restricted to students with disabilities. Three voucher programs targeting low-income students in large urban centers—in Milwaukee, Cleveland and the District of Columbia—have received much attention in the media since they address the nation’s most urgent and persistent educational problem: the low achievement and high dropout rates of disadvantaged minorities.
Voucher programs have also inspired enormous opposition, primarily from the education establishment and on the political Left. Union officials and public educators tend to see vouchers as a threat because they empower parents rather than school boards to decide where a child goes to school. The Left views vouchers as the conservatives’ way of introducing competition into public education as a whole, and as a means of promoting the privatization such programs as Social Security and replacing many others with free market initiatives. In addition, since most (85 percent) of the private voucher schools serving low-income students are religious (and mostly Catholic), vouchers inspire stiff resistance from those who believe in a strict separation of church and state.
In 2006, voucher opponents won a lengthy legal battle decided in Florida’s Supreme Court that struck down a statewide program for students attending failing public schools. In this politically charged environment, Obama’s pledge to remain objective about vouchers raised an important question: Do vouchers help educate the students who fare so miserably in the public system?
At the time of Obama’s editorial meeting, the most reliable information was found in nine gold-standard analyses of random assignment voucher programs serving disadvantaged students. Seven of these studies evaluated privately funded, partial-tuition voucher programs in New York City, the District of Columbia, Dayton, Ohio and Charlotte, North Carolina. One study evaluated a publicly funded program in D.C. and another evaluated the Milwaukee program.
Eight of these evaluations reported positive and statistically significant achievement gains for all students or at least for African Americans. One evaluation showed no benefits. However, it would be fair to say that the achievement impact is modest at best, especially in light of voucher having been heralded as the “panacea” of education reform by advocated in the early 1990s.
Shortly after Obama left Milwaukee, the School Choice Demonstration Project (SCDP), based in the University of Arkansas’ Department of Education Reform, published the first series of reports (five out of a total of thirty-six reports over five years) about the Milwaukee program that will constitute the most comprehensive, rigorously researched voucher evaluation to date.
At first glance, it seemed disheartening for advocates to see that after 17 years, voucher students performed at about the same levels as demographically-matched Milwaukee Public School (MPS) students (an apples-to-apples comparison). All students averaged around the 33rd percentile, typical for low-income students in urban centers across the country.
But it’s worth noting that the standard deviation for voucher schools was higher in all tests on both grade levels than for MPS schools. For example, on the 4th grade reading exams, students averaged between the 20th and 30th percentiles at 17 (or 14 percent of) voucher schools; in contrast, at 18 other voucher schools, students averaged higher than the 50th percentile. This increased dispersion “is likely because of the lower number of [voucher] schools and greater diversity of schools in that group,” the study states.
However, there are two other intriguing possibilities. Since test scores for individual voucher schools are not published in order to protect confidentiality, parents must rely on word of mouth and sources of incomplete data to make their choice of voucher school. This “might explain why some parents choose to place children in choice schools that don’t do well,” conceded the study’s lead researcher, Patrick J. Wolf, an education professor at the University of Arkansas, in a telephone interview with this article’s author.
Secondly, the increased dispersion of higher scoring schools might be accounted for by the large number of Catholic and Lutheran schools (which as Wolf notes, function quite similarly to their Catholic counterparts). In total, 36 Catholic schools educate 37 percent of voucher students, and 26 Lutheran schools educate 16 percent of these youngsters. Given the long history of Catholic and Lutheran school efficacy, it seems likely that these schools perform at or above the norm. When asked whether this was true here, Wolf answered, “It’s plausible. I’ve looked at parochial schools systems in other studies and they have solid reputations raising the educational aspirations, attainment and to some extent test scores of disadvantaged kids. But I can’t confirm or deny this in Milwaukee.”
It’s possible the voucher program would show a significant test score advantage over MPS if parents were properly informed, both about the schools dragging down average scores and the ones that excel. This dynamic of data and decision constitutes an integral part of school choice’s theoretical engine and needs to be considered. Market forces exerted by parental choice are supposed to put lousy schools out of business and promote good schools. In 2005, reporters from the Milwaukee Journal Sentinel visited 106 of the city’s voucher schools and found that “[a]bout 10% of the choice schools demonstrate alarming deficiencies” and many parents left their children in lousy schools long after it was obvious they were failing. If this observation was substantiated, then the utility of parental choice is questionable.
Putting this consideration aside, the existing test scores are consistent with the more important outcome: educational attainment. Voucher students scored slightly below MPS students on the 4th-grade level and slightly above at 8th grade. On the 10th grade level, voucher students improved again, but the sample was too small for analysis. This trajectory follows that of urban Catholic school students in other reports. A comprehensive study sponsored by New York University in 2001 comparing the city’s Catholic and public schools showed students performing very similarly in the lower grades. By the 8th grade, public school students lagged considerably on state tests. Evidence of this trend continuing through the 12th grade is expressed every year in much higher graduation and college acceptance rates at Catholic high schools.
Researchers from James Coleman in the 1980s to Derek Neal, an economics professor at the University of Chicago, in the 1990s, to a study released in May 2008 by Loyola Marymount University’s School of Education document very high graduation rates for minority students in urban Catholic high schools. Neal found that black and Hispanic students actually matriculated at higher rates (91 percent) than urban whites (85 percent). The Loyola study tracked 205 disadvantaged students who received private vouchers to attend Catholic high schools in Los Angeles and 98 percent graduated.
Also in May 2008, a report entitled, “Graduation Rates for Choice and Public School Students in Milwaukee: 2003-2007,” by John Robert Warren, a sociology professor at the University of Minnesota, showed that the proportion of voucher students graduating on time rose from 62 to 85 percent from 2003 to 2007, compared to public school graduation rates increasing from 49 to 58 percent. As Wolf points out, most voucher students attend Catholic or Lutheran secondary schools, meaning that the study could almost be included among those in the previous paragraph.
Warren’s report indicates that vouchers significantly improve educational attainment
and also inspire, or challenge, public schools to improve. Research—notably by Caroline Hoxby, a Harvard economics professor—confirms that competition helps make public schools better.
Warren cautions against drawing casual inferences about the effect of voucher schools on attainment. To address this question with unassailable research results, SCDP is conducting a five-year Longitudinal Educational Growth Study in which 2,727 freshmen in voucher high schools were matched demographically with the same number of public school 9th graders (1,926 voucher and (demographically matched MPS) students in grades 3-8, for the achievement study; 801 voucher and (demographically matched MPS) students in grade 9, for the attainment study). Currently the students are in the 10th grade, so definitive information about graduation, dropout and college acceptance rates won’t be known until after the 2011-12 school year, since the students are also being tracked for the year beyond high school.
Educational attainment is a vital national issue. As shown in The Race Between Education and Technology by Claudia Goldin and Lawrence Katz, the average American’s level of education rose by 0.8 years per decade beginning in 1870, such that American students spent 14 years in school on average by 1960. This was substantially more than students in most rival countries, boosting productivity and fiscal growth, and resulting in global economic dominance. The U.S. high school graduation rate peaked in the late 1960s and then declined by about 5 percentage points from 1970s through 1990s. At the same time test scores stagnated, allowing nations around the world that were decades behind to outpace us in academically.
Educational attainment and graduation
Meanwhile in the inner-cities, educational outcomes deteriorated beyond imagination. Consider the National Assessment of Educational Progress (NAEP), the nation’s report card. On the 2007 4th-grade reading test, only about 12 percent of African Americans in big city school districts scored at or above proficiency. Disadvantaged blacks achieved proficiency at even lower rates in the same districts, and in some cities, disadvantaged African American males averaged close to the three-percent statistical margin of error in proficiency.
These low test scores naturally translate into low graduation rates. While more than three-quarters of whites and Asians graduate high school on time nationwide, only about half of black students do so. Hispanics perform slightly better at 56 percent. Minority males fare significantly worse: only 44 percent for African American boys and 50 percent of Latino boys graduate on time nationwide. In large urban school districts, such as New York City, only a third of black and Latino males finish secondary school on time, and their plight is even worse in heavily disadvantaged areas such as Harlem and the South Bronx.
Dropouts have a 72-percent higher unemployment rate than high school graduates and a 400-percent higher rate than college graduates. After-tax median income (2005) follows suit at $19,000, $25,000 and $39,000 for a bachelor’s degree (and $47,000 for a master’s). Dropouts also have significantly higher rates of public assistance and are much more likely to become single parents, almost guaranteeing that their children will grow up in poverty. Welfare dependence (1,700 percent more common for children of never-married mothers) and single parenthood—not race—have become the two most important (and interrelated) causes of poverty, especially among young mothers with low academic skills.
Obviously this is an emergency situation, and if educational shortcomings were physical wounds, no-one would quibble about whether public money should be spent sending an accident victim to a Catholic hospital. Graduation as a life-and-death issue isn’t a metaphor in the inner-cities. Last December, a bipartisan organization with membership including hundreds of police chiefs, sheriffs, district attorneys and violence survivors released a report entitled, “School or the Streets: Crime and California’s Dropout Crisis,” linking low graduation rates with violent crimes and noting that 68 percent of state prison inmates are dropouts. The report contends that a 10 percentage point increase in graduation rates would cut murder and assault rates by 20 percent, preventing 500 murders and over 20,000 aggravated assaults in California each year.
Certainly the early optimism about vouchers’ capacity to turn around declining educational attainment was misplaced: Test scores haven’t rocketed, and are unlikely to. As Charles Murray notes in Real Education, there isn’t any evidence showing that academic ability can be raised much on a large scale. Academic ability, which he defines as the combination of logical-mathematical, linguistic and spatial abilities, is influenced greatly by parents and home environment and is set around the time a child begins the first grade.
While social factors in early childhood are currently beyond the reach of effective educational intervention, much can still be done to mitigate negative demographic factors once children reach school age. But we need to have a realistic sense of what can be achieved. At a relatively small proportion of schools, disadvantaged students post high test scores, including some of Milwaukee’s voucher schools. At Amistad Academy, a charter middle school in inner-city New Haven, Connecticut, students enter the 5th grade with scores on the state exams averaging in the 33rd percentile. Amistad students are predominately disadvantaged minorities admitted to the school via a blind lottery. At the end of 8th grade, the same youngsters score in the 61st percentile. How much this can be scaled up district-wide is an open question, and as Roland Fryer, a Harvard economics professor and recent recipient of a $44 million grant to research strategies to improve public education, said in an interview with this article’s author, “Scaling up is the problem.”
“The question is, to what extent do vouchers deliver educational value program-wide?” Wolf said. “We know it’s difficult to move student achievement for large numbers of disadvantaged kids. This requires an effective intervention on a sustained basis so that marginal improvements continue until children operate at a higher level. It’s not magical but the slow steady grind that characterizes the Knowledge is Power Program (KIPP) approach.”
Like the Achievement First network of charter schools, the first of which was Amisted, KIPP academies focus on a disciplined approach to teaching and re-teaching basic skills. Nationwide there are 60 KIPP schools serving 16,000 K-12 low-income students, 80 percent of whom graduate and go on to college. Like voucher schools, charter schools display a wide variety of pedagogical models and success rates. But those that succeed with low-performing students tend to share remarkable similarities with urban Catholic schools, aside from religious instruction. This is an anecdotal observation on my part, but based on considerable experience visiting inner-city public and private schools and on many interviews with administrators and other researchers—and accords with Wolf’s findings.
Wolf also agrees that the underlying problem for disadvantaged students is the ongoing breakdown of the family. In “Schools, Skills and Synapses,” James Heckman, an economist at the University of Chicago, identifies the deterioration of the family over the last 40 years as the main reason academic achievement and attainment have declined across the country. Nowhere has family structure suffered more than in inner-city minority neighborhoods where often over 80 percent of households are headed by single females.
Considering the efficacy of inner-city Catholic schools (and other academies that have adopted the same model), “from Coleman on, researchers have posed the question: do urban parochial schools succeed because they backfill what’s missing in students’ families?” Wolf related. “Do order, discipline, guidance and focus on basic skills shore up the efforts of a single parent and provide a substitute for an intact family that helps prepare children for success?”
Mounds of contemporary studies indicate this is true, and more significantly, the long history of parochial schools facilitating the transition of wave after wave of impoverished immigrants, with high rates of family dysfunction, to the mainstream leans heavily towards confirmation.
Wolf noted that a disproportionately high percentage of families that “show up for focus groups in the Milwaukee program are married couples. It’s clear that the father is playing a significant formative role and both parents share the responsibility and mission of educating their child properly. Although poor, they’re facing challenges together with the social capital of two responsible adults.”
None of this constitutes definitive proof regarding vouchers but as Wolf explained, his Longitudinal Educational Growth Study will ascertain “whether vouchers position disadvantaged kids in the kind of system where they experience small daily gains, as at KIPP schools, over a long enough period of time that the marginal payoff accumulates to a significant degree regarding both achievement and attainment.”
He adds that the evaluation “will determine not only if the program works as a whole. Detailed information will also be collected about school environments and pedagogies to see if we can identify the elements that work on a large scale. These best practices will be the most useful conclusions for educators.”
At the final presidential debate, the candidates argued over vouchers with Obama stating, “I disagree with [Sen. McCain] on this, because the data doesn't show that it actually solves the problem.” While voucher opponents might have been pleased by this response, Obama’s promise to keep an open mind wasn’t compromised. At the meeting with the Journal Sentinel editorial board in February, Obama was surprised to learn that Milwaukee’s voucher program had yet to be evaluated in a longitudinal study. “If there was any argument for vouchers it was, all right, let's see if this experiment works,” he said, “and then if it does, whatever my preconceptions, my attitude is you do what works for the kids.”
No quick fixes
The longitudinal study is in its first stages and as shown above, indications are positive regarding educational achievement. But in the short term, the research points to the fact that graduating high school is more significant measure of success for vouchers than higher test scores. It’s very difficult for today’s youth to become productive members of society without matriculating. Nor is the G.E.D. an acceptable substitute since students who take this route don’t fare better than dropouts vis-à-vis employment and earnings.
The more difficult aspect of this crisis to grasp is that, historically, it has taken every racial and ethnic group several generations to climb the socioeconomic ladder from underclass to working class to middle class and beyond. It seems more likely that test scores will rise accordingly than suddenly shoot up, which is consistent with voucher studies showing it takes several years for the program to show a positive effect, albeit usually small, on test scores. Seen in this light, voucher programs appear to play a significant role. Disadvantaged minority students are more likely to stay in school, graduate and transition to the mainstream where they will have increased access to the resources important in childrearing and a greater likelihood of establishing a stable, two-parent household.
There’s no conflict between strengthening public education and supporting all viable forms of school choice. After all, public funding of education doesn’t entail that all schools have to be publicly run. But they do have to fulfill the public mandate of providing a quality education to all children. Let open minds and the evidence decide.
Patrick J. McCloskey is the author of The Street Stops Here, a narrative non-fiction book about inner-city Catholic education, to be published by the University of California (Berkeley) Press in January 2009
Patrick J. Wolf, “School Voucher Programs: What the Research Says About Parental School Choice,” Brigham Young University Law Review, April 14, 2008.
J. Barnard, C. Frangakis, J. Hill and D.B. Rubin, “Principal Stratification Approach to Broken Randomized Experiments: A Case Study of School Choice Vouchers in New York City,” Journal of the American Statistical Association, Vol. 98, p. 299-323 (2003); Joshua M. Cowen, “School Choice as a Latent Variable: Estimating the 'Complier Average Causal Effect' of Vouchers in Charlotte,” Policy Studies Journal, Volume 36 Issue 2, (2008); Jay P. Greene, Paul E. Peterson and Jiangtao Du, “Effectiveness of School Choice: The Milwaukee Experiment,” Education and Urban Society, Vol. 31, No. 2, p. 190-213 (1999); William G. Howell, Patrick J. Wolf, David E. Campbell, Paul E. Peterson, “School Vouchers and Academic Performance: Results from Three Randomized Field Trials,” Journal of Policy Analysis and Management, Vol. 21 Issue 2, p. 191-217, (2002); Alan B. Krueger and Pei Zhu, “Another Look at the New York City School Voucher Experiment,” The American Behavioral Scientist, vol. 47, No. 5, p. 658-698, (2004); Alan B. Krueger and Pei Zhu, “Inefficiency, Subsample Selection Bias, and Nonrobustness: A Response to Paul E. Peterson and William G. Howell,” The American Behavioral Scientist, Vol. 47, No. 5, p. 718-728, (2004); Paul E. Peterson and William G. Howell, “Efficiency, Bias, and Classification Schemes: A Response to Alan B. Krueger and Pei Zhu,” The American Behavioral Scientist, Vol. 47, No. 5 699-717 (2004); Cecilia Elena Rouse, “Private School Vouchers and Student Achievement: An Evaluation of the Milwaukee Parental Choice Program,” The Quarterly Journal of Economics, Vol. 113, p. 553-602 (1998); Jay P. Greene, “Vouchers in Charlotte,” Education Matters, Vol. 1, No. 2, p. 55-60 (2001).
Caroline M. Hoxby, “Analyzing School Choice Reforms,” in Learning From School Choice, Peterson and Hassel, eds., Brookings Institution (1998); Caroline M. Hoxby, “The Rising Tide,” Education Next, Winter 2001; also see: Rajashri Chakrabarti, “Vouchers, Public School Response, and the Role of Incentives: Evidence from Florida,” Staff Reports 306, Federal Reserve Bank of New York (2007).