Dubious Data Awards 1997
It's A Barbie World
Most reporters and pundits repeated the popular factoid that Barbie's measurements, if projected to adult size, would be 38-18-34 ("a figure not found in nature," according to Tina Rosenberg of the New York Times). What most reports didn't mention was that the source for this projection was a 1995 article by Yale researcher Kelly Brownell in the International Journal of Eating Disorders.
But one journalist, Paul Mulshine of the Newark Star-Ledger, thought that this figure sounded like the product of "some militant feminist." Taking matters into his own hands, Mulshine used a ruler to measure Barbie himself. He found that by simply multiplying each of her dimensions by 6, one winds up with a 5' 9" woman with measurements of 33-18-31. Skinny, yes, but much more reasonably proportioned than the widely repeated figure, which turns out to be based on "holding hip size constant."
We here at STATS spend a great deal of time debating the best way to measure things, but when it comes to Barbie, quite frankly we admit to being stumped. And while we admire Mr Mulshine's gumption, he ultimately does not earn full STATS credit. It turns out that he failed to do enough reportorial leg work ("she didn't return my phone call") to learn that "militant feminist" Kelly Brownell is, well, ... a man. In fact, he looks quite a bit like Ken.
Strom Thurmond, Call Your Office....
This story accurately summarized a study published in the journal Nature, which found that the older the husband and the younger the wife, the more likely a couple's first child will be a boy. What's missing is the rest of the story. Three separate research teams dismissed these results two months later, in the November 20 issue of Nature. But no journalist covered this reversal, which showed that "no matter whom you marry, your chances of having a male child for any given live birth are about 51.4 percent."
The critical difference in the respective studies was sample size. The original finding was based on a study of 301 British couples. The debunkers looked at hundreds of thousands of records before dismissing this apparent association as mere sample error. Unfortunately for journalists, statistical anomalies are as real as regularities.
This Is Your Child's Brain On Drugs...
"Drug Use Rising Among Young Adults" -- Associated Press, a few hours later on the same day
These dueling headlines were based on the same National Household Study on Drug Abuse survey, which found that illicit drug use among the young was up, in some cases alarmingly. The AP's initial conclusion illustrates the perils of data slicing (focusing on only one segment of the study population) and a failure to appreciate statistical significance. Young people between the ages of 12 and 15 years old did report a slight decline in the use of marijuana. But the next age bracket, dubbed "young adults" aged 18 to 25, showed a significant increase in marijuana use. More importantly, the drop among younger teens was not statistically significant, which means that the apparent decrease was likely due to sample error.
This Is Not Your Child's Brain After All...
Media accounts trumpeted the alarm -- after the first three years, it's too late for your child! As The Nation reported, "studies indicate that the earliest years of a child's life are critical for developing the brain structures that make it possible to process information, express emotions normally and develop proficiency in language."
But how good was the actual science? In the journal Educational Researcher, the distinguished neuropsychologist John Bruer wrote that this notion "oversimplifies and misrepresents what we now know about critical periods in neural development... There is no evidence that critical periods exist for culturally transmitted knowledge systems...What neuroscientists know... does not support a claim that zero to three is a critical period for humans." The claim that "it's all over at age three" turns out to be based solely on research with rhesus monkeys, which have a different developmental cycle than do human children. (At three years of age, the monkeys are sexually mature.) Bruer concluded, "There may be reasons... to favor Sesame Street over Donkey Kong... but neuroscience does not provide the reasons."
Who's Afraid of Virginia ... Slims?
A recent Harvard School of Public Health survey asked parents to name the "most important health problems children in America face today." AIDS came in first, named by 23% of respondents, followed by other infectious diseases (17%), drugs (15%), smoking (11%) and cancer (10%). fewer than 1% named accidents as a serious health risk. This is a curious ranking, given the health risks that American children actually face. The Post article notes that "smoking, the leading preventible cause of illness in all Americans," ranks relatively low, but more surprising is the degree to which Americans misunderstand the risks posed by less "glamorous" threats like accidents. According to the National Center for Health Statistics, in 1994, 381 children under the age of 15 died of AIDS, 891 died of cancer, and 6,025 died in accidents.
Similarly, a recent survey from the American Heart Association demonstrated that American women consistently perceive cancer, particularly breast cancer, to be their number one health risk. 61% of women in the survey asserted that cancer was their greatest health threat, with 34% specifically citing breast cancer, while only 8% named cardiovascular disease or stroke. Heart disease and stroke are responsible for the deaths of over 500,000 American women every year, breast cancer 44,000.
These misperceptions aren't constrained by national boundaries. Quick -- what are the leading causes of death worldwide? If you said cancer, heart disease or AIDS, go to the back of the class. The number one and two killers in 1990, according to a recent report from the World Health Organization, were lower respiratory infections and diarrhea. AIDS ranked thirtieth. Accidents were ninth.
This would seem to be a case of the journalists getting it right and the readers getting it wrong. But where do people get the information that shapes the way they perceive risk?
Don't Toss Out Your Tea Leaves
In his bestselling book The Bible Code, a new book by former Wall Street Journal reporter Michael Drosnin argues that the Good Book holds secret hidden codes that predict modern events, including the assassination of Israeli Prime Minister Yithzak Rabin, the Gulf War, and the Watergate break-in. These codes can be decrypted by applying a complex statistical "key."
Drosnin was interviewed on numerous television news programs, including CBS and CNN. All this despite the fact that statisticians, cryptographers, and bible scholars have lambasted the book and its methodology.
The Bible Code is a classic example of the pitfalls of "data mining." Given sufficient time and technology, almost any pattern can be "detected" in a given large data set. Business Week's Peter Coy cites the fund manager who "sifted through a United Nations CD-Rom and found that historically, the single best predictor of the S&P 500 stock index was butter production in Bangladesh."
... Or Your Farmer's Almanac
"Global Warming's Good Side" -- Investor's Business Daily, November 13, 1997
These stories tout the benefits of global warming: Milder temperatures and longer growing seasons in northern areas, increased moisture aiding crop growth, etc. But they commit two of the same errors as many pessimistic stories. They assume that climate change is predictable and linear, and they fail to recognize that temperature is just one variable in complex ecosystems. Adjusting the temperature dial on an ecosystem by plus or minus 2% does not guarantee a measurable 2% change to the other components or outputs of that system. It may produce much more dramatic "gains" or "losses" -- or it might crash the whole system.
This Study Just Keeps Going and Going, Younger and Younger....
In April a study published in the journal Pediatrics found that the mean age of onset of menses occurred at 12.2 years for African American girls and 12.9 years for white girls. As the Washington Post reported, this meant that American girls were "developing pubertal characteristics at younger ages than currently used norms," which were based on British girls in the 1950's. But many journalists interpreted the findings as a new trend toward lower ages for puberty. This produced scary headlines like, "Girls Facing the Perils of Puberty Earlier" (Hartford Courant), "Puberty Find Could Point to Danger" (Pittsburgh Post Gazette), and "Girls Hitting Puberty at an Earlier Age; Some Worry Environmental Estrogens Could be Behind a New Study's Findings" (Des Moines Register). These fears of pollution-induced puberty ignore the fact that, as the Post reported, "the age at which girls first menstruate hasn't changed much since 1950."
Whoops, No Apocalypse
NASA's Cassini space probe to Saturn blasted off from Cape Canaveral on October 15 -- yet somehow we're still here. This despite dire claims made by anti-Cassini activists about the probe's plutonium energy supply: "We're on a countdown to nuclear space disaster," warned Karl Grossman in the Albany Times Union and other papers, days before the launch, adding that "What we are talking about here - and I use this word advisedly - is a holocaust in the making." Commentators in print and broadcast news predicted casualty figures ranging from thousands (networks) to "30 to 40 million" (NPR) to "60% of the earth's population ... and that's a conservative estimate" (The Boulder Weekly)
Indeed, the Cassini critics spun out such webs of improbable disaster scenarios involving the "Flying Chernobyl" that NASA had a hard time responding to them. Experts familiar with the astronomical improbability of even the least disastrous scenarios actually occurring were left wondering why major media ran scare stories unsupported by physics or statistical probability.
Down the Memory Hole.... Research too good to be reported on?
Penner, a professor of atmospheric, oceanic and space sciences terms "startling" her findings regarding the cooling effect of carbon aerosols trapped in clouds. The results caused her to re-think her stance. "I had not expected to get such a large negative forcing... if these results hold up, we are going to have to do a lot more work to understand how climate might change in the future." Penner remains concerned over the impact of accumulating greenhouse gases. But she now argues that carbon and sulfer emissions can have "the reverse effect, serving to cool down the planet.... If further research serves to uphold these initial findings, the warming we've seen over the last 100 years may simply be due to natural variability." Though this information was posted on a science journalist's electronic server, to date no journalist has found this research newsworthy.
Down the Memory Hole, II.... Research too bad to be reported on?
"Healthy Start" is a $500 million Health and Human Services (HHS) program which aims to reduce infant mortality rates by 50% in poor communities. An evaluation by Mathematica Policy Research, Inc. was prepared for a meeting of the American Public Health Association.
Then the problems started. It seems that the data, findings from 9 of the 15 total Healthy Start sites, were less than politically satisfying. In fact, according to Mathematica, the program had little to no impact on death rates during its first year. A report by the Philadelphia Inquirer (November 12) quoted a member of the Healthy Start advisory committee, Dr. Milton Kotelchuck, as saying, "In terms of infant mortality, (the) figures are pretty depressing, especially in light of the fact that it's a big program." What is HHS to do when findings are embarrassing? Noted the Inquirer, "Top federal officials have barred researchers from releasing" the report. Claiming that the results were "too preliminary," the presentation was abruptly canceled.
Down the Memory Hole, III....
Mutant Beef: It's What's For Dinner
The debate over irradiating beef to kill dangerous pathogens like E coli stayed in the headlines all year, and all the moo-ing from both sides produced some genuinely frightening pseudo-scientific scares. While the beef industry found itself in a precariously tricky PR spot ("Our beef is so riddled with scary pathogens and microbes that you'd be insane not to irradiate it"), irradiation opponents had the news pasture largely to themselves. In a letter-writing campaign that yielded placements in newspapers around the country, as well as in the article above, anti-irradiation advocates Food and Water warned that irradiating beef left it contaminated with "radiolytic products."
Sounds scary. No one wants to order their burger with an extra helping of radiolytic products. But it turns out that lots of other harmless processes -- like canning, dehydrating, and cooking food -- impart equivalent products. According to the FDA, none of these products has been shown to be dangerous. But for those insufficiently frightened by radiolytics, Food and Water concocted another those microorganisms that were not killed by radiation would have the meat to themselves, and would assumedly grow stronger. By this logic, surgeons shouldn't wash their hands before surgery because by eliminating one type of dangerous bacteria they might be creating fertile grounds for new bacteria.
The X Fools
Time's June 23 cover story, which sported a photorealistic close-up of an "extraterrestrial," typified the goofy science fiction that the media served up as news in 1997. In the same vein, CNN's extensive coverage of the fiftieth anniversary of the "Roswell incident" consistently referred to "the UFO sighting" as fact.
Just harmless fun? A study by Purdue researcher Glenn Sparks found that "when uncritical news accounts of UFO sightings are aired in the news, belief in UFO's increases among the audience." Result: a Yankelovich poll showed that about a third of all Americans now believe that a flying saucer crashed at Roswell.
Kudos to US News and World Report's Art Levine, who decried the attempts by some journalists to achieve "objective" coverage by equally balancing the unsupported claims of UFO true-believers against actual scientists and government sources, labeling this effort "preposterous" and a disservice to society.
Sometimes, it's the little things that mean the most....
"Superbugs Escape Antibiotics' Punch" -- Sacramento Bee, September 12, 1997
According to the Bee, a report that a Michigan man was infected by Staphylococcus bacteria resistant to the most powerful antibiotic "should send shudders through the medical community and the public... If those bacteria spread, the most deadly type of hospital-acquired infection will become untreatable." In this case, the fear is genuine -- but the response may be mis-directed. As Dr. William Jarvis said in the December 17 Journal of the American Medical Association (JAMA), "The emergence of vancomycin-resistant Staphylococcus is a public health emergency." Nosocomial, or hospital-borne diseases, are among the most frequent and deadly sources of emerging infectious diseases, striking about 2 million patients each year and killing 80,000, according to public health experts. What should be done?
The government has put in place a new multi-million dollar global program designed to track epidemiologic trends in bacterial infections, called the SENTRY Antimicrobial Surveillance project. Moreover, according to JAMA, a primary contributor to resistant infections is antibiotic overuse and misuse. But Dr. Jarvis also noted another, and far simpler, "bedrock principle" in infection prevention -- handwashing. "Infections are most often transmitted from patient to patient on the hands of health care workers. If they would just wash their hands between all patient contacts, I wouldn't be standing up here lecturing on prevention."