2009 | 2008 | 2007 | 2006 | 2005 | 2004 | 2003

STATS "Dubious Data" awards 2008

By STATS staff, December 2008
Honoring some of the worst abuses of statistics and science in the past year.

Baby Bottle Baloney
Throughout 2008, parents were treated to alarming chemistry lessons from the press about plastic containing bisphenol a (BPA), which, among other uses, helps to make polycarbonate plastic shatter proof. According to the press, BPA is highly toxic, an endocrine disruptor, and may be responsible for a host of illnesses and even obesity. USA Today, the Milwaukee Sentinel Journal, and many other media organizations repeatedly turned to biologist Frederick Vom Saal to explain why BPA is so dangerous, noting that he was a leading expert on the chemical.

What the media failed to tell the public is that Vom Saal’s research has been rejected by not one, but two major risk assessments on BPA, both of which were carried out by independent scientists and both of which found no evidence of a risk to adults or infants. In February, the independent, non-profit consumer investigative group NSF International published a risk assessment explaining why Vom Saal’s research methods were flawed, irrelevant to human safety, and contradicted by more statistically rigorous research. In July, the European Food Safety Agency both reaffirmed the safety of BPA and criticized regulators in the U.S. and Canada for giving too much credence to research on BPA that was “limited in rigour, consistency and biological plausibility.”

Treats, Shoots and Misses
In July, the Commonwealth Fund published its second evaluation of U.S. health care, Why Not the Best?: Results from the national scorecard on U.S. Health System Performance, 2008. The results were dismal, the U.S. was outscored on a range of different measures by other countries,  leading the New York Times to write that “The findings are likely to provide supporting evidence for the political notion that the nation’s health care system needs to be fixed.” What the media failed to report was that many of the scorecard’s conclusions were not drawn from patient outcomes (they got better or worse) but from surveys of patient satisfaction (they thought they got better or worse). In other words, patients with no medical training decided whether the treatment they received was mistaken or unnecessarily repetitious. Add this to the fact the scorecard mashed together studies without adjusting for different methodologies, sample sizes, and collection techniques, and you arrive, ironically, at a diagnosis that simply looks like it’s scientific but isn’t.

And the “Shameless Exploitation of Children for Shock Value In the Furtherance of Misinformation” Award Goes To….
…The Physicians Committee for Responsible Medicine. This group, which anyone can join for $20 dollars, has been denounced by the American Medical Association for “inappropriate and unethical tactics” and for giving medical advice that was “irresponsible and potentially dangerous to the health and welfare of Americans.”

So when the group launched a YouTube advertisement in July with young children talking directly to the camera about getting cancer from hot dogs – “I thought I would live forever,” said one child, “I was dumfounded when the doctor told me… I have late-stage colon cancer.” – you might have expected a firestorm of controversy to ensue.  Wrong.

Though CNN noted that the ad was the work of “an animal rights group that wants us all to be vegans,” it also told viewers that there was “research” to support the link between hot dogs and cancer. “The problem experts believe it is in nitrites in the processed meats,” said CNN’s Elizabeth Cohen. “That's apparently what's causing the cancer link.” (Nitrites are present in salt and used in meat to prevent spoilage and protect against botulism. They are especially important for meats cooked at relatively low heat, such as hot dogs), 

What the CNN report did not relay is that there are no substantiated links between nitrites and cancer, according to the American Medical Association. Or that we get 90 percent of our intake of nitrites from vegetables.

Rainfall and Autism: The Sound of Statistical Noise
There was a flood of media coverage of a study by an economist claiming that counties with high levels of rainfall in the Pacific Northwest had higher rates of autism among children. “Research published today suggests that regular rainfall may help trigger autism in children,” said USA Today. Largely missing in flurry of speculation over what accounted for the findings was that the study didn’t, in fact, report a direct link between rainfall levels and autism. Instead, they found a link between a “relative preciptiation variable” and autism. Relative precipitation is a measurement of how far a county’s rain level is from the average level of precipitation. So the study is in fact saying that when rain levels are “out of the ordinary,” autism is more likely to develop or be diagnosed. And the problem with that claim is that autism diagnoses have increased in all climates.

Toxic Shower Curtain Smell
If you hated the shower scene in “Psycho” the Los Angeles Times had even more disturbing news for you in June: “Vinyl shower curtains sold at major retailers across the country contain toxic chemicals that may cause serious health problems,” reported the paper citing a study by The Center for Health, Environment and Justice. The group had found “high concentrations of phthalates and other chemicals whose “potential health effects included developmental damage as well as damage to the liver and central nervous, respiratory and reproductive systems.”

But if you read the actual report the group admits that the laboratory it used “was not able to achieve the lower detection limits necessary to identify phthalates off-gassing to the air.” In other words, the study found phthalates in the shower curtains but couldn’t find phthalates coming off the shower curtains. The study was also unable to show that the levels of chemicals released from shower curtains have actually been shown to pose a risk. The group cited American Lung Association warnings about indoor pollution, but the ALA doesn’t actually mention a risk from shower curtains or, for that matter, vinyl.

Acupuncture and Fertility: The Media Screws Up
In February, news organizations, including Fox News, MSNBC, The Examiner, and The Washington Times, all declared that “fertility benefits from old remedy.” According to a new study in the British Medical Journal, scientists had shown, conclusively, that acupuncture fosters successful pregnancies. What the study actually said was that the theory is “far from proven.” In the seven studies conducted for the paper, only three found the effect of acupuncture on IVF patients significant. One of the studies found no relation between IVF success and acupuncture, and the remaining three “found a trend toward benefit.”  The authors pooled the seven studies in a meta-analysis to find that the odds ratio – the odds of pregnancy through IVF and acupuncture divided by the odds of pregnancy through IVF without acupuncture – was 1.65. Here the word ‘odds’ is used in the statistical sense, not the layman meaning. But the media reported this figure as if acupuncture increases the success rate of IVF by 65 percent. The actual increase in the likelihood of pregnancy according to this meta-analysis is about ten percent.

And the “Larry Summers-This Could Get You Fired From Harvard Award” goes to…
...The Boston Globe. In May, the Globe  reported on the latest research about the lack of women in science and engineering, and decided that it means women were embracing “the freedom to say ‘no.’” As in saying “no” to careers in science.


Well, not really. To make the claim that women are just not interested in science stick, the Globe excluded medicine and biology, where the gender gap has disappeared over the past 50 years. In fact, 60 percent of biology degrees now go to women, and recent data show 49.5 percent of first-year medical students in 2004 were women, and that they formed a majority of applicants.

In chemistry, another subject relevant to medicine, 41.5 percent of BAs went to women in 1996. The number of women entering medical school increased five-fold between the 1960s and 1980s, a period that saw not only tremendous gains in equality, but also the social acceptance of women pursuing science.

First, Do the Data No Harm
A survey conducted by the Physicians' Foundation finds that almost half of all practicing physicians in the US plan to cut back or quit practicing medicine. According to the survey’s press release, “49 percent, or more than 150,000 practicing doctors– say that over the next three years they plan to reduce the number of patients they see or stop practicing entirely." But it turns out only four percent of respondents actually filled out a questionnaire (11,960 out of 270,000 mailed surveys) – begging the question of whether doctors closer to retirement, or unhappy doctors, are more likely to fill out a questionnaire to begin with. Without a representative sample of doctors, the survey only tells us something about the four percent who completed the survey, not everyone. The statistics professor who advised the group that there was a one percent margin of sampling error should consider renaming his department "bullistics".

"Dublin Inhabited by Little People"
The story didn't say that exactly, but you'd have to be pretty darn small to live in Dublin if the math-challenged New York Times is to be believed. In September, the Times told readers that to call Dublin, the capitol of Ireland, a city, "is something of a misnomer. A million people live inside Dublin's official borders, which are such that you can literally walk anywhere in about half an hour."

Actually, you'd be walking nowhere fast if Dublin were both that small and that densely populated. Assuming you could cover two miles in 30 minutes Dublin would occupy a circular area with diameter no longer than 2 miles. That means the area of the city would be approximately 3.14 square miles; with a million residents, that comes to a staggering 318,000 people per square mile.  That would make Dublin one of the most densely populated cities in the world, despite having few residential building higher than six stories. Manhattan, by comparison, has 66,940 people per square mile – and skyscrapers.

According to the 2006 Irish census data, Dublin City has a population of 505, 379, and the city boundaries encompass a much more generous 45.5 square mile

Giving the finger to Pixie Dust
In April, the BBC claimed a “worldwide exclusive” on the story of Lee Spievack, a hobby store salesman who had an unfortunate accident with a model plane. "I put my finger in," Mr Spievak told the BBC, pointing towards the propeller of a model aeroplane, "and that's when I sliced my finger off. It took the end right off, down to the bone, about half an inch. We don't know where the piece went." Thanks to the application of what Mr. Spievak called “pixie dust,” a powder extract of pig’s bladder labeled “extracellular matrix” and sent to him by his surgeon brother, he grew his finger back.

The news had already been reported in the U.S. by CBS in February, when the Evening News with Katie Couric announced that the “holy grail of healing” had been found: “You might become a believer in the power of magic dust,” said reporter Andrew Wyatt, “when you see how a special powder re-grew the tip of Lee Spievack's finger,” (In fact, the miracle had been briefly reported by the AP in 2007)

But when Dr. Ben Goldacre, the Guardian newspaper’s “bad science” columnist and a full-time medical doctor looked at a photo of Spievack’s wound prior to treatment, he observed a slight problem: “there is no missing finger.” Spievack had simply lost some flesh from the tip. As an embarrassed BBC, reported several days later, citing an expert in hand surgery, finger tips can actually grow back all by themselves. Even more embarrassing, the expert in “regeneration” cited in all the stories turned out to be chief scientist for the company Spievack’s brother owns.


Technorati icon View the Technorati Link Cosmos for this entry