Toys containing phthalates have been banned for fear that infants will put them in their mouths. People have been warned that chemicals producing a “new car smell” can poison them, and that even sunlight warming the plastic in a baby stroller can endanger their babies from toxic off-gassing. Activist groups have warned of chemical perils in iPods, air fresheners, pizza boxes, lipstick, perfume, window blinds, mattresses, and sunscreen.

These warnings have generated thousands of news stories. But in all the media coverage over the past few years, and for all the scientists who were quoted on one side or the other, the community of experts who study the toxic risks of chemicals were never canvassed for their collective informed opinion on how much the public really was at risk.

Surveying Expert Opinion
In the past we had surveyed expert communities on the environmental sources of cancer, the risks of nuclear energy, and the likelihood and direction of climate change. Amid heightened public concern over manmade chemicals, it seemed an opportune time to survey the expert community on this topic – the science of toxicology, which focuses on the adverse effects of chemicals on living organisms.

In order to determine the collective judgments of toxicologists on chemical risks, we asked the Society of Toxicology (SOT), the professional association of this scientific discipline, for permission to survey their members. The SOT supplied us with a list of full members of the organization, with the understanding that this did not constitute an institutional endorsement of the study’s methodology or findings. Among the criteria for full membership are several years of professional experience in toxicology.

We created an online questionnaire with the assistance of Harris International, a prominent international survey research firm and an industry leader in online polling. Respondents were contacted by email requesting their participation. They were given passwords with which to log onto the questionnaire. From January 27 through March 2 we contacted 3562 SOT members, 1136 of whom responded, for a return rate of 32 percent. However, almost 200 of these filled out only part of the questionnaire, and many of these provided demographic information but skipped the key attitude questions. This initial presentation of our findings is based on the responses of the 937 who responded to every question.

We inquired into four different areas of toxicologists’ attitudes, perceptions, and opinions about issues related to chemical risk. First, we asked whether they agreed or disagreed with a number of statements about the safety of currently used chemicals, the process of determining their safety, and the basis for making scientific judgments and regulatory decisions. Second, we asked them to rate the risk to human health posed by current levels of exposure to a list of chemical substances that have spurred controversy.

Third, we asked them to rate the quality of information about chemical risk associated with a wide variety of government, nonprofit, and private sector organizations that frequently address this issue. Finally, we asked their opinions on media coverage of chemical risk, including the media’s ability to explain scientific issues in a way that will help audiences reach their own conclusions.

The overall findings on all these questions are provided in the attached tables. The exact wording of the items discussed below are presented in Appendix A. We are continuing to analyze these data and will present additional results in scholarly journals.

Video highlights from the
National Press Club discussion

"This press conference is long overdue" - Michael Holsapple, Ph.D, Vice President of the Society of Toxicology.

"There is a long history of controversy over how the media cover health risks" - S. Robert Lichter, Ph.D, President of STATS gives an overview of the survey findings.

"What news source did best?" - S. Robert Lichter, Ph.D, President of STATS reveals how new media have displaced the old.

"This is a tremendously important social issue" - Gary Kreps, Ph.D, Director of the Center for Health and Risk Communication at GMU.

Lousy studies make news - Steve Ross, author, editor and journalism educator on how surprising findings are frequently not reliable findings.

"Anyone know how bandages are sterilized?" - Steve Ross, author, editor and journalism educator on how to communicate with the press.

Linda Greer, Ph.D, of the Natural Resources Defense Council (NRDC) challenges STATS President Dr. Lichter, saying the study should not have been released without being peer-reviewed (even though the results of most surveys, such as political polls, are released without peer review). 79 percent of the toxicologists surveyed were critical of the NRDC.

Specific chemicals
We then presented respondents with a list of specific substances and asked them to rate the risk to human health posed by current levels of exposure to a list of specific substances to each, on a scale from very low to very high risk. Table 2 shows the results in terms of both mean scores and proportions reading each substance as high risk. For example, 89 percent of toxicologists rate smoking tobacco as a high-risk activity, 44 percent regard second-hand smoke as high in risk to health, and 37 percent say exposure to mercury as a high risk. Next in line are exposure to sunlight, rated as high risk by 26 percent, and aflatoxin, a naturally occurring fungus found in peanut butter, rated as high risk by 29 percent.

In comparison, toxicologists rate certain chemicals that have generated considerable public controversy as significantly less dangerous to human health. Phthalates, which are added to plastic products to make them flexible, including many children’s toys, are rated as high risk by just 11 percent of respondents. High fructose corn syrup, seen by many people as a cause of obesity was also rated as high risk by 11 percent.

Bisphenol A, or BPA, which is used to harden plastics, and was recently discontinued by makers of baby bottles, was rated as high in risk to human health by 9 percent. Despite recent controversy over the safety of Teflon coatings, it is rated as a high health risk by just 3 percent of toxicologists. Similar results were obtained for several other magnets of public controversy, from flame retardants to genetically modified organisms.

Getting accurate information
In addition to their own views on chemical risk, we asked toxicologists to rate the organizations involved in public debate over chemical risks in terms of how accurately they portray these risks. The results appear in Table 3. There were considerable variations in the number of respondents who were familiar enough with the various organizations to rate their accuracy. To insure that the comparisons are commensurable, the percentages exclude “don’t know” responses. We added a column indicating the percentage of respondents who rated each organization. The table includes only organizations rated by more than one-third of respondents. Organizations failing to meet this level of recognition included the Biotechnology Council, National Nanotechnology Initiative, Pew Charitable Trusts, and American Council of Science and Health.

There were considerable variations in the number of respondents who were familiar enough with the various organizations to rate their accuracy. As a result, some apparent differences in perceptions of accuracy were artifacts of the proportions who expressed no opinion. To insure that the comparisons are commensurable, the percentages in Table 3 exclude “not sure” responses. We added a column indicating the percentage of respondents who rated each organization. In addition, we present the same data with the “not sure” responses included in the rating percentages in Table 3A. (Please note, the full names associated with these abbreviations and acronyms are listed in Appendix A.)

Among respondents who rate these organizations, large majorities view the leading environmental groups as overstating risk. 96 percent believe Greenpeace overstates chemical risk, 85 percent say the same of the Environmental Defense Fund’s risk portrayals, as do 80 percent of those rating PETA. 79 percent believe that chemical risk is overstated by the Environmental Working Group, the Natural Resources Defense Council, and the Center for Science in the Public Interest.

Conversely, smaller majorities see industry related groups as understating chemical health risks. This includes 57 percent of those rating the American chemistry Council, which represents the chemical industry, and 60 percent of those rating PhRMA, the Pharmaceutical Researchers and Manufacturers of America.

By contrast, majorities rate most government agencies and all professional associations as providing mainly accurate portrayals of chemical risk. An exception is the Environmental Protection Agency, which is rated as overstating risk by 41 percent, accurately stating risk by 40 percent, and understating risk by 19 percent. But increasingly large majorities see accurate risk portrayals coming from such agencies as OSHA, the FDA, the CDC, and the National Science Foundation, whose portrayal of chemical risk is rated as accurate by 85 percent of toxicologists.

At the opposite end of the reliability scale are the news media, which are seen as overstating risk to an even greater degree than the environmental groups. Public broadcasting does best among the mainstream media with “only” two out of three toxicologists describing PBS and NPR as overstating chemical risk. Over 80 percent see America’s leading newspapers, news magazines, and health magazines as overstating chemical risk, and the proportion rises above 90 percent for both broadcast and cable television networks.

New media trumps old
In perhaps the most surprising finding in the entire study, all these national media outlets are easily eclipsed by two representatives of “new media” – WebMD and Wikipedia. WebMD is the only news source whose coverage of chemical risk is regarded as accurate by a majority (56 percent) of toxicologists, closely followed by Wikipedia’s 45 percent accuracy rating. By contrast, only 15 percent describe as accurate the portrayals of chemical risk found in the New York Times, Washington Post, and Wall Street Journal. The preference for Wikipedia in particular seems like an indictment of professional journalism, since anyone can contribute to this site.

Figure 1 presents this information in a more compact form my arraying all the organizations that were rated according to their mean scores along a spectrum from “strongly understates” (scored as 1) to “strongly overstates” ( scored as 5). The government agencies and professional bodies are clustered near the midpoint of 3, while the media outlets and environmental groups cluster together almost interchangeably from 4.0 to 4.3.

The only exceptions are public broadcasting, whose 3.8 rating represents a slightly lesser degree of overstating risk, and Greenpeace, whose 4.5 rating (representing slightly more overstatement) is the highest in the study. Of course the two industry organizations, PhRMA and the American Chemistry Council, are rated as understating risk. Perhaps surprisingly, however, their 2.3 ratings put them considerably closer to the midpoint (3.0, representing an “accurate” appraisal of risk) than any of the environmental groups and any of the traditional media outlets except for public broadcasting.

Scientific illiteracy
The disdain that toxicologists apparently feel toward traditional journalism is evidenced by their unwillingness to credit the media with getting almost anything right in covering chemical risk. Table 4 shows that nine out of 10 fault the media for not seeking out diverse scientific views to balance stories, and it only gets worse from there. At least 95 percent describe the media’s performance as “poor” in distinguishing good from bad studies, distinguishing correlation from causation, explaining the trade-off between risks and benefits, distinguishing absolute from relative risk, explaining the odds ratios, and explaining that “the dose makes the poison” – a fundamental tenet of toxicology.

The Internet - a sober corrective to unruly journalists?
A decade ago, the Internet was seen by professional journalists as the “Wild West” of news and information, carefree to the point of lawless – and in need of marshalling for accuracy and reliability. Now, it seems that the positions have switched, at least in terms of scientific information, and it’s the Internet that is providing sobriety and balance to a chronically careless and sensationalistic mainstream media.

The survey results showing that WebMD is the only news source rated as accurate by a majority (56 percent) of toxicologists for covering the risks of chemicals, followed by Wikipedia (45 percent), whereas only 15 percent described similar coverage in the national print media (i.e., the New York Times, Washington Post, and Wall Street Journal) as accurate. This figure dropped to 6 percent for USA Today and 5 percent for broadcast network news

At a press conference at the National Press Club to release the preliminary results of the study, Dr. S. Robert Lichter, described the Wikipedia finding as an indictment of the mainstream media - " it's disturbing that someone off the street seemingly can do a better job than the media."

But the result doesn’t surprise Andrew Lih, journalist and author of the “Wikipedia Revolution: How a Bunch of Nobodies Created the World’s Greatest Encyclopedia.”

“This reminds me of the Nature study that was done in December 2005 where it found that on average, Britannica had 3 errors per article, and Wikipedia had 4 errors,” Lih says by email. “It was surprising because Wikipedia did much better than expected, given its foreign work process and Britannica did much worse. People had presumed a certain level of accuracy from Britannica's reputation, and it was knocked down from that pedestal. To me the WebMD and Wikipedia results here are similar – they're much closer than what one would expect. Wikipedia doing better, WebMD doing worse.”

[Update, June 16, 2009, Britannica reminds STATS that it and others refuted the study - see Britannica versus Wikipedia]

With the mainstream press financially battered by the Internet’s flattened, impecunious economic model, the news that the Internet appears to have created a more successful model for purveying accurate information than traditional journalism seems doubly cruel. But to Alissa Quart, author, contributing editor/columnist at the Columbia Journalism Review, and recent recipient of a Niemen Fellowship at Harvard, the results denote a positive trend.

“Maybe the toxicologists are right in their assessment of Wikipedia, maybe not,” says Quart. “But at any rate, they will eventually get what they need online. A hierarchy of authority is going to reassert itself on the Internet – the wide, open, amateur planes will have a ruling class, a scrim of experts who will act like institutions.”

Such expertise has a fundamentally different motivation to that of a journalist. Think of it as a clash of narratives. As Quart notes,

"Journalists fall into storylines, because that’s how we write. There are three narratives, that we use, which can make us great but also get us into trouble – one narrative to please our editors, one to please our readers, and one which leans toward our sources, because we identify with them. WebMD and Wikipedia contributors are disconnected from most of those narratives – maybe they are trying to please certain readers, but they aren’t ‘the reader.’ Their model of knowledge doesn’t ask for stories, or sentiment or people.

In short, argument trumps aesthetics. Lih, an engineer by education, concurs. The clash of narratives “also says something about motivation, in that the mainstream press will be driven by reports, PR bring shoved at them, and also the market and the desire by editors (in a top-down manner) to demand reporters find a story in the latest research, even if in the greater context of the field, it doesn't warrant so much attention. In that sense, Wikipedia's motivations are different, in that the ‘crowd’ helps moderate and even dampen the type of ‘recentism’ that is so pervasive in news coverage.”

WebMD is a hybrid of conventional reporting with what might be seen as an added scrim of 'expert' crowd sourcing. As Michael Smith MD, Chief Medical Editor of WebMD says via email, "we have an in-depth news process of choosing appropriate studies for reporting, including critical review of studies by board-certified physicians. In addition, we provide a wide array of health information, including news, features, videos, and reference, all of which is reviewed by a physician prior to appearing on WebMD.com. Not only does this help us identify study weaknesses, it helps assure that WebMD readers are getting the most accurate and up to date health information to help them in making important health decisions with their doctor."

In other words , WebMD places far more emphasis on consensus – the weight of evidence on a given topic at any given time in the medical community or medical specialization – over what is ‘new,’ the focus of most reporters. In a sense, this is much closer to the ‘science’ of journalism Joseph Pulitzer hoped to create by founding a journalism school at Columbia than the journalism that Pulitzer’s papers actually practiced. So in many ways, the Internet is speeding us back to the Enlightenment, a place where rationality is king, and where journalism, as the philosopher David Hume saw it, was a new way of doing philosophy.

Of course, the survey findings overwhelmingly show that the aesthetics of narrative persuasion, journalistic formula, and of ‘recentism’ are still very much in the ascendant in public and political debate. It’s clear that if toxicologists were in charge of communicating risk to the public, the news would be different in substance and in style. Whether the news would be persuasive is another matter; style is, after all, a matter of aesthetics. But the survey points to a signal shift in the status of the Internet, with the threat to the mainstream media to adapt to a more “scientific” way of doing journalism, of rebalancing ‘the new’ with consensus, or to risk losing what remaining credibility and relevance it has left.

- Trevor Butterworth