Nature438, 900-901
(15 December 2005)
| doi:10.1038/438900a
Special Report Internet encyclopaedias go head to head
Jim Giles
Abstract
Jimmy Wales' Wikipedia comes close to Britannica in terms of the accuracy of its science entries, a Nature investigation finds.
One
of the extraordinary stories of the Internet age is that of Wikipedia,
a free online encyclopaedia that anyone can edit. This radical and
rapidly growing publication, which includes close to 4 million entries,
is now a much-used resource. But it is also controversial: if anyone
can edit entries, how do users know if Wikipedia is as accurate as
established sources such as Encyclopaedia Britannica?
AP PHOTO/M. PROBST
Several
recent cases have highlighted the potential problems. One article was
revealed as falsely suggesting that a former assistant to US Senator
Robert Kennedy may have been involved in his assassination. And
podcasting pioneer Adam Curry has been accused of editing the entry on
podcasting to remove references to competitors' work. Curry says he
merely thought he was making the entry more accurate.
However, an expert-led investigation carried out by Nature
the first to use peer review to compare Wikipedia and Britannica's
coverage of science suggests that such high-profile examples are the
exception rather than the rule.
The exercise
revealed numerous errors in both encyclopaedias, but among 42 entries
tested, the difference in accuracy was not particularly great: the
average science entry in Wikipedia contained around four inaccuracies;
Britannica, about three.
Considering how Wikipedia
articles are written, that result might seem surprising. A solar
physicist could, for example, work on the entry on the Sun, but would
have the same status as a contributor without an academic background.
Disputes about content are usually resolved by discussion among users.
But
Jimmy Wales, co-founder of Wikipedia and president of the
encyclopaedia's parent organization, the Wikimedia Foundation of St
Petersburg, Florida, says the finding shows the potential of Wikipedia.
"I'm pleased," he says. "Our goal is to get to Britannica quality, or
better."
Wikipedia is growing fast. The
encyclopaedia has added 3.7 million articles in 200 languages since it
was founded in 2001. The English version has more than 45,000
registered users, and added about 1,500 new articles every day of
October 2005. Wikipedia has become the 37th most visited website,
according to Alexa, a web ranking service.
But
critics have raised concerns about the site's increasing influence,
questioning whether multiple, unpaid editors can match paid
professionals for accuracy. Writing in the online magazine TCS
last year, former Britannica editor Robert McHenry declared one
Wikipedia entry on US founding father Alexander Hamilton as "what
might be expected of a high-school student". Opening up the editing
process to all, regardless of expertise, means that reliability can
never be ensured, he concluded.
Yet Nature's
investigation suggests that Britannica's advantage may not be great, at
least when it comes to science entries. In the study, entries were
chosen from the websites of Wikipedia and Encyclopaedia Britannica on a
broad range of scientific disciplines and sent to a relevant expert for
peer review. Each reviewer examined the entry on a single subject from
the two encyclopaedias; they were not told which article came from
which encyclopaedia. A total of 42 usable reviews were returned out of
50 sent out, and were then examined by Nature's news team.
Only
eight serious errors, such as misinterpretations of important concepts,
were detected in the pairs of articles reviewed, four from each
encyclopaedia. But reviewers also found many factual errors, omissions
or misleading statements: 162 and 123 in Wikipedia and Britannica,
respectively.
D. I. FRANKE/WIKIMEDIA FDN
Kurt
Jansson (left), president of Wikimedia Deutschland, displays a list of
10,000 Wikipedia authors; Wikipedia's entry on global warming has been
a source of contention for its contributors.
Editors
at Britannica would not discuss the findings, but say their own studies
of Wikipedia have uncovered numerous flaws. "We have nothing against
Wikipedia," says Tom Panelas, director of corporate communications at
the company's headquarters in Chicago. "But it is not the case that
errors creep in on an occasional basis or that a couple of articles are
poorly written. There are lots of articles in that condition. They need
a good editor."
Several Nature reviewers
agreed with Panelas' point on readability, commenting that the
Wikipedia article they reviewed was poorly structured and confusing.
This criticism is common among information scientists, who also point
to other problems with article quality, such as undue prominence given
to controversial scientific theories. But Michael Twidale, an
information scientist at the University of Illinois at
Urbana-Champaign, says that Wikipedia's strongest suit is the speed at
which it can updated, a factor not considered by Nature's reviewers.
"People
will find it shocking to see how many errors there are in Britannica,"
Twidale adds. "Print encyclopaedias are often set up as the gold
standards of information quality against which the failings of faster
or cheaper resources can be compared. These findings remind us that we
have an 18-carat standard, not a 24-carat one."
The
most error-strewn article, that on Dmitry Mendeleev, co-creator of the
periodic table, illustrates this. Michael Gordin, a science historian
at Princeton University who wrote a 2004 book on Mendeleev, identified
19 errors in Wikipedia and 8 in Britannica. These range from minor
mistakes, such as describing Mendeleev as the 14th child in his family
when he was the 13th, to more significant inaccuracies. Wikipedia, for
example, incorrectly describes how Mendeleev's work relates to that of
British chemist John Dalton. "Who wrote this stuff?" asked another
reviewer. "Do they bother to check with experts?"
But
to improve Wikipedia, Wales is not so much interested in checking
articles with experts as getting them to write the articles in the
first place.
As well as comparing the two encyclopaedias, Nature surveyed more than 1,000 Nature
authors and found that although more than 70% had heard of Wikipedia
and 17% of those consulted it on a weekly basis, less than 10% help to
update it. The steady trickle of scientists who have contributed to
articles describe the experience as rewarding, if occasionally
frustrating (see 'Challenges of being a Wikipedian').
Greater
involvement by scientists would lead to a "multiplier effect", says
Wales. Most entries are edited by enthusiasts, and the addition of a
researcher can boost article quality hugely. "Experts can help write
specifics in a nuanced way," he says.
Wales also
plans to introduce a 'stable' version of each entry. Once an article
reaches a specific quality threshold it will be tagged as stable.
Further edits will be made to a separate 'live' version that would
replace the stable version when deemed to be a significant improvement.
One method for determining that threshold, where users rate article
quality, will be trialled early next year.
Additional research by Declan Butler, Jenny Hogan, Michael Hopkin, Mark Peplow and Tom Simonite.