The Cortical Column

Note: This entry was initially going to be a comment to the Cognitive Daily post, “Is there wisdom in crowds?” But as you can see, it is far too long to be a comment

The precursor to Wikipedia was Nupedia. It was run by Larry Sanger, a proper epistemologist. It had a seven point review process, as stringent as the Encyclopedia Britannica, if not moreso. Over three years, they finished around 25 articles. Relative to Wikipedia, Nupedia was a dismal failure. There are many lessons to be learned from this. Wikipedia is successful in part because it is able to tap into the wisdom of the crowds, as discussed at Cognitive Daily. It is also true that their is a sort of evolutionary process at work; memes, if you will. A researcher from UC Santa Cruz recently came up with a metric of judging text quality called Trust (http://trust.cse.ucsc.edu/). Although a complicated algorithm, what it boils down to is this: If you are a piece of text that is written by an author who tends to write text that tends to stay in the encyclopedia for a long time, and you have been in the encyclopedia for a long time, then you have a high Trust score. Luca has provided an interesting demo at the above URL, which you should try.

I have also done some work on predicting the quality of Wikipedia articles (pdf). This is a difficult machine learning task because of a particular point that I would like to add to this conversation: The encyclopedia is mostly unwritten. There is, as of yet, very little high quality text to train on. It is a work in progress! If you click on the Random Article link, you have a roughly 75% chance of landing on a “stub,” which is a very short article, that has yet to receive significant community attention. Keep in mind that the encyclopedia was started in the year 2000, and is fast approaching ten million articles in roughly 250 languages. The combined fortunes of Bill Gates and Warren Buffet could not have hired enough people to write as much as they have. And yet, the encyclopedia is mostly unwritten.

Nobody knows better the failings of Wikipedia than the Wikimedia Foundation (WMF), who run the encyclopedia, as evidenced by the Wikipedia article on “Criticism of Wikipedia.” In my view, quality control in an Internet-scale collaborative authoring system, with anonymous authors, is an unsolved problem. There is no one researcher that can solve this problem. Somewhat akin to the game Nomic, the rules of governance and quality are ever changing, albeit collaboratively determined, in much the same way that the encyclopedia is written. I see no reason to discredit such a process before it has had a reasonable amount of time to run its course. None of us knows what will emerge from it, but it is certainly exciting. New experiments in quality control methods are being conducted by the WMF, but they are just that - experiments.

Epistemologists love to sit around and critique the encyclopedia. I’m sure it’s no coincidence that it tends to have low quality philosophy articles. But its merits seem to outweigh any significant criticisms. During midterms and finals, walk into a busy computer lab on your campus and take a brief census of the number of students scanning Wikipedia articles. We use it as a first source of information on our lab mailing list, I have caught professors giving lectures based on information in Wikipedia articles, and I use it myself and find it immensely useful. While there is certainly significant difficulty in judging the veracity of the text contained within, students should not be taught to avoid the encyclopedia and go find an ‘authoritative’ primary or secondary source. They should be taught to judge critically all of the information they digest. Educators and institutions from the previous generations who are not yet on board with the 21st century may never have this realization. But the next generation of students, who are growing up without having a good reason to reference a printed encyclopedia such as Britannica, will recognize how valuable it is. In fact, they won’t recognize Wikipedia as being any different from anything else, except perhaps, more useful.

If anything, Wikipedia is disillusioning. It is removing the bias in our educational system that, when you want “facts,” you should go to the library. Because everything you find written there is true. This isn’t the exact message being taught in grade school, but it’s certainly the impression I had growing up, before I knew any better. I suspect I’m not alone… As long as we continue using it, the evidence seems to be that it will only get better. Britannica, on the other hand, is for the most part stagnant. Once a factual error has been printed, it can never be corrected. So why not continue by supporting the encyclopedia that was started with the goal of making all human knowledge free, for all humans? That’s what I plan to do…