Ran across this blog post from Powerset concerning the usage of wikipedia by students in higher education. Once you get beyond the plug for Powerset, some rather interesting research results are presented.

Initially, one might be quick to  jump on the results in bold [as is intended]:

 90% of students have used Wikipedia to complete an assignment

73% of students have been explicitly told by their professor not to use Wikipedia.

The first stat is nothing new. Wikipedia is quicker and easier than an actual trip to the library or a institution's online resource as well, students love it.

The second stat is what I found most intriguing. There has to be more to the story. There MUST be intervening variables, covariates, something. In an attempt to gain more insight, and hopefully an actual copy of the study, I shot off a message to the press contact, press@powerset.com. Their reply was basically "we're a media contact for Microsoft, please contact them directly for more information", a brush off. They were more concerned with Microsoft's image in the whole process than actually providing any useful information.

Refusing to accept the response, I went back to the blog post to find out more on my own. First, I took a look at Peanut Labs, who was credited with carrying out the study. In a nutshell [ha!], they are an agency that carries out marketing research, focusing on "Gen Y, Gen X, Boomers and highly targeted consumer segments", via social networks.

Ok....the picture was beginning to get somewhat clearer. This wasn't a piece of scientific research, it was just marketing. Even taking that into consideration, I couldn't help but think does Powerset actually believe their study isn't biased and their results are generalizable?

I tracked down the author of the post, Mark Johnson. Mark was kind enough to send me a spreadsheet containg the questions asked and frequencies of response. This gave more insight to the original post....let's check it out item by item:

about a quarter of students always use Wikipedia when they’re completing an assignment.

This is about right. However, they don't mention that the other 2 response options were "never" and "sometimes." The "sometimes" response carried 66.34% of the votes. When added to 23.76% who always use it, the results is the same as the 90% stat cited above and somewhat redundant in nature.

Not surprisingly, the most common use for Wikipedia is initial research.  However, about half of the respondents said that they use Wikipedia as a link to secondary sources of information.  That suggests that the reference section at the bottom of a Wikipedia article is an extremely valuable starting point for many topics.  Also, a third of students say that they’ve used Wikipedia as a primary research source.

Yep....initial research is 47% Links to secondary [wouldn't those be primary?] sources  came in second at 30 and as a primary source came in with 20%. Although...the percentages in the spreadsheet actually added up to 171%, and 312 people answered this question, but the percentages were correct in the blog.

In any case, this seems to sound a more like what I've seen in non-marketing research. You can use wikipedia for rudimentary research and links, but citing it outright is not often allowed.

Students find Wikipedia very valuable.  28% of students thought that Wikipedia was a very valuable resource, 49% thought that Wikipedia was relatively important, and 23% thought that Wikipedia was only marginally helpful.

This is somewhat deceptive.....here's the actual question:

When you have used Wikipedia for completing assignments, in general how important/relevant/helpful has it been?

Notice no mention of value in the question itself.  In fact, there could have been 3 separate questions for each category (important/relevant/helpful), but instead, the answers contained different terms:

Marginally helpful - 22.53%
A relatively important source of information - 49.45%
A very valuable resource - 28.02%

If they had just kept the question to "how would you rate the information you find on wikipedia?" everything would have been fine!

I finished up boiling the data and asked  Mark Johnson "given the methodology, do you believe these results are generalizable to a different population other than students who use social networks?" Here was his reply:

Hi, Dr. Naegele,

That’s an excellent question, though my understanding is that the majority of students use Wikipedia.  The statistics about the professors would probably remain constant (I don’t see why social networks would influence that), but I could imagine the Wikipedia usage statistics could fluctuate.  You should do a scientific study on this.  I think it would be really interesting!

Cheers,
Mark

Kind of what I expected. Basically, I believe my results and they conform to my needs,  go run your own study. I can't blame him for that response, he's not a scientist...as he states in his blog: "Geeky people usually think I’m a dumb marketing guy and dumb marketing people usually think I’m a geek."

There's nothing wrong with Powerset or the service it provides and the stats from their marketing research wouldn't have been posted if they hadn't supported their business model. Still...given the methodology, sample size, and sample demographics, the report is quite deceptive.