Mozilla’s data project could be useful, but still biased.
UPDATED Wed May 14 16:55:17 EDT 2008
Online metrics are hard enough when you’ve limited it to just a specific site with all the help the metric measurer could want. It becomes orders of magnitude harder to try and figure out this information internet wide. It seems like Mozilla is planning on stepping into that ring.
From what TechCrunch reports it looks like they plan on being a competitor to Alexa and Compete. Both terribly, terribly flawed stats. They both work using the panel method - people install a little software on their computer and their web visits are tracked. Elves then use their sorcerous ways and a little statistical voodoo to come up with the publicized stats.
This is similar to how Nielsen does it’s TV ratings, taking a sample of the US population and extrapolating countrywide viewing from that. Arguably, the Nielsen’s though are significantly more accurate because they go to lengths to make sure that their panel members are a more statistically random set of people in the population. So theoretically, all types of folk are represented and so their extrapolation is more meaningful. On the other hand, Alexa and Compete are voluntary - you decide all by yourself to go and be counted - so in fact they get a skewed picture of web traffic. Alexa, as an example is thought to exaggerate the traffic going to SEO type sites - since a lot of people who decide to install the software are interested in SEO.
So where does Mozilla fit into the picture? According to the reporting it is going to be something that will be built into Firefox and will ask its user base if they want to opt into this anonymous program. Given the auto-upgrading joy of Firefox it is reasonable to believe that a very significant portion of Firefox users will get this version as it becomes available - so most of the currently 170 million users will have the choice. if even a small percentage of people opt into the program it will be quite a large sample set to extrapolate from - Lilly says even 1% - 1.7 million - would be enough. Alexa doesn’t publish their numbers but I found an old post that suggests that Alexa’s got 20 million members that gets qualified down to about 2.5 million on the panel - which suggests to me that in fact Mozilla will need much more than 1% to work.
Assuming that they do get more, which seems reasonable to believe, what about the bias? It seems to me that while there will definitely be less of an obvious bias than with Alexa - since it will be built into the browser instead of a download that you have to go and install. It still remains that installing Firefox itself creates a bias. That is, at this point in life both Microsoft and Apple have perfectly useful (although, sometimes frustrating) browsers that ship with their OS’s. The people who decide that those browsers aren’t good enough and go through the trouble to download and install Firefox, well that’s kind of a very specific group of folks. Another example would be that it excludes, most likely, whole swathes of corporate america who have standardized on IE.
The difficulty with these panel based models is that you need to get a statistically random group of members. None of these options (Alexa, Compete or, as far as I can tell, Mozilla) have any semblance of looking for that - it’s strictly (and obviously necessarily) a voluntary action where the future member goes out, unsolicited, and downloads his membership. I’d say that they all provide something interesting, but I wouldn’t bet the farm on any of them being consistently accurate images of these stats.
UPDATE: TechCrunch has a few more details. Mostly just talk about intention, which seems to be the stage the project is at. Still, having a public database of that data, biased as it may be, could be quite valuable - as opposed to public access only to the fruits of such data like Alexa provides. I’m definitely interested in seeing how this takes shape.
