It sucks when your Web site’s traffic isn’t being measured correctly.
It also sucks when you’re trying to measure the significance of someone else’s site, and are getting conflicting signals.
Here’s what we’ve learned over the past few days, after our initial piece on the problems of Alexa, Quantcast and Compete, all sites that independently verify how much traffic a site is getting.
We’ve learned that if a measuring company doesn’t have a tracking pixel directly on the Web site it is tracking, the numbers are extremely unreliable.
This may be nothing new for insiders, but the public data provided by Alexa, Quantcast and Compete can be misleading for the rest of us, because they purport to be accurate. In fact, that data is almost all collected from the surfing patterns of a very small sample of people. Those people are not going to be surfing certain niche sites as much as the general population.
Take the example of Streetfire, a very popular video site for car enthusiasts. Until our initial post, co-founder Adam Bruce was somewhat content with Google analytics, which showed his site had three million unique visitors a month (he let us login to his Google account to confirm). This matched information of his server logs, which Google also manages, and it was compatible with the ad server tracker from Adjuggler. Of course, this was all internal information.
Outsiders had access to Alexa (see its Streetfire rankings), a free site measuring tool, but Bruce believes most people recognize Alexa’s shortcomings, so he isn’t taking Alexa too seriously. Then along came two more sites, Quantcast and Compete, saying they can do a better job than Alexa. After we pointed to them in our post, Bruce checked them, but he found they were both showing his traffic at about an eighth to a twelfth of his internal numbes. Alarmed, and filled with renewed doubt about the reliability of his internal numbers (because server logs can be unreliable too, as can Google Analytics), he called up Compete and Quantcast to find out more.
Notably, both companies cooperated. Quantcast responded by offering him a tracking pixel, to be put directly on his site. This opened Quantcast’s eyes: It saw Streetfire’s traffic was indeed much higher than it had estimated, and told Bruce it will readjust its numbers after a full month’s reading. Compete, however said it couldn’t deliver a tracking pixel — in part, it said, because Nielsen/Netratings owned patents on this sort of thing, and that Quantcast can get away with it because it is too small to sue. (Although, we’re not certain whether Netratings really does have a patent on this).
To be fair, Compete does have impressive data: It says it gets data from the surfing habits of two million people, which is more than comScore, Hitwise, Nielsen and Alexa. It takes extensive measures to ensure data is diverse — gathering it from multiple Internet and application service providers, panels and its own toolbar — all this is more than can be said for the others. Compete has an impressive list of testimonials, but read through the list — most of those people aren’t experts, and even if they are, our hunch is that they wrote positively about Compete, just like we did initially, because it looked so much better than what Alexa was offering.
All these strengths of Compete, however, serve to underscore our point. If it really has such a industry-leading sampling, why did it go so horribly wrong with Streetfire? We called up Compete, and asked. We talked with product manager Jay Meattle, and Stephen DiMarco, who is in charge of “emerging markets.” They said they went back to their panel sampling and studied why a video site like Streetfire would be undercounted. Their analysis found video-oriented sites like Streetfire were being undercounted by a factor of four. Adjusting their numbers, that put Streetfire’s monthly uniques back up to 1 million or so, but not at the 3 million the other trackers were showing. In other words, it is still undercounting — and it came only after Streetfire requested a revision.
This is particularly worrying because Streetfire is a major site. If a car enthusiast site of 3 million uniques is being undercounted by the biggest objective panel out there, then well, the rest of us little chickens are hosed. The Compete guys told us they are more interested in serving general consumer sites (their main customers), and don’t really care about niche sites like say, VentureBeat ;). “Our clients are big consumer marketers,” they said. Fair enough. They take Streetfire seriously, they said, because it is a big site. But that also leaves us with this hunch: Most consumer sites are overcounted by Compete. Both may have an interest in that being the case.
Meanwhile, Adam Bruce, of Streetfire, is relieved because Quantcast’s tracker, at least, vindicated him. He can still trust his internal numbers. Bruce’s conclusion: Quantcast offers the best of both worlds. It offers you all the third-party data it can get (see screenshot at bottom; it shows gender, geographic and other data that Bruce says coincides with his sense of his readers), but if you ask it to, it will put a tracker on your site, to verify. Now, not all sites will allow Quantcast do to this. But its livelihood is on the line, like Streetfire’s small team’s is, it makes sense.
Conclusion: When you go to Quantcast for data on a site, be sure it has a tracking pixel on the site before drawing conclusions.
Finally, a note on Site Meter. Bruce said that company’s stats make sense since they are in line with his other sources, though Site Meter doesn’t report unique visitors, just visits.
34 Comments
-
Stat Head said:
According to Google Analytics, Streetfire.net received about 1.79 million U.S. uniques in January — not 3 million. Compete’s correction at 4x, would bring it to 1.1 million rounded up.
Also take a look at:
http://www.alexaholic.com/streetfire.net+venturebeat.comAccording to Alexa, looks like Venturebeat gets more traffic than Streetfire.net… Matt, do you get 2 million uniques a month? ;)
-
Stat Head said:
More from Alexa: According to them, only 36.9% of Streetfire.net traffic is US, which would put them at 1.1M U.S. uniques.
Who is correct?
-
Jeff said:
NetRatings patent portfolio includes United States Patent Nos. 5,675,510; 5,796,952; 6,108,637; 6,115,680; 6,138,155; 6,643,696 and 6,763,386
Netratings has been on a patent acquisition spree for the last several years. Almost all of their patents have been purchased.
I am really interested to hear how Quantcast bypasses these patents.
In recent years, they have sued Coremetrics, Omniture, Jupiter Media Metrix, NPD Group, etc.
If you go to NetRatings.com, click on press releases and search for “Patent” you can find out more about their patent “enforcement” program. This has also been a prominent topic in their last several earnings calls which you can probably find online too.
-
Matt Marshall said:
Stat Head, where are you getting the Google Analytics numbers from? I’m staring at it again here on my computer (with password from Streetfire guys) and it says 4,278,856 uniques in January. The extra numbers come from the company’s other properities. They’re all housed under the Vidiac brand. Since Streetfire makes up 71.74 percent of Vidiac (again, Google’s numbers), that’s 3 million.
-
Stat Head said:
Matt,
Compete stats are U.S. only.
According to Adam’s previous comment, 59.42% of Streetfire.net’s traffic is U.S.
Total Streetfire.net traffic = 71.74% * 4.2M = 3M
Total Streetfire.net *U.S.* traffic = 3M * 59.42% = 1.79M
-
Adam Bruce said:
Stat Head,
Were that true we would have no advertisers at all! That tells me Alexa is under-counting our US traffic.See for yourself.
http://www.google.com/analytics/
User Name: review.streetfire@gmail.com
Password: Review99I’ll leave this account open for a few days. so you all can see for yourself what or internal numbers are saying and draw your own conclusions.
-Adam
-
Stat Head said:
Also important to note: Compete stats do not count bots, spiders, pingbacks, agents, etc etc. They count actual live people.
Also, measurement done via cookies is always error prone for counting unique visitors as people tend to delete cookies fairly frequently. Cookie based techniques will be fairly accurate for pageviews, visits, etc… but break down for counting Uniques pretty damn fast — as you’re essentially assuming the person did not delete their cookie, and if they did delete the cookie, the next time they visit your website… that’s another Unique in Google Analytics eyes.
-
Matt Marshall said:
yes, cookies are pain, and one more reason why uniques are problematic. still, why did compete undercount by a factor of four in the first place? they’re counting live people, but far too few. that tells me something is very wrong, and that a tracker is much more reliable.
…but heck, i’ll take your alexa conclusions about VentureBeat traffic ;)
-
Adam Bruce said:
“Also important to note: Compete stats do not count bots, spiders, pingbacks, agents, etc etc. They count actual live people.”
I would agree with that statement If Compete had a tracker on my site and was measuring people, but it’s my understanding that they survey 2 million people who may or may not go to Streetfire and then make statistical assumptions based on what those 2 million people do.
If those 2 million people simply do not go to your site, then Compete is not counting live people to StreetFire at all, rather it’s making assumptions based on what their users are doing.
As for Bots, ping backs, etc, Our AD-Server is very sensitive to those as it costs us money to try and serve an ad to a bot, and I can say that our Ad-Server Correlates with what we see on analytics. I don’t think we have 2million bot and spiders driving up our numbers, that wouldn’t give us any room at all for real people!
-
Stat Head said:
Adam,
What about the well documented problem of people deleting cookies? The same cookie issue effects Google Analytics and Quantcast for measuring uniques, so of course the two would report similar numbers.
I am not arguing that Compete is more accurate for “Quantified” sites at all. Data directly from the source will of course be more accurate, than an estimate based off a smaller sample.
Panel companies have complex methodologies to project data. Compete takes 2 million people, and projects that sample to fit the U.S. internet population.. which I believe is around 150 million. Any person dealing with data will tell you that all data is biased. When you’re estimating traffic for over a million domains, it is impossible to detect and correct every single bias systematically/automatically for such a big dataset. As you know. Alexa does not correct for bias at all (they can’t detect any as they only have toolbar data — and the crazy data is very evident by the Streetfire.net vs Venturebeat stats on Alexa).. Compete tries, and in my experience for the bigger sites does a good job. There is very much room for them to improve for estimating smaller sites, and I look forward to them addressing this. They are headed in the right direction.
My guess is that this particular Streetfire case is an anomaly for them.
Matt, you should get 15-20 site owners, with traffic over 1 Million to release their internal US numbers, and then have a bake-off, vs. basing your opinion on a single case :)
-
Adam Bruce said:
“…Matt, you should get 15-20 site owners, with traffic over 1 Million to release their internal US numbers, and then have a bake-off, vs. basing your opinion on a single case :)…”
So you’re saying that making a larger assumption off a small sample is an incorrect methodology? (*jab* *jab* :-)
-
Stephen DiMarco said:
Matt, thanks for trying to clarify for everyone how web-wide analytics work and for taking the time to learn more about Compete before you posted your follow-up article. It’s clear we are all striving for accurate web measurement. Tracking pixels and consumer panels ete) are two ways of getting accurate web metrics. Since Compete has multiple sources of data, we were able to show that one of our sources was in fact under-representing Streetfire. And based on this we can now better normalize and extrapolate usage data for the Streetwise site. Nothing controversial here, just a straightforward solution.
I like the Quantcast approach, too. The problem is that not every site is going to use their tracking pixels (their legal exposure is another looming problem). So my suggestion is that site owners work with all web measurement firms to make sure their site metrics are being represented consistently and fairly. This is the exact approach Steetfire and Compete are taking.
Side note: Despite what you said in your post, we do care about Venture Beat. That’s why we answered your questions honestly and are continuing an open dialogue.
-
Matt Marshall said:
I’d like to keep tracking this issue. What’s the consensus estimate for how much bots, spiders, etc, distort traffic?
Also, I’m game for rounding up a group of >1 million-reader Web sites. If there’s anyone from such a site reading, feel free to drop me an email (see contact email from front page) or leave your name here in comments…
-
StartupDunia said:
My biggest gripe against compete is that they provide stats for US based visitors only.
Ughhh….
Never understood why they’re supposed to be better than Alexa.
Pranav.
-
Mike said:
To triangulate Alexa, Compete and Quantcast traffic charts, see http://www.attentionmeter.com
-
Search Engines WEB said:
-> seoptimization.blog.com/1221628/
You can compare these stats of varied top Technorati blogs which use the public Sitemeter - with the stats of Compete and Quantcast to further analyze their reliability
-
Jess said:
has some really insightful thoughts on competitive intel tools:
“It is really impossible, and perhaps not even advisable, to tie the competitive intelligence data with the clickstream data from your own website web analytics tool. The data collection is very different and the purpose is very different.
In the context of competitive intelligence you are comparing two entities (you and your competitor) using the same tool / process / application and that normalizes the “biasâ€. All other things being equal you can still find actionable insights and make fundamental changes to your strategy.”
-
Ed said:
I agree with Seth Godin on the issue of pageviews and time spent:
http://sethgodin.typepad.com/seths_blog/2007/02/meaningless_to_.htmlAlso, I disagree with Compete’s following post:
http://blog.compete.com/2007/01/25/top-20-websites-ranked-by-time-spent/It doesn’t say much that Yahoo comes in 2nd. Most of the people visiting yahoo.com do so for YahooMail and leave it logged on. I know alot of people who do this. If those same folks had free access to POP3 with Yahoo and did so, I’m pretty sure Yahoo’s numbers would drop drastically.
-
Jess said:
Ed,
Don’t people do the same with Gmail — especially with Chat built in?
-
Webanalyticsbook said:
Compete and other tools work great for me:
http://www.webanalyticsbook.com/archives/568 -
web 2.0 innovations said:
hey guys it is http://www.streetfire.net and NOT http://www.streetfire.com
-
Matt Marshall said:
Sorry about that. I’ve corrected to .net.
-
Jay Meattle said:
Ed, Thanks for the comment.
The market currently relies heavily on visitors and page views to measure performance, but we both know these metrics can be gamed.
Time, on the other hand, is finite and selfishly managed by the user making it harder to game. Therefore if a site can garner more of an individual’s time it should be considered a good thing, right?
With that said, I know there are exceptions (Search for example), which is why we haven’t presented Attention as the king of all metrics. We see Attention as an additional piece of the puzzle.
For example:
- MySpace attracts 16% of all pageviews on the internet (U.S.)
- However, U.S. users only spend 11.9% of their time on MySpace
- MySpace provides a service valued by consumers (11.9% Attention Share), but the site is terribly designed and thus inflates misleading engagement metrics like pageviews
-
Adam Bruce said:
from Jess
“…In the context of competitive intelligence you are comparing two entities (you and your competitor) using the same tool / process / application and that normalizes the “bias .”I agree it’s important to try and compare using similar methodology, but the problem is the assumption that they have normalized the bias.
Compare CarDomain.com to StreetFire.net (5x difference)
http://snapshot.compete.com/cardomain.com+streetfire.net+And on Alexa (smaller difference)
http://traffic.alexa.com/graph?w=725&h=340&r=2y&y=r&a=1&z=5&u=streetfire.net&u=cardomain.comAnd in reality, I know for fact that CarDomain has equal traffic to StreetFire.net (we work closely with CarDomain and have visibility into their analytics).
I would agree there is a normalized bias on pixel tracking or measuring actual server logs in the same manner, but when it comes to survey methodology it’s incorrect to assume the results themselves are properly normalized.
-
Webanalyticsbook said:
————-
Quote:
metrics can be gamed
————-
Metrics are heavily gamed and most metrics are not even defined by the WAA. Ask the average webmaster for the definition of page views and/or unique visitors and you get 90% a not accurate answer. -
web 2.0 innovations said:
What would happen with Alexa, Quantcast and Compete when/if Google decides to open its Google Analytics up to the public, or at least leave that decision to the publishers?
-
Webanalyticsbook said:
As far as I know Google Analytics has about 3-400.000 “customers”. It will be a small percentage of websites unless they add some data from their toolbar (like Alexa).
-
Pan_theFrog said:
Using Quantcast I’ve noticed several sites that seem to be getting few hits, but still get a decent pagerank. Sites like betterworld.net (rank 258,369 w/ under 25 hits a day), pimpnflyguy.com (rank 257,382 w/ under 4 hits a days since Jan 23) archimedes-lab.org (rank 156,874 which has had 1 hit since Jan 24).
I noticed that my stats were changing each day (from 10-1000 ranks), but in tracking pimpnflyguy.com (It was the first one I saw) he only changed 1-10 ranks.
I posted something on their blog about it, but it got removed. I also posted something again last night at the same place: http://quantcast.typepad.com/quantcast/2006/12/compare_site_tr.html
I am wondering if it will be yanked as well. -
Pan_theFrog said:
Well they didn’t yank this post, and even acknowledged that it is a problem they are aware of and working on.
Meanwhile, I am ranked 246,985 (With ~400-700 US hits a day), pimpnflyguy.com (rank 242,599), betterworld.net (rank 264,462), archimedes-lab.org (rank 138,976).Now why is this notable? Because when this saga started on Feb 24, 2007 pimpnflyguy.co was ranked 257,380, which was about 300 ranks ahead of me (He was the next highest quantified user I could find). Now here it is a week later, and while I have moved up 10.5k ranks, pimpnflyguy has moved up even more then that, but still has no more then 4 hits a day since Jan 23, 2007.
archimedes-lab.org has moved up 18k ranks, and still shows 1 hit since Jan 23, 2007 (And I think that was me). betterworld.net has dropped 6k ranks, but is getting more hits per day then either pimpnflyguy or archimedes-lab.
Is there any logic in what they are using for stats? -
Igor said:
Matt, great article.
I had a wake-up encounter with “Compete” when one of VCs we know pointed at them as a source of info on our (Fotki.com) traffic “falling”. Now, at Fotki, we have the ever-increasing traffic and fast growing usage, making us buy more and more servers (soon getting to 100 mostly 2U and 3U machines) , serving hundreds of millions of monthly pages so, when I hear things like that I laugh with not much joy in my giggles, to say the least. So, I went to Compete.com and saw for myself - wow… 600K uniques, huh? :) Google Analytics and our internal logs are in more than 90% sync showing 5M uniques - 50% of them in the US - so, that’s 2.5M, 4 times more than Compete shows.
So, I wrote to Compete.com telling them that their stats are off by a LOT - what do I get as a reply? - A letter stating that for sites with small traffic their data may be significantly off :)
Now, interestingly, even on Compete.com we are in top 3,000 (far cry from reality), and “Compete SnapShot provides information for the top 1,000,000 web sites”, so, basically, in their own words, their data should be wrong at least to 99.7% of sites they measure (since we are in their top 0.3% sites.)
I am far from saying that Compete is completely useless (I simply don’t have time or need to do a research) but it’s completely useless to me. One should look at HOW Compete gets its “2M unique panelists” - and a lot of questions will be resolved. So, folks at Compete, if you really want to, um, *compete* with Alexa - tables like these - http://www.compete.com/help - don’t really go far, what does is accuracy. Reconsider the way you get your sample, listen to people like Matt and adjust, rather than trying to argue for how good you are. - Then and only then Seth Godin, Om Malik, Esther Dyson and other influentials, as well as millions of users like myself will be using you.
Cheers,
igor -
Pan_theFrog said:
My logs show that I had 17,691 unique visitors in Feb. But today Quantcast.com has me ranked at 246,823. pimpnflyguy.com is at 242,598, betterworld.net is 264,460, archimedes-lab.org is 138,983.
Quantcast shows me getting an estimated 5,050 unique US visitors a month (With about 400 US visitors a day), http://quantcast.com/pimpnflyguy.com shows 5,147 US visitors (With no more then 4 visitors a day), http://quantcast.com/betterworld.net shows 4,677 US visitors (No more then 25 visits a day), http://quantcast.com/archimedes-lab.org has 10,080 US visitors (even though they have had only 1 visitor since Jan 24 and stats for them end on Jan 29).
So using the information I have listed for you, which you can check yourself, please tell me who Quantcast is actually working correctly for?
-
uynczxowtp said:
Hello! Good Site! Thanks you! ybfmishnxievd
-
James said:
Great article. Any thoughts about the accuracy of the demographics supplied by these companies?
- James
-
Ineteette said:
Two new studies show why some people are more attractive for members of the opposite sex than others.
The University of Florida, Florida State University found that physically attractive people almost instantly attract the attention of the interlocutor, sobesednitsy with them, literally, it is difficult to make eye. This conclusion was reached by a series of psychological experiments, which were determined by the people who believe in sending the first seconds after the acquaintance. Here, a curious feature: single, unmarried experimental preferred to look at the guys, beauty opposite sex, and family, people most often by representatives of their sex.
The authors believe that this feature developed a behavior as a result of the evolution: a man trying to find a decent pair to acquire offspring. If this is resolved, he wondered potential rivals. Detailed information about this magazine will be published Journal of Personality and Social Psychology.
In turn, a joint study of the Rockefeller University, Rockefeller University and Duke University, Duke University in North Carolina revealed that women are perceived differently by men smell. During experiments studied the perception of women one of the ingredients of male pheromone-androstenona smell, which is contained in urine or sweat.
The results were startling: women are part of this repugnant odor, and the other part is very attractive, resembling the smell of vanilla, and the third group have not felt any smell. The authors argue that the reason is that the differences in the receptor responsible for the olfactory system, from different people are different.
It has long been proven that mammals (including human) odor is one way of attracting the attention of representatives of the opposite sex. A detailed article about the journal Nature will publish.
One Trackback
1:59 pm
Top ten things you must do for a good user experience | The Next Engine: Beyond Campaign Thinking said:
[...] to analytics: It’s not enough to look at just how many visitors you get. That number is generally misleading anyway. Look closely at exit pages. Exit pages are crucial. If everyone leaves after your homepage, [...]