VentureBeat

When respected traffic measurement company Comscore released data several days ago showing that Facebook’s U.S. visitors dropped more than 9.3 percent last month, it sparked widespread disbelief among bloggers and others. Facebook is hot, and growing. The data must be wrong!

Comscore has just published data showing Facebook worldwide unique visitors are up, to 73.5 million from 69.2 million, suggesting not all is too bad afterall.

So what’s with the earlier national numbers? Well, to everyone’s relief, there came the quick red herring explanation from Comscore competitor, Quantcast: Student members of Comscore’s panel went back to school in September, and so their surfing patterns were no longer being tracked. Facebook visits measured by Comscore plunged, even though the students were still using Facebook from their dorms. Comscore was wrong! Facebook was doing fine after all.

However, in talking with each of the traffic measuring services, including Comscore, Quancast, Compete and Hitwise, we’ve concluded the explanation isn’t this easy. While Facebook may be doing fine, it points to some serious problems with the leading measurement companies that are more complex than most people realize — and advertisers could get very unhappy.

Every one of the panel companies showed a decline for Facebook in September. Moreover, their panels are pretty sophisticated. They’re each tracking more than a million people’s traffic patterns. And Comscore even has a special “educational panel.” Comscore won’t say exactly how large it is, but it’s reported to be something like 15,000, filled with students, and in fact they don’t vanish from the panel in September. There is no “home bias,” as Quantcast alleged, Comscore tells us. The panel tracks student traffic patterns from student in their dorm-room computers, too. Andrew Lipsman, spokesman for Comscore, says it is “nonsense” to say the September decline stemmed from students vanishing from panel.

Moreover, Comscore readjusts the weight of its educational panel, so that its influence in Comscore’s data corresponds with the overall student population of the U.S.

So, if the panel methods are robust, and they’re all showing Facebook traffic down, you’ve got to assume Facebook dropped in September, right? The traffic companies tried to assert this politely: “I’m going to settle the question once and for all, said Bill Tancer, Hitwise’s manager of global research, at the Web 2.0 Summit in San Francisco Thursday. “It hasn’t plateaued, it’s just experienced a seasonable dip in traffic.”

Not so fast, Bill. There’s a big problem with even saying its seasonal. Facebook itself says traffic was up. It saw active monthly users – those defined as returning to the site at least once a month – increase to 19.4 million in September, from 17.7 million. That’s not merely growth. That’s huge. That’s a 10 percent increase! Globally, the growth is more impressive still: Facebook ended September with 44 million users, compared to nearly 9 million a year ago, spokeswoman Brandee Barker tells us. [Comscore, however, today says that number is 73.5 million, which looks to be way off. Facebook wouldn't undercount by that much.] Now, you can choose to be a conspiracist, and believe Facebook is fudging numbers. But here’s why that’s very unlikely. Facebook, a venture backed company in the midst of raising big cash, and has no interest in fudging. There’d be internal dissension, and investors and potential suitors would call them on it. This place is so incestuous – Silicon Valley is so networked, and so prone to leaks – they’d never get away with it. So, our default is to take Facebook at its word. Page view growth is also straight in line with the growth, says Barker, at 57 billion, up from 54 billion.

You can’t dismiss Faceook’s growth as being driven by “cookies” either. If you use Facebook, you’ve got to login. So if you’re using Facebook from the office, from home or the dorm, only one of you shows up in Facebook’s stats. In other words, Facebook’s way is more reliable than that used by other sites. Many companies use Google Analytics, which tracks visitors by placing a cookie on the visitor’s browser. This inflates the visitor count up to about 2.5 times, say experts at Comscore and elsewhere. Facebook confirmed that it only measures member activity: It isn’t counting random hits on their page from say, a non-member sitting in a cybercafé in China.

So where does this leave us? Well, unfortunately, it means you’ve got to throw out of the window any short term statistics trends from the panel guys. The New York Times has also just weighed in this morning with a story about the statistics mess.

One notable model is pursued by Quantcast, which places a pixel tracker directly on a Web site (with the permission of the site’s owner, of course), and so is as good as Google or your own server log in getting direct results. While Quantcast’s approach is promising, it also has serious shortcomings – in part because of misunderstandings from site masters about where to place the code (more on that in a second).

Moreover, panels break down badly for niche sites. If you have only a thousand readers, for example, it’s possible there’s no one on the panel reading you, and so the bias can’t even be compensated for with statistical methods.

Here’s a summary of key points on each measurement service:

Comscore – Notably, despite the decline it showed for Facebook in September, it shows unique monthly users for Facebook at 30 million in the U.S., a rate that is not only higher than Facebook’s 19.4 million, but ridiculously higher. And we’re talking only U.S. It’s global numbers are even more out of line on the high side. See chart below:

Comscore is the measurement company we all rely on (for example, VentureBeat partners with Federated Media to handle much of our advertising, and they tell us advertisers all track Comscore data). That it is apparently so wrong raises huge questions about the underpinnings of the Internet advertising economy. Advertisers may only pay for the page views their ads are being placed on, but they’ll never know for sure how many unique readers they’re getting.

To be sure, after cross-examining Comscore about its panel, we felt pretty comfortable with the seriousness of Comscore’s approach, and came away understanding why it is considered industry standard, even if there are apparently big bias problems with its end results.

Regarding the student Comscore panelists: Let’s say a student logs on from a dorm room, and then later logs on from a university terminal. Comscore doesn’t want to count them twice. For this reason, Comscore doesn’t track results from university terminals, Internet cafes or other public computers. That way, they avoid inflating numbers. [As noted, Google Analytics is misleading, as are most server logs, because they do double count users. They'll count you if you come in from home, and they'll count you if you come in from work, giving your browsers at each place separate cookies. Google provides you a number called “absolute unique visitors,” because it can’t tell how many of them are duplicates. Divide by 2.5, and you’ll get a rough estimate of real unique visitors.]

Here’s an example of why the double-counting issue is so frustrating: Sugar Inc. says it has 5 million “absolute” unique visitors, while Comscore shows it having less than two million. This would be understandable if Sugar was counting each visitor twice as they sign up from work and home. However, CEO Brian Sugar says 80 percent of his traffic comes between 9am and 5pm, suggesting there’s not a lot of home usage going on. In other words, even though Comscore says it isn’t double counting, Sugar still says Comscore “is so radically wrong.” (Sugar also says some of his sites aren’t tracked by Comscore — in part, perhaps, because they are too small for Comscore’s panel to pick up).

Comscore suggests the Facebook dip came because September is a seasonal month. February has 28 days and is the most seasonal. September is the second most seasonal. It has one fewer day than August, and has a holiday weekend in it. Search data shows major dips during this month too. But this doesn’t explain a nine percent dip.

Comscore data is relatively clean compared to other services. It doesn’t get its data through Internet Service Providers, like some other vendors do. This can be biased. High-speed internet users, via Comcast, are likely to consume twice as much content as dialup users, and they’re likely to traffic different site. Here’s another example of bias: Since SBC has a relationship with Yahoo, users of SBC are more likely to start on Yahoo’s homepage, thus distorting that company’s traffic. (Competing service s like Hitwise, Quancast and Compete all rely on ISP data).

Comscore has a 125,000 people on its core US Media Metrix panel for the U.S., but has a million in U.S. total counting panels it uses for other purposes. It has 2 million worldwide.

Hitwise – This company’s data is not easily accessible by the masses. You have to be a subscriber. It also doesn’t show unique, just relative market share. Here’s what Hitwise analyst Bill Tancer says of the “dip.” It is seasonal, he says, and it’s something he’s seen in past years. Looking at his chart, you’ll see the decline in Facebook’s traffic came earlier last year. This is logical. In past years, Facebook was more student focused, and traffic dropped when school was out. Only last year did Facebook allow non-students. These new users, many professionals, actually have more time to play with Facebook in August than they may in early September, when they’re back and rushing kids to school. Note how this year’s drop happened more pronounced in September. This is a highly plausible explanation. But again, Facebook has rejected these explanations.

Compete – These guys, like Quantcast, are no longer to be dismissed. It decided to pass on offering a direct pixel tracker, but it also has a panel of two million users. Like Quantcast, however, it relies on ISPs, which can be dangerous for the reasons explained above. It showed a 7.7 percent dip in Facebook’s traffic in September. It’s 24 million number is way higher than Facebook’s own numbers.

Quantast — Quantcast’s advantage is that it can put a tracker on your site, and get direct measurement. However, many sites are reluctant to give it such access. In return for giving Quantcast the tracker, the publisher must let Quantcast publicly release monthly user stats. Facebook does not use Quantcast. Quantcast’s panel also shows Facebook having 24 million unique in the U.S. (see chart). It also has a panel of two million, some of them ISP customers, and so it has the same bases as Compete and others.

A site called Streetfire however, shows the risk. Earlier this year, the race car enthusiast site chose to use Quantcast’s direct tracker (see our coverage). However, Quantcast soon disappointed, said Adam Bruce. First, it was causing too much latency, since it was piled on top of all the other scripts Streetfire was running. While Quantcast wasn’t taking much more time to load on average than other scripts, sometimes it would take seconds, says Bruce. Other times, it looked like the script would time out, and just not record any traffic at all. It depended on whether the tracker script was higher or lower on the page. If it was higher, Quantcast would load first, and perform accurately. If was lower down, it would the last to load, and sometimes readers would be gone before it could count them.

Finally, Streetfire decided to remove Quantcast from its site. But things didn’t’ get any better. Streetfire’s traffic has plunged, according to Quantcast site. It’s apparently due to misunderstandings. Bruce assumed Quantcast had defaulted to using its panel to measure Streetfire. However, this wasn’t the case. Streetfire still had some Quantcast tags on some Streetfire subdomains. That meant Quantcast’s servers were still relying on the direct tracker even though it was woefully underreporting, says Quantcast’s Feldman. The two sides are now trying to clear it up. Conclusion: If Streetfire, a venture backed company obsessed with its traffic couldn’t figure this out, imagine what most other sites will go through.

Konrad Feldman says Web performance problems are unavoidable, for all companies. However, he rejects the notion that his servers have suffered under increased load. Tracking is difficult, he said, It requires tracking billions of page views daily, and having to do it with no latency, and with enough backup measures to ensure nothing is lost. That means hundreds of terabytes of data.

While Facebook hasn’t signed up for Quantcast’s tracker, hundreds of others have, from Fox and CBS to new media companies, Hi5, Slide, and Rockyou.

Alexa – This is a toolbar. Toolbars are unreliable, and Alexa has been so discredited and so wrong in the past, there’s no point trying to debunk it. Its problems have been thoroughly documented.

Statsaholic – This is a repackaging of Compete data.

Updated

More and more publishers and retailers rely on so-called “widgets,” little boxes placed on other web sites, to deliver their news, entertainment and product advertisements.

So measuring Web traffic to those widgets is important for deciding what content to deliver, and what sites to deliver it on. A number of companies now offer widget traffic measurement tools, with Quantcast being the latest.

Here’s a summary of some of the main players in this nascent but increasingly important field:

Quantcast – This relatively new San Francisco company monitors traffic to Internet sites. Today, it adds a video and widget measurement service, also free. It is still a test version and publishers include Slide, PictureTrail, RockYou, MetaCafe and MochiMedia. It reports on traffic to all Flash-based media, including online games and downloaded desktop widgets. The service reports a widget’s “reach,” the number of times a video has been “played” or that widget has been clicked on, and the categories chosen (a publisher can tag a widget or video as a “game” or “comedy,” for example). It will also report things such as which users clicked on the widget most often (frequency), and the demographics of these users, based on other information Quantcast collects. It provides this information in a pie chart. To use it, you have to register here www.quantcast.com/quantified-publisher.jsp, and more info is here www.quantcast.com/quantified-video.jsp. See chart below.

Clearspring – The company offers traffic analytics for publishers like Time, NBC, Universal and Maxim – which together now have more than 4.2 billion widget views. Last month, the company began letting developers write, distribute and tracking widgets through Clearspring. Like other offerings, it provides a dashboard, and tracks things like a widget’s source domain, number of visits and geography of visitor. Clearstone, like Quantcast, is hands-off on the widget creation process. You’ll need a third-party developer to build a widget if you want one built.

WidgetBox – This company provides a “wrapper” to widget developers so that they can track the widgets. Again, you’ll need to build the widget yourself.

Musestorm -– The company distributes widgets for both the Web and the desktop. Notably, though, this company goes a step further. Other companies put a “wrapper” around the widget and then track that widget. However, they can’t track activity of content within the widget. Musestorm lets you do that -– for example letting you know how many times a video within the widget has been played, and how users switch between surrounding audio and text. Quantcast, Widgetbox and Clearspring don’t offer that. Musestorm also lets you update widget content, adding and removing items, and so lets you track performance of variable kinds of content. Musestorm has a new release coming this summer containing other features.

Update: Comment below mention other providers, Comscore (see here) and Yourminis.

It sucks when your Web site’s traffic isn’t being measured correctly.

It also sucks when you’re trying to measure the significance of someone else’s site, and are getting conflicting signals.

Here’s what we’ve learned over the past few days, after our initial piece on the problems of Alexa, Quantcast and Compete, all sites that independently verify how much traffic a site is getting.

We’ve learned that if a measuring company doesn’t have a tracking pixel directly on the Web site it is tracking, the numbers are extremely unreliable.

This may be nothing new for insiders, but the public data provided by Alexa, Quantcast and Compete can be misleading for the rest of us, because they purport to be accurate. In fact, that data is almost all collected from the surfing patterns of a very small sample of people. Those people are not going to be surfing certain niche sites as much as the general population.

Take the example of Streetfire, a very popular video site for car enthusiasts. Until our initial post, co-founder Adam Bruce was somewhat content with Google analytics, which showed his site had three million unique visitors a month (he let us login to his Google account to confirm). This matched information of his server logs, which Google also manages, and it was compatible with the ad server tracker from Adjuggler. Of course, this was all internal information.

Outsiders had access to Alexa (see its Streetfire rankings), a free site measuring tool, but Bruce believes most people recognize Alexa’s shortcomings, so he isn’t taking Alexa too seriously. Then along came two more sites, Quantcast and Compete, saying they can do a better job than Alexa. After we pointed to them in our post, Bruce checked them, but he found they were both showing his traffic at about an eighth to a twelfth of his internal numbes. Alarmed, and filled with renewed doubt about the reliability of his internal numbers (because server logs can be unreliable too, as can Google Analytics), he called up Compete and Quantcast to find out more.

Notably, both companies cooperated. Quantcast responded by offering him a tracking pixel, to be put directly on his site. This opened Quantcast’s eyes: It saw Streetfire’s traffic was indeed much higher than it had estimated, and told Bruce it will readjust its numbers after a full month’s reading. Compete, however said it couldn’t deliver a tracking pixel — in part, it said, because Nielsen/Netratings owned patents on this sort of thing, and that Quantcast can get away with it because it is too small to sue. (Although, we’re not certain whether Netratings really does have a patent on this).

To be fair, Compete does have impressive data: It says it gets data from the surfing habits of two million people, which is more than comScore, Hitwise, Nielsen and Alexa. It takes extensive measures to ensure data is diverse — gathering it from multiple Internet and application service providers, panels and its own toolbar — all this is more than can be said for the others. Compete has an impressive list of testimonials, but read through the list — most of those people aren’t experts, and even if they are, our hunch is that they wrote positively about Compete, just like we did initially, because it looked so much better than what Alexa was offering.

All these strengths of Compete, however, serve to underscore our point. If it really has such a industry-leading sampling, why did it go so horribly wrong with Streetfire? We called up Compete, and asked. We talked with product manager Jay Meattle, and Stephen DiMarco, who is in charge of “emerging markets.” They said they went back to their panel sampling and studied why a video site like Streetfire would be undercounted. Their analysis found video-oriented sites like Streetfire were being undercounted by a factor of four. Adjusting their numbers, that put Streetfire’s monthly uniques back up to 1 million or so, but not at the 3 million the other trackers were showing. In other words, it is still undercounting — and it came only after Streetfire requested a revision.

This is particularly worrying because Streetfire is a major site. If a car enthusiast site of 3 million uniques is being undercounted by the biggest objective panel out there, then well, the rest of us little chickens are hosed. The Compete guys told us they are more interested in serving general consumer sites (their main customers), and don’t really care about niche sites like say, VentureBeat ;). “Our clients are big consumer marketers,” they said. Fair enough. They take Streetfire seriously, they said, because it is a big site. But that also leaves us with this hunch: Most consumer sites are overcounted by Compete. Both may have an interest in that being the case.

Meanwhile, Adam Bruce, of Streetfire, is relieved because Quantcast’s tracker, at least, vindicated him. He can still trust his internal numbers. Bruce’s conclusion: Quantcast offers the best of both worlds. It offers you all the third-party data it can get (see screenshot at bottom; it shows gender, geographic and other data that Bruce says coincides with his sense of his readers), but if you ask it to, it will put a tracker on your site, to verify. Now, not all sites will allow Quantcast do to this. But its livelihood is on the line, like Streetfire’s small team’s is, it makes sense.

Conclusion: When you go to Quantcast for data on a site, be sure it has a tracking pixel on the site before drawing conclusions.

Finally, a note on Site Meter. Bruce said that company’s stats make sense since they are in line with his other sources, though Site Meter doesn’t report unique visitors, just visits.