VentureBeat

When respected traffic measurement company Comscore released data several days ago showing that Facebook’s U.S. visitors dropped more than 9.3 percent last month, it sparked widespread disbelief among bloggers and others. Facebook is hot, and growing. The data must be wrong!

Comscore has just published data showing Facebook worldwide unique visitors are up, to 73.5 million from 69.2 million, suggesting not all is too bad afterall.

So what’s with the earlier national numbers? Well, to everyone’s relief, there came the quick red herring explanation from Comscore competitor, Quantcast: Student members of Comscore’s panel went back to school in September, and so their surfing patterns were no longer being tracked. Facebook visits measured by Comscore plunged, even though the students were still using Facebook from their dorms. Comscore was wrong! Facebook was doing fine after all.

However, in talking with each of the traffic measuring services, including Comscore, Quancast, Compete and Hitwise, we’ve concluded the explanation isn’t this easy. While Facebook may be doing fine, it points to some serious problems with the leading measurement companies that are more complex than most people realize — and advertisers could get very unhappy.

Every one of the panel companies showed a decline for Facebook in September. Moreover, their panels are pretty sophisticated. They’re each tracking more than a million people’s traffic patterns. And Comscore even has a special “educational panel.” Comscore won’t say exactly how large it is, but it’s reported to be something like 15,000, filled with students, and in fact they don’t vanish from the panel in September. There is no “home bias,” as Quantcast alleged, Comscore tells us. The panel tracks student traffic patterns from student in their dorm-room computers, too. Andrew Lipsman, spokesman for Comscore, says it is “nonsense” to say the September decline stemmed from students vanishing from panel.

Moreover, Comscore readjusts the weight of its educational panel, so that its influence in Comscore’s data corresponds with the overall student population of the U.S.

So, if the panel methods are robust, and they’re all showing Facebook traffic down, you’ve got to assume Facebook dropped in September, right? The traffic companies tried to assert this politely: “I’m going to settle the question once and for all, said Bill Tancer, Hitwise’s manager of global research, at the Web 2.0 Summit in San Francisco Thursday. “It hasn’t plateaued, it’s just experienced a seasonable dip in traffic.”

Not so fast, Bill. There’s a big problem with even saying its seasonal. Facebook itself says traffic was up. It saw active monthly users – those defined as returning to the site at least once a month – increase to 19.4 million in September, from 17.7 million. That’s not merely growth. That’s huge. That’s a 10 percent increase! Globally, the growth is more impressive still: Facebook ended September with 44 million users, compared to nearly 9 million a year ago, spokeswoman Brandee Barker tells us. [Comscore, however, today says that number is 73.5 million, which looks to be way off. Facebook wouldn't undercount by that much.] Now, you can choose to be a conspiracist, and believe Facebook is fudging numbers. But here’s why that’s very unlikely. Facebook, a venture backed company in the midst of raising big cash, and has no interest in fudging. There’d be internal dissension, and investors and potential suitors would call them on it. This place is so incestuous – Silicon Valley is so networked, and so prone to leaks – they’d never get away with it. So, our default is to take Facebook at its word. Page view growth is also straight in line with the growth, says Barker, at 57 billion, up from 54 billion.

You can’t dismiss Faceook’s growth as being driven by “cookies” either. If you use Facebook, you’ve got to login. So if you’re using Facebook from the office, from home or the dorm, only one of you shows up in Facebook’s stats. In other words, Facebook’s way is more reliable than that used by other sites. Many companies use Google Analytics, which tracks visitors by placing a cookie on the visitor’s browser. This inflates the visitor count up to about 2.5 times, say experts at Comscore and elsewhere. Facebook confirmed that it only measures member activity: It isn’t counting random hits on their page from say, a non-member sitting in a cybercafé in China.

So where does this leave us? Well, unfortunately, it means you’ve got to throw out of the window any short term statistics trends from the panel guys. The New York Times has also just weighed in this morning with a story about the statistics mess.

One notable model is pursued by Quantcast, which places a pixel tracker directly on a Web site (with the permission of the site’s owner, of course), and so is as good as Google or your own server log in getting direct results. While Quantcast’s approach is promising, it also has serious shortcomings – in part because of misunderstandings from site masters about where to place the code (more on that in a second).

Moreover, panels break down badly for niche sites. If you have only a thousand readers, for example, it’s possible there’s no one on the panel reading you, and so the bias can’t even be compensated for with statistical methods.

Here’s a summary of key points on each measurement service:

Comscore – Notably, despite the decline it showed for Facebook in September, it shows unique monthly users for Facebook at 30 million in the U.S., a rate that is not only higher than Facebook’s 19.4 million, but ridiculously higher. And we’re talking only U.S. It’s global numbers are even more out of line on the high side. See chart below:

Comscore is the measurement company we all rely on (for example, VentureBeat partners with Federated Media to handle much of our advertising, and they tell us advertisers all track Comscore data). That it is apparently so wrong raises huge questions about the underpinnings of the Internet advertising economy. Advertisers may only pay for the page views their ads are being placed on, but they’ll never know for sure how many unique readers they’re getting.

To be sure, after cross-examining Comscore about its panel, we felt pretty comfortable with the seriousness of Comscore’s approach, and came away understanding why it is considered industry standard, even if there are apparently big bias problems with its end results.

Regarding the student Comscore panelists: Let’s say a student logs on from a dorm room, and then later logs on from a university terminal. Comscore doesn’t want to count them twice. For this reason, Comscore doesn’t track results from university terminals, Internet cafes or other public computers. That way, they avoid inflating numbers. [As noted, Google Analytics is misleading, as are most server logs, because they do double count users. They'll count you if you come in from home, and they'll count you if you come in from work, giving your browsers at each place separate cookies. Google provides you a number called “absolute unique visitors,” because it can’t tell how many of them are duplicates. Divide by 2.5, and you’ll get a rough estimate of real unique visitors.]

Here’s an example of why the double-counting issue is so frustrating: Sugar Inc. says it has 5 million “absolute” unique visitors, while Comscore shows it having less than two million. This would be understandable if Sugar was counting each visitor twice as they sign up from work and home. However, CEO Brian Sugar says 80 percent of his traffic comes between 9am and 5pm, suggesting there’s not a lot of home usage going on. In other words, even though Comscore says it isn’t double counting, Sugar still says Comscore “is so radically wrong.” (Sugar also says some of his sites aren’t tracked by Comscore — in part, perhaps, because they are too small for Comscore’s panel to pick up).

Comscore suggests the Facebook dip came because September is a seasonal month. February has 28 days and is the most seasonal. September is the second most seasonal. It has one fewer day than August, and has a holiday weekend in it. Search data shows major dips during this month too. But this doesn’t explain a nine percent dip.

Comscore data is relatively clean compared to other services. It doesn’t get its data through Internet Service Providers, like some other vendors do. This can be biased. High-speed internet users, via Comcast, are likely to consume twice as much content as dialup users, and they’re likely to traffic different site. Here’s another example of bias: Since SBC has a relationship with Yahoo, users of SBC are more likely to start on Yahoo’s homepage, thus distorting that company’s traffic. (Competing service s like Hitwise, Quancast and Compete all rely on ISP data).

Comscore has a 125,000 people on its core US Media Metrix panel for the U.S., but has a million in U.S. total counting panels it uses for other purposes. It has 2 million worldwide.

Hitwise – This company’s data is not easily accessible by the masses. You have to be a subscriber. It also doesn’t show unique, just relative market share. Here’s what Hitwise analyst Bill Tancer says of the “dip.” It is seasonal, he says, and it’s something he’s seen in past years. Looking at his chart, you’ll see the decline in Facebook’s traffic came earlier last year. This is logical. In past years, Facebook was more student focused, and traffic dropped when school was out. Only last year did Facebook allow non-students. These new users, many professionals, actually have more time to play with Facebook in August than they may in early September, when they’re back and rushing kids to school. Note how this year’s drop happened more pronounced in September. This is a highly plausible explanation. But again, Facebook has rejected these explanations.

Compete – These guys, like Quantcast, are no longer to be dismissed. It decided to pass on offering a direct pixel tracker, but it also has a panel of two million users. Like Quantcast, however, it relies on ISPs, which can be dangerous for the reasons explained above. It showed a 7.7 percent dip in Facebook’s traffic in September. It’s 24 million number is way higher than Facebook’s own numbers.

Quantast — Quantcast’s advantage is that it can put a tracker on your site, and get direct measurement. However, many sites are reluctant to give it such access. In return for giving Quantcast the tracker, the publisher must let Quantcast publicly release monthly user stats. Facebook does not use Quantcast. Quantcast’s panel also shows Facebook having 24 million unique in the U.S. (see chart). It also has a panel of two million, some of them ISP customers, and so it has the same bases as Compete and others.

A site called Streetfire however, shows the risk. Earlier this year, the race car enthusiast site chose to use Quantcast’s direct tracker (see our coverage). However, Quantcast soon disappointed, said Adam Bruce. First, it was causing too much latency, since it was piled on top of all the other scripts Streetfire was running. While Quantcast wasn’t taking much more time to load on average than other scripts, sometimes it would take seconds, says Bruce. Other times, it looked like the script would time out, and just not record any traffic at all. It depended on whether the tracker script was higher or lower on the page. If it was higher, Quantcast would load first, and perform accurately. If was lower down, it would the last to load, and sometimes readers would be gone before it could count them.

Finally, Streetfire decided to remove Quantcast from its site. But things didn’t’ get any better. Streetfire’s traffic has plunged, according to Quantcast site. It’s apparently due to misunderstandings. Bruce assumed Quantcast had defaulted to using its panel to measure Streetfire. However, this wasn’t the case. Streetfire still had some Quantcast tags on some Streetfire subdomains. That meant Quantcast’s servers were still relying on the direct tracker even though it was woefully underreporting, says Quantcast’s Feldman. The two sides are now trying to clear it up. Conclusion: If Streetfire, a venture backed company obsessed with its traffic couldn’t figure this out, imagine what most other sites will go through.

Konrad Feldman says Web performance problems are unavoidable, for all companies. However, he rejects the notion that his servers have suffered under increased load. Tracking is difficult, he said, It requires tracking billions of page views daily, and having to do it with no latency, and with enough backup measures to ensure nothing is lost. That means hundreds of terabytes of data.

While Facebook hasn’t signed up for Quantcast’s tracker, hundreds of others have, from Fox and CBS to new media companies, Hi5, Slide, and Rockyou.

Alexa – This is a toolbar. Toolbars are unreliable, and Alexa has been so discredited and so wrong in the past, there’s no point trying to debunk it. Its problems have been thoroughly documented.

Statsaholic – This is a repackaging of Compete data.