Micro-messaging site Twitter has gone from zero to transparent in a hurry. That is to say, it has gone from keeping users in the dark during its downtime to explaining the problems it is having in detail. Tonight, developer Alex Payne wrote a Q&A post on the Twitter Developer Blog in which he answered some users’ questions about Twitter’s recent woes.
One particularly interesting question was the following:
charles asks if there’s anything users can do to lighten our load.
To which Payne responded:
The events that hit our system the hardest are generally when “popular” users - that is, users with large numbers of followers and people they’re following - perform a number of actions in rapid succession. This usually results in a number of big queries that pile up in our database(s). Not running scripts to follow thousands of users at a time would be a help, but that’s behavior we have to limit on our side.
Most users of Twitter will single out one person who this points to: blogger and Fast Company employee Robert Scoble.
Scoble, with his 25,000+ followers and 21,000+ people he is following is a beast on the service. I would consider myself a fairly heavy user of the service and I’ve sent 3,598 tweets (Twitter messages) — Scoble has sent 12,318. This is clearly putting a strain on the service.
Perhaps more interesting was that in multiple responses, Payne more or less said not to blame Ruby on Rails (RoR) for Twitter’s woes. A lot of commenters on this blog and others are quick to jump on and blame RoR, which has even led to rumors that Twitter was abandoning it. These turned out not to be true, and in fact Twitter will continue to use RoR for the time being — at least for its front end work.
Payne also said he would use Rails again if he were looking for rapid prototyping, as Twitter was. (Payne was not around however during the initial building of Twitter.) That idea is in line with a post Nati Shalom, GigaSpaces’ chief technology officer, wrote earlier this month detailing his thoughts on Twitter’s problems from a back-end perspective.
I had a chance to talk with Shalom today about his post and his thoughts on Twitter’s woes. He believes the problem lies in the explicit contradiction between scaling and time to market. It’s also a problem with not knowing if a company will be successful and as such being cautious with spending money and time upfront to build something slowly that is fully scalable.
Shalom fully understands the burden the Twitter team now has. It basically has to take its first architecture, which isn’t working, and re-build it on the fly while the service is still running.
While exploring the issue for his own knowledge, Shalom noted Twitter’s unique position as a web application. It is neither solely a messaging app or a social networking app, but a combination of both. In the past companies had been able to build applications based on one of those foundations, but not both.
In some ways, Shalom finds it similar to the business-to-business (B2B) model that was exploding on the Internet in 2000. However this is even more complicated because with Twitter when you send out a message, you don’t know who is going to respond — it could be anyone.
Shalom agrees that all of the Ruby on Rails hatred in the blogosphere is misguided. That framework should still be fine for the front-end (just as Payne indicates), but it’s the site’s architecture or “engine” that needs to be rebuilt.
He notes that an OpenSpace community member he’s been in contact with claims that they can put a Twitter interface on top of GigaSpaces‘ own application servers to build a completely scalable version of the service. While the person claims to have a prototype working, it won’t be ready to show off until sometime this summer.
One thing is for certain with all of Twitter’s recent issues. A lot of start-ups in the future are going to learn from mistakes made. Today’s post follows Payne’s post last week stating that Twitter wasn’t built to handle its current usage.
While the core team that built Twitter at the time may have had no way of foreseeing today’s issues, companies in the future should learn from this and choose the correct architecture while taking the time to think about scalability, just in case they do hit it big.
You can find me on Twitter here along with fellow VentureBeat writers Eric Eldon, Dean Takahashi, Anthony Ha and Chris Morrison. Oh, and we have a VentureBeat account (for our posts) as well.
[photo: flickr/david sifry]
11:45 pm
Robert Scoble is responsible for killing Twitter. Did he tell them to use Tumblr too ? | My Blog Posts said:
[...] How ? One particularly interesting question was the following: charles asks if there’s anything users can do to lighten our load. [...]
12:38 am
Twitter’s Got Answers : The Last Podcast said:
[...] p.s. MG Siegler has a good take on another of the answers which seemingly blames the Twitter users with too many followers…. [...]
2:40 am
I’m Telling You … About this guy… | IT Management and Cloud Blog said:
[...] Twitter: Don’t blame Ruby, blame Scoble [...]
5:49 am
tou.ch » Blog Archive » Robert Scoble hat Twitter kaputt gemacht said:
[...] von Twitter ist. Sein Name Robert Scoble. Vielen ist er als Super-A Blogger bekannt. Hier “Twitter: Don’t blame Ruby, blame Scoble” wird gesagt, wieso Scoble Schuld sein soll. Robert sieht es natürlich ganz anders und [...]
6:13 am
Hey Scoble — you’re killing Twitter » mathewingram.com/work | said:
[...] service’s architecture, including the use of Ruby on Rails. But the real problem, it seems, is Robert Scoble. Well, maybe not Scoble specifically, but “super users” like him who have tens of [...]
7:05 am
Twitter: The Too-Popular-For-Its-Own-Good Restaurant » Webomatica - Technology and Entertainment Digest said:
[...] MG Siegler over at Venture Beat adds more insight: I had a chance to talk with Shalom today about his post and his thoughts on Twitter’s woes. He believes the problem lies in the explicit contradiction between scaling and time to market. It’s also a problem with not knowing if a company will be successful and as such being cautious with spending money and time upfront to build something slowly that is fully scalable. [...]
7:14 am
Zašto je Twitter down? at tamburix said:
[...] suočava sa velikim problemima. Na različitim stranama po Internetu se potegla rasprava na temu ovoga problema i analize da li je zaista problem u platformi koja počiva na Ruby on Rails programskom [...]
7:15 am
A Day In The Life Of Twitter | The Beta News said:
[...] course, this could also be Robert Scoble and other twitter super users (Fake Steve Jobs calls them twittertards) fault. I doubt it though. They just need a solid backbone [...]
7:55 am
Twitter kauputt und Robert Scoble ist schuld - immer! at franztoo said:
[...] ist es amtlich. Robert Scoble ist an den häufigen Problemen von Twitter schuld. Scoble löst Kanada ab. Blame [...]
8:34 am
Twitter Down Again? It’s Scoble’s Fault | Travel All Over said:
[...] VentureBeat comes to similar [...]
9:25 am
Kung Fu Quip » Twitter’s Capacity Problems and Net Neutrality said:
[...] Word out of Twitter yesterday was their more “popular” users were fuqing it up for everybody else. Anytime someone with 25,000 plus followers and 21,000 followees sends a message, it craps out their database. This has led to more than a few helpful suggestions for them to redesign their backend to better manage the load. [...]
9:36 am
Why Twitter Needs It’s Power Users: They Play Host to Newbies | RyanSpoon.com said:
[...] has a post saying that Twitter doesn’t blame Ruby’s infrastructure, they blame power users like Scoble. The following quote is from Twitter developer Alex Payne. His point is accurate - but I would be [...]
10:08 am
Yet Another Drama About Twitter - GigaOM said:
[...] most popular user is Robert Scoble, and as far as numerous successive posts have argued, he is the real source of the problem (prompting some not-unexpected foot-stomping on Scoble’s [...]
11:09 am
- Yet Another Drama About Twitter said:
[...] most popular user is Robert Scoble, and as far as numerous successive posts have argued, he is the real source of the problem (prompting some not-unexpected foot-stomping on Scoble's [...]
12:01 pm
Is twitter killing itself? - Ragnar Harper`s blog on Microsoft technology and security said:
[...] http://venturebeat.com/2008/05/29/twitter-dont-blame-ruby-blame-scoble/ Published May 30 2008, 09:01 PM by ragnar Filed under: web 2.0, ramblings, random [...]
12:32 pm
Incremental Blogger » Blog Archive » 10 things we can all blame Robert Scoble for said:
[...] not just Twitter that thinks that Robert Scoble and his 21,000 plus followers is at the heart of the service&…. No, if you look around, you’ll see that Scoble is to blame for a whole slew of things. 10 in [...]
1:10 pm
Twitter-Probleme: Scoble hat Schuld? | Twitterdings said:
[...] ordentliche Last auf dem System sorgen. Darauf schreibt VentureBeat einen Artikel mit dem Tenor Twitter: Don’t blame Ruby, blame Scoble: Most users of Twitter will single out one person who this points to: blogger and Fast Company [...]
1:15 pm
Twitter’s Technical Issues Mount as Top Users Revolt | Equari said:
[...] is the most popular explanation about the problem), or their own slow reaction time to the problem, Twitter is choosing to blame its most loyal and heavy users, such as tech blogger ubermensch Robert [...]
2:29 pm
Twitter Dee, Twitter Doh… « Living A Creative Life said:
[...] May 30, 2008 at 4:29 pm · Filed under Social Networking and tagged: twitter Apparently there are some issues going on with Twitter at the moment. I’ve always been a huge fan of Twitter, and hope they can fix the issues soon. To read the full story, visit: http://venturebeat.com/2008/05/29/twitter-dont-blame-ruby-blame-scoble/ [...]
3:13 pm
Twitter != Business Model :: In140 said:
[...] twitter saga. Blame some users who push the site to the limit without caps. How about tariff like service with gasp free & [...]
3:55 pm
Twitter Faces Social Graph Scaling Problems - Covering All That's Social On the Web said:
[...] on the site. The center of the discussion revolved around the following statement which VentureBeat translated as Robert Scoble being the source of the problem: The events that hit our system the hardest are [...]
5:20 pm
Twitter culpa usuários por ‘crack off-line’ « Webmanário said:
[...] Ainda pior foi a desculpa encontrada por Alex Payne, um dos responsáveis técnicos do site. Segundo ele, os usuários mais “populares” _aqueles que seguem muita gente e que têm muitos seguidores_ são os responsáveis pelo crack. [...]
6:01 pm
News » Twitter Blames Power users for their Uptime Issues! said:
[...] you could follow. How stupid to blame the people that are helping to evangelize the service. [Venture Beat] Tags: [...]
6:56 pm
Twitter follow-up: Don’t blame Ruby, don’t blame Scoble — though he is kind of a problem » VentureBeat said:
[...] who read the comment’s on yesterday’s post, Twitter: Don’t blame Ruby, blame Scoble, will know that blogger Robert Scoble was not too happy that Twitter seemed to be placing some of [...]
8:47 pm
And Now . . . The Backlash | How To Split An Atom said:
[...] as of late, and they’ve admitted at least once that they had no idea what was going on and now they’re blaming it on “popular users” (which definitely pissed off one Robert Scoble). To be truthful, downtime is nothing new for [...]
1:32 am
links for 2008-05-31 | FullosseousFlap's Dental Blog said:
[...] Twitter: Don’t blame Ruby, blame Scoble » VentureBeat (tags: twitter scoble) [...]
2:12 am
Twitter: weiterhin häufige Downtimes | TechBanger.de said:
[...] muss hoffen, dass Twitter augenblicklich erreichbar ist. Für die Überlastung sollen vor allem Heavy User wie Robert Scoble verantwortlich sein, die das System mit ihren tausenden von Followern Tag für [...]
2:39 am
Scoble and Twitter, Behind the Scenes | Stay N' Alive said:
[...] I had a very unique opportunity to in many ways get in the middle of the Twitter Fiasco, the VentureBeat article suggesting that Robert Scoble was the reason for Twitter’s failures, and [...]
9:37 am
links for 2008-05-31 — contentious.com said:
[...] Twitter: Don’t blame Ruby, blame Scoble » VentureBeat “Twitter’s core team may have had no way of foreseeing today’s issues, companies in the future should learn from this and choose the correct architecture while taking the time to think about scalability, just in case they do hit it big.” (tags: infrastructure technology twitter scalability planning reliability performance problems) [...]
9:38 am
links for 2008-05-31 « A little Jack with that? said:
[...] Twitter: Don’t blame Ruby, blame Scoble » VentureBeat Kevin: Twitter says that programming language Ruby isn’t to blame for its recent stability issues. It’s down to super users like Robert Scoble. (tags: RobertScoble scalability twitter) Twittering awaykevglobal: @Suw and I are packing for a trip to the States. It will be good to be home. Not exotic. Just familiar.kevglobal: Glad that I won't be London when 10,000 people drunkenly mourn the 'loss' of public piss ups on the Tube.kevglobal: stick a fork in me. I'm done. @Suw and I are off to the US for the next week, extending wedding celebration into its 4th glorious month.kevglobal: @vanderwal Maybe its just Defenders of the Media trying to take out the Bit Torrent Pirates ARGGHH! with their DoS attacks.kevglobal: @katebevan Reporter is obviously confused because we don't call it St Louis on Mississippi. Search for: [...]
2:45 pm
Twitter Downtime: Scoble ist schuld! « Sprechblase said:
[...] Scoble ist schuld! 31. Mai 2008 In jüngster Zeit ist Twitter häufig down oder off. Venture Beat erklärt warum [Link via franztoo]: Tonight, developer Alex Payne wrote a Q&A post on the Twitter Developer [...]
9:52 pm
Don’t Blame Twitter, Blame Scoble! | Charles Lau dot Com said:
[...] Everybody follows Scoble. Scoble follows everybody as well. I had asked Scoble how he managed to handle so many followers. He replied that he has answered that many times in Qik videos. Just like myself, I guess many people are infected with the curious virus of handling so many followers while he kept on increasing his twitter posts and followers… Now, everybody blames Scoble for the Twitter problem, not ruby. [...]
9:59 pm
Twitter To Users - Outage is Your Fault! | Blogitics.org said:
[...] Article at VentureBeat [...]
9:29 am
Scoble, hepsi senin suçun! | www.frmedya.com Blog Arşivi said:
[...] Bu elbetteki kullanıcı odaklı bir sistemde söylenmemesi gereken birşeydi. Dolayısıyla tüm camia bir anda bu açığı yakaladı ve tepkiler [...]
3:28 pm
Josh on the Web » Blog Archive » Twitter is CRUD said:
[...] ones; and something that no one really seems to get right. I’ve read some interesting posts discussing Twitter’s problems lately (I’m a little slow to jump on this bandwagon, but like that’s going to stop me). [...]
3:14 am
Twitter Connects Popular Twitter Users to Twitter Problems | Blogging & Writing said:
[...] 2, 2008 · Print This Article Bloggers here, here, and here are discussing Twitter developer Alex Payne’s recent comment that Twitter is [...]
11:56 am
Devil in the Details » And Now…The Backlash said:
[...] as of late, and they’ve admitted at least once that they had no idea what was going on and now they’re blaming it on “popular users” (which definitely pissed off one Robert Scoble). To be truthful, downtime is nothing new for [...]
4:03 am
Scott O’Raw’s Journal » Why The World-Wide Talk Show Doesn’t Need A Host said:
[...] wrote a Q&A post detailing what the company has been going through. In reporting the story VentureBeat (no I hadn’t heard of them either) pulled out the following quote: The events that hit our [...]
3:10 pm
Four More Annoyances of Social Networking. -- UnderCurrent said:
[...] have been numerous excuses for this unexplained phenomenon (too many people following Robert Scoble http://venturebeat.com/2008/05/29/twitter-dont-blame-ruby-blame-scoble/ or the technology was never built to handle so many followers, blah, blah, blah). The bottom line [...]
11:54 am
Four more annoyances of Social Networking -- UnderCurrent said:
[...] buggy. There have been numerous excuses for this unexplained phenomenon (too many people following Robert Scoble or the technology was never built to handle so many followers, blah, blah, blah). The bottom line [...]
6:27 pm
Can Twitter survive the WWDC Stevenote? » VentureBeat said:
[...] of you may recognize that Twitter’s recent downtime issues are a result of its architectural problems exposed by its rapid growth. However, back in the day [...]
10:36 pm
handling community growing pains » nickbouton.com said:
[...] technology platform, Ruby on Rails, for the scalability issues, Twitter has gone so far as to shift the blame to these highly-followed users for some of the architectural issues they’re currently [...]
12:26 pm
Messaging service Twitter hires top developer firm Pivotal Labs to help rebuild its software architecture » VentureBeat said:
[...] the messaging service that many of us love — and love to complain about when it experiences downtime — has hired a premier software development firm to help it [...]
11:14 am
Voting for MobileBeat2008 best companies begins » VentureBeat said:
[...] Twitter — a micro-messaging service, featuring one-liners, that has become all the rage among techies and beyond, despite performance problems. [...]
4:30 pm
A Twitter emergency broadcast system in the works » VentureBeat said:
[...] often write about Twitter’s downtime issues, but the fact remains that it is a very powerful communication tool when working properly. The Buck [...]
12:48 pm
Twitter finally gets its new round of funding; “very strong” valuation » VentureBeat said:
[...] we’ve been quick to point out the difficulties Twitter has been having, I’ve remained excited about its possibilities as a mass [...]
8:50 am
ruby payne said:
[...] not to blame ruby on Rails RoR for Twitter??s woes. …Twitter Rocks on ruby on Rails Internet Newshttp://venturebeat.com/2008/05/29/twitter-dont-blame-ruby-blame-scoble/Twitter: Don??t blame Ruby, blame Scoble MG Siegler/VentureBeatTwitter: Don??t blame Ruby, blame [...]
3:01 am
Friday Frustrations | Social Media Explorer said:
[...] while I’m at it, Robert Scoble shutting down Twitter? Give me a break. Robert’s a nice guy, but no one with his job, number of online followers or [...]
9:44 pm
Twitter or FriendFeed? There can be more than one. » VentureBeat said:
[...] It’s that simple. But we’re not there yet for Twitter, and despite my dozens of posts complaining about the site’s issues, I don’t think we’re going to get there. Twitter will turn things [...]
10:19 pm
Twitter refugees find a new home (at least for this week): Identi.ca » VentureBeat said:
[...] much as we all complain about Twitter’s inadequacies, it still has the single most important feature of any service: a large, active user [...]
10:19 pm
Twitter refugees find a new home (at least for this week): Identi.ca » VentureBeat said:
[...] much as we all complain about Twitter’s inadequacies, it still has the single most important feature of any service: a large, active user [...]
2:34 pm
Why Twitter Must Die | Chris Baskind dot com said:
[...] taking venture funding, which assumes further growth. Nor has it prevented the company from blaming virtually everyone but themselves for their current state of affairs, shutting down basic services, and leaving key segments of its [...]
4:21 pm
A Payne in the Twitter Fail Whale’s side » VentureBeat said:
[...] a few months back when a post he wrote on the Twitter Developer blog seemed to suggest that power users of the site were to blame for its woes. Twitter came back to explain that while power users do take up a lot of resources, it’s the [...]
4:21 pm
A Payne in the Twitter Fail Whale’s side » VentureBeat said:
[...] a few months back when a post he wrote on the Twitter Developer blog seemed to suggest that power users of the site were to blame for its woes. Twitter came back to explain that while power users do take up a lot of resources, it’s the [...]
8:02 am
Popular Bloggers Robert Scoble — TechWag said:
[...] how he uses the product to bring people closer. He does not worry about thousands of followers on twitter , or thousands of followers on Facebook . He is more interested in how those thousands of followers [...]
11:34 am
Understanding Twitter said:
[...] if the Scobelizer is “a beast on the service“, he has a point - your connections and conversations are what count on Twitter - people [...]
4:51 pm
Once again, Twitter’s death is laid out. Once again, users will fail to notice » VentureBeat said:
[...] died by now, I’m not convinced that it’s ever going to die. After all, this is the service that was basically unusable for a few months earlier this year, and came back stronger than ever. It’s also seen a number [...]