Sept. 9 - 10, 2013
San Francisco, CA

Early Bird Tickets on Sale

Apple iCloud services for identity, Game Center, account creation and sign-in, iTunes store purchases are experiencing service failures this morning for some users, Apple says on its system status page.

iCloud is Apple’s cloud service that stores data like calendar events, mail, bookmarks, photos, and iPhone and iPad backups in Apple’s datacenters, and then syncs it across all your Apple devices. It also includes critical services like Find my iPhone, which helps you locate lost devices, and Back to my Mac, which allows you to access your laptop or desktop for critical information while using a mobile device like an iPad.

According to Apple’s system status page, the iCloud failures have been occurring this morning from 4:30AM PST to now. But they may go back farther than today — yesterday my events from Calendar, Apple’s built-in scheduling tool, were not syncing with the calendar on my iPhone.

Apparently I’m not the only one.

AppleID failures are critical issues, since if you cannot sign into your Apple identity, it’s difficult to do anything with iCloud, including setting up a new device, or simply signing into your own account. According to 9to5Mac, Apple’s iCloud services have been experiencing major outages every couple of weeks.

The current outages remind me of Apple’s original cloud and syncing service, MobileMe, which was notoriously unfriendly and unreliable … and of how Steve Jobs dealt with the situation in 2008:

Jobs walked in, clad in his trademark black mock turtleneck and blue jeans, clasped his hands together, and asked a simple question: “Can anyone tell me what MobileMe is supposed to do?” Having received a satisfactory answer, he continued, “So why the fuck doesn’t it do that?”

For the next half-hour Jobs berated the group. “You’ve tarnished Apple’s reputation,” he told them. “You should hate each other for having let each other down.” The public humiliation particularly infuriated Jobs. Walt Mossberg, the influential Wall Street Journal gadget columnist, had panned MobileMe. “Mossberg, our friend, is no longer writing good things about us,” Jobs said. On the spot, Jobs named a new executive to run the group.

I’ve asked Apple for a comment on the ongoing situation and will update this post as I learn more.

photo credit: Sean MacEntee via photopin cc

Sept. 9 - 10, 2013
San Francisco, CA

Early Bird Tickets on Sale

Google has confirmed what many of its users have already been painfully aware of: Gmail is down.

“We are currently experiencing an issue with some Google services. For everyone who is affected, we apologize for any inconvenience you may be experiencing,” a company spokesperson told VentureBeat.

While the service seems to have been restored to at least some users, Google’s App Status dashboard says the company is still investigating the reports.

Gmail service disruptions aren’t at all unprecedented, but what’s notable in this case is that some users have reported instability with Google’s Chrome browser as well. As some have speculated, it’s possible that Gmail’s issues have caused problems for users whose Chrome browsers sync with their Google accounts.

If this is true, it exposes a major flaw in how the Google browser hooks into Google’s servers. While this is conjecture at present, the post-mortem for today’s outage should be particularly interesting.

Photo: Meghan Kelly/VentureBeat

Sept. 9 - 10, 2013
San Francisco, CA

Early Bird Tickets on Sale

With scattered reports of customer defections in the wake of major outages, Amazon Web Services (AWS) announced today that it has expanded to the Asia Pacific region after opening a new data center in Australia.

AWS is a collection of web services that Amazon offers; the most well-known include Amazon EC2 and Amazon S3. It’s the largest cloud computing provider in the world, but it faces heat from Rackspace, Microsoft Azure, and newcomers like Joyent.

For AWS, Australia and New Zealand are ripe for market domination: 10,000 local businesses already use AWS, including the Commonwealth Bank of Australia. No doubt, a local data center should help improve the responsiveness of cloud-based applications and could enable AWS to grow its customer-base.

AWS also revealed that it will be working with a number of partners as it moves into the region, including RightScale, a cloud management technology, and Acquia, which provides hosted Drupal to over 2,000 customers.

“With the capability to achieve single-digit millisecond latency to end-users in Sydney [and] store data locally in Australia. We expect the launch of AWS’s Sydney Region to further increase the amount of Australian and New Zealand customers leveraging AWS,” Andy Jassy, AWS’s senior vice president, said in a statement.

The new Sydney region supports myriad AWS services including Amazon Elastic Compute Cloud (EC2), Amazon Elastic Map Reduce, Amazon SimpleDB and Amazon CloudWatch. Access the full list here. 

At a time when its primary competitors are picking up steam, Amazon is focused on international expansion. Asia Pacific is its ninth region since it launched in 2002; the company already has a strong presence in three regions in the U.S, as well as Tokyo, Sao Paolo and Ireland. Only the Middle East and Africa lack a strong AWS presence.

This summer, high-profile AWS customers were affected by several major blackouts that was caused by a latent bug. Netflix, Foursquare, Reddit and others were down.

Image via Shutterstock

Sept. 9 - 10, 2013
San Francisco, CA

Early Bird Tickets on Sale

Amazon’s EC2 cloud service appears to be fully restored after yesterday’s big outage, which took down sites and services including Reddit, Airbnb, Pinterest, Flipboard, GetGlue, Coursera, Foursquare, and more.

As of today at 6:33 a.m. PT, Amazon’s Status Dashboard says EC2 at its North Virginia data center is “operating normally.” The company writes:

23rd Oct 6:33 AM PDT The remainder of the affected ELB load balancers have been recovered and the service is operating normally.

We have restored IO for the majority of EBS volumes. A small number of volumes will require customer action to restore IO. We are in the process of contacting these customers directly with instructions on how to return their volumes to service.

Volumes affected during this event are continuing to re-mirror (which we expect will continue through the remainder of the day). While this process continues, customers may notice increased volume IO latency.

While EC2 appears to be back to normal, Amazon said the Relational Database Service for its North Virginia data center was still not completely restored. It writes:

23rd Oct 6:37 AM PDT We continue to make progress towards restoring access to the remaining DB Instances in the affected Availability Zone. The service continues to operate normally for the rest of DB Instances in the Region.

A member of hacker collective Anonymous claimed responsibility for yesterday’s outage, but Amazon flatly denied that an attack occurred. We believe Amazon, as the evidence suggests this person is routinely lying about attacking websites and services including GoDaddy and Facebook.

Yesterday’s crippling outage isn’t the only one Amazon has had this year. The company suffered a major outage in June that took down services including Netflix, Instagram, Pinterest, and Heroku. Amazon also suffered large EC2 outages in April and August of 2011.

If you plan to change cloud infrastructure providers because of this outage and the ones before, shoot me a note at sean AT venturebeat.com.

We’ve asked Amazon for more details about what exactly happened with yesterday’s EC2 outage and will update this post if it gets back.

Clouds photo via Tokinu_Unikot/Flickr

Sept. 9 - 10, 2013
San Francisco, CA

Early Bird Tickets on Sale

Updated at 3:30 p.m. PT with Amazon’s response.

A member of hacker collective Anonymous who previously claimed to take down GoDaddy and Facebook has now claimed responsibility for today’s big Amazon’s EC2 cloud outage.

The Amazon outage caused headaches for many large and small web services, including Reddit, Airbnb, Flipboard, GetGlue, Heroku, and Coursera. Amazon blamed the issues on “degraded performance for a small number of EBS volumes in a single Availability Zone in the US-EAST-1 Region.”

Amazon flat-out said the claims of an attack were untrue. “This is not accurate,” an Amazon spokesperson told Venturebeat via email. “An attack did not happen on Amazon EC2.”

Anonymous Own3r, a purported member of Anonymous, said he or she was the one who caused a major service disruption of GoDaddy in September and an outage of Facebook in parts of Europe in October. Both GoDaddy and Facebook claimed that both events were due to internal problems, not an outside attack. Now there’s a third company on Anonymous Own3r’s list.

Anonymous Own3r posted several tweets in poor English, claiming responsibility for an attack on Amazon EC2:

Based on the available evidence, we find the hacker’s claims to be extremely suspect and we believe Amazon. The primary outlets for Anonymous to spread its message on Twitter, @YourAnonNews, @AnonymousIRC, and @AnonOps, have said nothing about today’s Amazon outage. On top of this, both Facebook and GoDaddy claimed there were no outside attacks involved with their recent outages.

This pattern makes it look like Anonymous Own3r is simply claiming responsibility for any outage happening on the web. Of course, we can’t be 100 percent certain that Anonymous Own3r isn’t involved because Anonymous is a purposely fractured organization and major companies don’t want to admit when they’ve been successfully attacked.

Anonymous image via liryon/Flickr

Facebook says reports that its servers have been hacked by Anonymous are false, and that the company is actually experiencing outages on the European continent.

A hacker that goes by the Twitter handle @AnonymousOwn3r claimed to have taken down Facebook in Europe earlier today, siting a vulnerability called cross-site request forgery. @AnonymousOwn3r is otherwise known for hacking into GoDaddy in September. “Evidence” was posted on Pastebin, but Facebook called the hacker’s bluff, telling VentureBeat in an email:

There has not been a hack of Facebook, we have investigated these claims and they are not valid. The evidence cited was produced by an automated vulnerability scanner that alerts developers of potential vulnerability, and we have found these all to be false alerts.

We expect Anonymous just like we expect any other attack on any other day. Due to our size, we face the same threats as seen everywhere else on the web, but we have developed partnerships, backend systems, and protocols to confront the full range of security challenges we face. Facebook has always been committed to protecting our users’ information, and we will continue to innovate and work tirelessly to defend this data.

“Earlier today we made a change to DNS as part of a traffic optimization test, and that change resulted in some users being temporarily mis-routed. We detected and resolved the issue immediately, but a small number of users located primarily in Western Europe experienced issues accessing the site while the DNS addresses repopulated. We are now back to 100 percent, and we apologize for any inconvenience.”

Sept. 9 - 10, 2013
San Francisco, CA

Early Bird Tickets on Sale

Web services provider GoDaddy has apologized to its customers for the Monday’s outage that crippled countless sites across the web, and it is offering one free month of service to nullify complaints.

“We let you down, and we know it,” GoDaddy CEO Scott Wagner said in an e-mail to customers. “We take our responsibilities — and the trust you place in us — very seriously. I cannot express how sorry I am to those of you who were inconvenienced.”

GoDaddy experienced a widespread outage Monday that stopped many small business and personal websites to stop functioning, including my own site.

A member of hacker collective Anonymous, Anonymous Own3r, originally claimed responsibility for the outage. But GoDaddy went on the record saying to say that the outage was “not caused by external influences.” The company said it was not a “hack,” or a denial of service attack (DDoS) but instead blamed the event on “corrupted router data tables.”

Let us know in the comments if you are a GoDaddy customer who was affected by the outage and if you might change providers.

Sept. 9 - 10, 2013
San Francisco, CA

Early Bird Tickets on Sale

After an investigation, GoDaddy claims that its major outage yesterday was “not caused by external influences” and that a DDoS attack or a hack aren’t to blame.

A member of hacker collective Anonymous, Anonymous Own3r, originally claimed responsibility for the outage. It’s possible this person claimed to facilitate the outage to heighten their Twitter profile, which has more than doubled its follower count since yesterday.

GoDaddy’s full statement from interim CEO Scott Wagner is below:

Yesterday, GoDaddy.com and many of our customers experienced intermittent service outages starting shortly after 10 a.m. PDT. Service was fully restored by 4 p.m. PDT.

The service outage was not caused by external influences. It was not a “hack,” and it was not a denial of service attack (DDoS). We have determined the service outage was due to a series of internal network events that corrupted router data tables. Once the issues were identified, we took corrective actions to restore services for our customers and GoDaddy.com. We have implemented measures to prevent this from occurring again.

At no time was any customer data at risk or were any of our systems compromised.

Throughout our history, we have provided 99.999 percent uptime in our DNS infrastructure. This is the level our customers expect from us and the level we expect of ourselves. We have let our customers down and we know it.

We take our business and our customers’ businesses very seriously. We apologize to our customers for these events and thank them for their patience.

Sept. 9 - 10, 2013
San Francisco, CA

Early Bird Tickets on Sale

Popular web hosting company GoDaddy has restored most of it services after a member of hacker collective Anonymous attacked the site.

On Twitter, GoDaddy said that most of its customer-hosted sites were back online following the attack:

I can confirm that my own personal site is back online after being taken down during the outage.

A seemingly high-ranking member of Anonymous, Anonymous Own3r, claimed responsibility for the attack that took down countless sites. GoDaddy’s DNS servers were taken offline for several hours on Monday. Anonymous Own3r said this was not an attack made on Anonymous’ behalf, and it is just from a single member.

GoDaddy indicated that the takedown was isolated to just a denial-of-service attack and that customer data was not compromised..

“All services are restored and at no time was sensitive customer information, such as credit card data, passwords, names, addresses, ever compromised,” GoDaddy spokeswoman Elizabeth Driscoll told CNET.

GoDaddy illustration: Sean Ludwig/VentureBeat

Sept. 9 - 10, 2013
San Francisco, CA

Early Bird Tickets on Sale

Popular web services site GoDaddy has experienced a widespread outage today, causing countless sites around the web to cease functioning.

A member of hacker collective Anonymous has claimed responsibility for the outage, and they said GoDaddy’s DNS servers were all taken offline. “I’m taking godaddy down bacause well i’d like to test how the cyber security is safe and for more reasons that i can not talk now,” Anonymous Own3r wrote on Twitter.

Anonymous Own3r has expressly said this is not an attack made on Anonymous’ behalf, and it is just from this one member.

GoDaddy acknowledged the outage on Twitter with two tweets:

I can confirm that the outage for GoDaddy sites is authentic because my personal URL, SeanLudwig.com, is currently not working.

We will update this post when the outage is resolved.

Developing post. Refresh for updates.

Sept. 9 - 10, 2013
San Francisco, CA

Early Bird Tickets on Sale

(Updated 7/9/2012 with comment from Netflix engineer Ariel Tseitlin.)

Netflix has published an explanation of the Amazon EC2 outage that took down the video-streaming service for hours on June 29 and 30.

In addition to Netflix, the Amazon outage took down Instagram, Pinterest, and other sites that rely on Amazon’s cloud-based computing service. On July 4, Amazon published its own explanation of the snafu, which started with a power failure, led to a failure of the backup generators, and then caused a series of cascading failures with its load-balancing hardware.

Netflix’s analysis discovered that it, too, had problems with its own load-balancing service. “This caused unhealthy instances to fail to deregister from the load-balancer which black-holed a large amount of traffic into the unavailable zone,” Netflix engineers Greg Orzell and Ariel Tseitlin wrote in the post.

Despite the “storm” of failures, Netflix remains confident about its decision to move to cloud-based services in late 2010. The company’s resiliency to cloud failures has actually gotten better, the post argues.

“While it’s easy and common to blame the cloud for outages because it’s outside of our control, we found that our overall availability over the past several years has steadily improved,” Orzell and Tseitlin wrote.

However, the company is looking for help. If you’re an experienced cloud operations engineer, Netflix wants to see your résumé, as it is continuing to hire people for its Cloud Operations and Reliability Engineering team.

“We’re still actively beefing up our reliability engineering team…always looking for good people,” Tseitlin tweeted to VentureBeat.

Story via TNW.

Photo credit: Arthur40A via photo pin

Twitter is notorious for going offline, but usually not for more than an hour. Today, however, the site was inaccessible — on the web and through Twitter clients — for just about one hour.

As of 10:20 a.m. Pacific time, the service and website are back online.

I noticed at around 9:00 am Pacific time that Twitter.com was showing a “could not connect” page. After checking TweetDeck, which was also down, I also checked http://www.downforeveryoneorjustme.com/twitter.com, which also confirmed the site is down.

A quick email to Twitter verified as the company provided some insight:

Users may be experiencing issues accessing Twitter. Our engineers are currently working to resolve the issue.

We will update as we learn more.

Update (10:45 Pacific time): Twitter has updated its status saying all issues have been resolved. Despite that, we’ve noticed that the site is still experiencing hiccups.

Update (12:45 Pacific time): Twitter has announced that the issues have been caused by “a cascaded bug in one of our infrastructure components.”

July 9-10, 2013
San Francisco, CA

Early Bird Tickets on Sale

Verizon Wireless appears to be having yet another wide outage of its 4G LTE data network, according to various reports on Twitter.

The network reported three data outages in the month of December. Verizon swiftly blamed the down time on “growing pains.”

Verizon said today that it is looking into the issues, which are reportedly affecting users in Indianapolis, Milwaukee, Phoenix, and parts of Pennsylvania and Ohio.

“VZW is investigating customer issues in connecting to the 4GLTE data network,” Verizon said via Twitter. “3G data, voice, and text services are operating reliably.”

While Verizon is saying its 3G network is fine, several users of 4G phones say they can’t even access 3G on their phones. Eric Gilkey said on Twitter: “Your 3g WAS out for a while. Let’s not act like it was unaffected.”

Let us know in the comments if you are having any trouble connecting to Verizon’s 4G or 3G networks and where you are located.

VentureBeat is holding its second annual Mobile Summit this April 2-3 in Sausalito, Calif. The invitation-only event will debate the five key business and technology challenges facing the mobile industry today, and participants — 180 mobile executives, investors, and policymakers — will develop concrete, actionable solutions that will shape the future of the mobile industry. You can find out more at our Mobile Summit site.

July 9-10, 2013
San Francisco, CA

Early Bird Tickets on Sale

Verizon Wireless may be paying the price for being the first carrier in the world to widely roll out LTE 4G technology.

The carrier has pointed to the relative immaturity of LTE as the reason for its 4G service outages throughout 2011, reports GigaOm, who chatted with Verizon VP of network engineering Mike Haberman yesterday.

“Being the pioneers, we’re going to experience some growing pains,” Haberman said. “These issues we’ve been experiencing are certainly regrettable but they were unforeseeable.”

Verizon’s 4G network suffered three separate outages during December, as well as a major failure in April. According to Haberman, the issues stemmed from bugs with Verizon’s service delivery core, also known as the IP Multimedia Subsystem (IMS). Notably, every outage was the result of a completely new bug, so at least Verizon seems to be successfully squashing issues as they pop up.

“Verizon’s IMS systems are a complex network of databases, servers, routers, gateways and policy managers supplied by multiple vendors,” writes GigaOm’s Kevin Fitchard. “Alcatel-Lucent, Nokia Siemens Networks, Acme Packet and Tekelec all provide different parts, but Haberman declined to identify which particular elements or which particular vendors were responsible for the problems. In fact, Haberman defended Verizon’s vendors saying that they were experiencing the same LTE growing pains as Verizon.”

To minimize future outages, Verizon will begin segmenting its LTE network by geographic location so that they don’t affect the network nationally, according to GigaOm. The company is also upgrading its IMS software and making other tweaks to prevent future issues.

July 9-10, 2013
San Francisco, CA

Early Bird Tickets on Sale

BlackBerry users can breathe a sigh of relief. Research in Motion announced today that global BlackBerry services are “operating well” following a massive outage that began Monday and disrupted web, email, and messaging capabilities for tens of millions of customers.

The BlackBerry outage — which was the worst in the company’s history — initially started in Europe, the Middle East, and Africa but spread yesterday to North America. With Apple’s iPhone 4S hitting shelves on Friday, this was certainly the worst possible week for RIM’s services to fall apart.

RIM co-CEO Mike Lazaridis issued a video apology for the service disruption (above) and joined co-CEO Jim Balsillie on a conference call this morning to discuss the situation.

“I want to personally apologize to all of the BlackBerry customers we have let down,” Lazaridis said. “You expect better of us. I expect better of us,” he added.

RIM yesterday pointed to a “core switch failure” within its infrastructure as a cause for the failure. On the call, RIM said that it’s working with vendors to prevent any future failures like this, and it’s conducting an analysis of what took its system so long to restart. At this point, it’s not clear which vendors within RIM’s infrastructure failed.

As for what took so long for an apology, Lazaridis said he was busy leading the team to restore the BlackBerry services. The company hasn’t yet decided on how it will make reparations to customers, but that will be a focus now that the services are back up.

When asked if the company is concerned about alienating customers so close to the new iPhone’s launch, one of the executives said, ” We’ve worked 12 years since the launch of the BlackBerry … We’re going to fully commit to win that trust back.”

Via AllThingsD

The information network said it cut the default rate limit, or number of times an application can call the service on behalf of a user, to 175 from 350 per hour today after suffering outages and downtime through the World Cup tournament. All third-party apps, including ones that make the heaviest use of Twitter, will see their rate limits fall by a proportional amount. The reduction applies to Twitter’s mobile apps too.

“Reducing the API rate-limit is not the first or only method we use and is one we try to avoid if possible,” said the company’s developer advocate, Matt Harris. “However, it is also one of the more effective ways to reduce load and ensure that the Twitter service remains available to the greatest number of users.”

The change is a drastic move given that 75 percent of the service’s traffic comes from external developers. The company owes much of its success to its developer community. The company’s recent unreliability — despite its four years of age and more than $150 million in capital — may renew concerns about whether it is really fit to be the backbone of real-time information for the world.

Twitter has had the largest amount of monthly downtime since August of last year, as pictured below in a report from Pingdom. Part of that is due to new growth. The company saw new all-time records in usage during the Japan-Denmark World Cup game and the NBA finals.

“We’re all in the same boat here,” Harris said. “Twitter uses the API for its own services, so we also feel these reduced limits.”