Netflix suffers Christmas Eve outage. Amazon to blame again.

[amenditman]

When will these big companies learn? Amazon's AWS service fails major network customers again!

 

Now, Netflix is down most of the day Christmas Eve. TechCrunch has the whole story.

[V.T. Eric Layton]

*yawn*

 

Don't make no nevermind to me.

[Guest LilBambi]

Good thing there is Hulu Plus!

[V.T. Eric Layton]

Good thing there's still over-the-air-waves TV. I'm watching Perry Mason on MeTV right now, actually.

[amenditman]

Good thing there is Hulu Plus!

I wonder what cloud service Hulu Plus is delivered by. I haven't seen them in the news for major outages before.

[lewmur]

Good thing there's still over-the-air-waves TV. I'm watching Perry Mason on MeTV right now, actually.

I'm watching Netflix ritght now. But I was really ticked off earlier when I was trying to find somethibg to watch other than Christmas specials. I enjoy Christmas but they start the specials so early on TV that by the time it actually gets here, I'm tired of them. How many times can you watch "It's a Wonderful Life"? Edited December 25, 2012 by lewmur

[Guest LilBambi]

It looks like Hulu may be doing their own content delivery network and their bandwidth costs are high but a cursory look didn't glean info on who provides their backend bandwidth but it does look like they are doing it themselves.

 

Last year in 2011, articles were asking if Hulu had too many ads. I understand it's gotten worse in that regard. But they apparently still do have a great service.

 

Based on their Hulu Tech Blog (which is very cool), they have some great techniques for technically keeping things moving on as even a keel as possible. They make use of open source projects, and give back to the community through GIT.

 

Can also find new info about the offerings themselves on their regular Hulu Blog.

 

What I like the most is the site's depth of content and its incredible back end. It seems as if the only performance limitation is the bandwidth you bring to the table. The search is lightning quick and complete and the video pops up in a few seconds.

 

Hulu has structured partnerships with more than 40 websites to date, enabling users to connect with their favorite video content across such popular destinations as Yahoo!, MSN, AOL, IMDb TV Guide, Facebook, Twitter, and more — offering them a way to flexibly and easily integrate Hulu's content into their online entertainment offerings.

 

Engadget 2012-11-05: CBS has been the lone Hulu refusenik among the biggest US broadcasters -- even with rumors of licensing discussions underway at least two years ago, we've usually had to visit the network's own site if we wanted a CSI fix without paying by the show. The deadlock is over at last now that CBS and Hulu have struck a deal. The agreement isn't quite what we'd hope for, focusing almost exclusively on back catalog titles like Medium and Star Trek, although celebrity gossip junkies will like knowing that Entertaintment Tonightsegments will be viewable the day they air on regular TV. We'll have to wait until January 2013... more »

 

Quote above from @WN.com

 

Hulu has continued its steady growth in recent months and on Monday CEO Jason Kilarannounced the streaming service now has more than 3 million paying subscribers. The service had an explosive year and was released on a variety of devices including the Apple (AAPL) TV, Nintendo Wii and Wii U, a number of Android tablets and smartphones, and Windows 8 tablets. Kilar revealed that Hulu Plus can now be accessed from more than 320 million Internet connected devices in the U.S., not including laptop and desktop computers. Hulu also increased its catalog of movies and TV shows by over 40% over the past two years and now has more than 430 content partners providing more than 60,000 TV episodes from 2,300 different series.

 

Hulu is my favorite video site. Sadly for some time now we no longer have unlimited cellular bandwidth so we can't watch it unless we are away from home on an unlimited type connection. Sigh...

Edited December 25, 2012 by LilBambi

[V.T. Eric Layton]

How many times can you watch "It's a Wonderful Life"?

 

At least once a year.

[ross549]

When will these big companies learn? Amazon's AWS service fails major network customers again!

 

Now, Netflix is down most of the day Christmas Eve. TechCrunch has the whole story.

 

Well, it was only one region that went down. If Netflix was doing their own CDN, it is likely they would experience their own problems too. Once you start working on this scale, downtime for some customers is inevitable. No system can have 100% uptime.

 

Netflix is huge, and delivers 1/3 of the internet traffic during peak hours. Any outage even at 1% of total users will be quite noticeable.

 

Adam

[lewmur]

At least once a year.

Yeah, but it's played more than ten times just since Thanksgiving! :'(

[amenditman]

Well, it was only one region that went down. If Netflix was doing their own CDN, it is likely they would experience their own problems too. Once you start working on this scale, downtime for some customers is inevitable. No system can have 100% uptime.

 

Netflix is huge, and delivers 1/3 of the internet traffic during peak hours. Any outage even at 1% of total users will be quite noticeable.

 

Adam

Google handles way more traffic than Amazon AWS and their downtime is substantially less. Amazon has had quite a few high profile outages this year.

[Guest LilBambi]

http://downrightnow.com/

[ross549]

Google handles way more traffic than Amazon AWS and their downtime is substantially less. Amazon has had quite a few high profile outages this year.

 

Google is also running their own platform, and has had their own outages. (Gmail) AWS is serving content for a variety of high profile customers. Amazon's retail site has not gone down this year- at least not that I have heard. Maybe Netflix was not doing something right, or maybe Amazon's AWS API clls were not as well documented as they could be. Suffice to say, depending on another entity to provide services means some things will be out of your direct control.

 

It should be noted that this outage was in the "east 1" region of AWS. It affected some folks, but I never saw an outage.

 

Is a small localized outage a huge problem? I don't think so. Will cloud services like this get better over time? Absolutely.

 

Adam

[V.T. Eric Layton]

Yeah, but it's played more than ten times just since Thanksgiving! :'(

 

I don't have to suffer through that with broadcast only TV. It only showed twice on one local channel the entire season.

 

Sadly, I think a lot of those old classic Christmastime movies are fading away. The newer gens have no interest in watching Miracle on 34th Street or It's a Wonderful Life these days. Only the old grippers sit at home and watch them every year.

 

The world moves on...

[ross549]

We still watch them here, and will introduce our children to them.

 

Adam

[zlim]

I dropped Netflix streaming when the price increased. I still get DVDs by mail simply because all the titles in my queue are not available for streaming.

I went to this site

http://instantwatcher.com

typed in my movie titles and can see if a movie can be streamed. I removed 1 or 2 that could be streamed from my queue. I had 28 titles remaining.

My DVD arrived on Monday and I watched it that night.

I'll continue to get DVDs by mail as long as the movies I want to see can not be streamed.

[Guest LilBambi]

I wouldn't call more than 27 Million a few though...

 

Netflix blames Christmas Eve outage that affected 27 million users on Amazon - Raw Story

 

 

More than 27 million Netflix members in the Americas may have been unable to access shows or films online due to a problem at Amazon Web Services, which rents out computing power in datacenters in the Internet "cloud."

 

The Netflix outage began mid-day in California on Monday and lasted late into the night, according to the company. No problems were reported with viewing offerings at Amazon’s rival online film service.

 

I would call it more Netflix's problem than Amazon. Amazon has redundancy in place, that Netflix apparently didn't take advantage of it.

 

 

(Historical perspective: 1, 2, 3)

Edited December 27, 2012 by LilBambi

[Guest LilBambi]

Blame Netflix outage on the lack of cloud cover - StarTribune

 

In updates on a website that reports on the status of its online services, Amazon traced the trouble to Elastic Load Balancing, a part of its service that helps spread heavy traffic among multiple servers to prevent overload. The company gave few details about the problems in its data center in Northern Virginia beyond this and did not offer an official explanation.

 

Social networks filled with complaints. Some customers also complained that Amazon's own streaming service, Amazon Prime, was down. Amazon said it had fixed the problem completely by the afternoon of Christmas Day, and Netflix said it had restored its services to most of its affected consumers by late Christmas Eve. But the episode highlighted how consumers are increasingly using "the cloud."

 

Wait...apparently Netflix did take advantage of redundancy across zones. Not sure if they did so with Amazon's redundancy or not.

 

Netflix has said it has built several redundancies into its cloud-based system. For instance, it stores its data across multiple "zones," so if there is a failure in one zone, it can retry in another. It says it also spends money on more capacity than it needs, so if there are large spikes in customer activity, the service is less likely to go down.

 

Joris Evers, a Netflix spokesman, declined to elaborate on why Netflix went down despite the safeguards. He said the company was investigating the cause and would do what it could to prevent a recurrence.

[ross549]

I wonder if the outage meant the 27 million users could not access *any* part of Netflix, or just part of the catalog.

 

I'd also love to see statistics on Netflix's uptime. I bet it is very good, all things considered, when compared against smaller providers.

 

Adam

[ross549]

http://techblog.netf...igh-volume.html

 

Here Netflix discusses some of their strategies for fault tolerance.

 

 

On slide 4, Netflix claims 99.7% uptime.

 

Adam

[amenditman]

The article about this outage mentions that it starts in California but fails to mention that it affected various users throughout the country.

 

I am in Florida and had zero service from Netflix until Christmas Day. I did stop trying after 10 pm Christmas Eve.

[Guest LilBambi]

http://techblog.netf...igh-volume.html

 

Here Netflix discusses some of their strategies for fault tolerance.

 

http://www.slideshar...pi-july-18-2012

 

On slide 4, Netflix claims 99.7% uptime.

 

Adam

 

That second one is a video? Looks like it might be. I didn't want to load it since I had no idea how long it was.

 

Yes, the Netflix blog has lots to say on the subject. Two of the three historical links I have in a previous posting were from their blog.

[Guest LilBambi]

Heard that amenditman. Here's an interesting article:

 

The horror! Tweets from Netflix's Christmas outage - CNN

 

 

Netflix said service was restored Christmas morning. "Special thanks to our awesome members for being patient. We're back to normal streaming levels. We hope everyone has a great holiday,' said the company on Twitter.

 

Still, the Internet was not amused -- which didn't stop some social-media users from trying to be amusing.

 

Lots of funny Tweets in the article.

[Guest LilBambi]

There was such a big fuss about Netflix and many seemed to have missed that Netflix was not the ONLY big name website affected by the outage:

 

Amazon cloud outage takes down Netflix, Instagram, Pinterest, & more - Venture Beat

 

An outage of Amazon’s Elastic Compute Cloud in North Virginia has taken down Netflix, Pinterest, Instagram, and other services. According to numerous Twitter updates and our own checks, all three services are unavailable as of Friday evening at 9:10 p.m. PT.

 

Amazon’s service health dashboard indicates that there are power issues in its North Virginia data center, most likely caused by severe storms in the region.

[Guest LilBambi]

That link to severe storms in the region was from back in June 30, 2012 when there was a similar outage as noted in earlier link I posted from Forbes (in the Historical note). That AWS data center in Northern VA was also flaky in October 2012 as well.

 

I live in VA and we had no severe weather on Christmas Eve or Christmas Day. I do not think there was any SEVERE weather in northern/north eastern va on Christmas Eve or Christmas Day either. The only bad storm was the fast moving storm (moving at 60 MPH moving northward and it was severe, but it was after Christmas and was gone very quickly. A splash in the pan as it were.).

Edited December 27, 2012 by LilBambi

[ross549]

That second one is a video? Looks like it might be. I didn't want to load it since I had no idea how long it was.

 

It is slideshare, a power point presentation displayed in a browser window. It does use flash, I think. I did not look too closely.

[ross549]

There was such a big fuss about Netflix and many seemed to have missed that Netflix was not the ONLY big name website affected by the outage:

 

Ah. Power issues at the data center. That would explain where the outage came from. It cascaded from there, I suppose. Nothing much that can be done about these types of issues. You cannot harden against every possible situation and maintain cost-effectiveness.

 

Adam

[Guest LilBambi]

I agree, overall Netflix service is likely overall very good on uptime. But the timing really sucked this year. And it may put more people off from moving away from their Cable to online alternatives.

 

I would think it would not benefit ANY online video service including Amazon for any of the major players to have such a long term outage over a major holiday. It certainly wasn't good for Netflix - especially after trying to rise above a major mess up on their part Sept 2011:

 

 

Don't get me wrong, I love the whole idea of Netflix. But the company has made some big screw ups in the past. This is not one of them. This whole situation certainly couldn't be a helpful one. It will rein long in the minds of users who were frustrated trying to entertain family holiday visitors for as much as 24 hrs for some this Christmas. That's all I am saying. It's not good for them or the video on demand industry. And that really stinks.

Edited December 27, 2012 by LilBambi

[ross549]

The event you reference for 2011 was a business decision, not a technical one. That is an entirely different matter that had nothing to do witht he performance of their streaming system.

 

However, we are entering an age where we expect instant gratification. i should look up the numbers for Netflix DVD vs. streaming. I am willing to bet the streaming is picking up more and more momentum, while the DVD subscriptions are flatlining.

 

With that being said, there is no good time for downtime. It will almost always happen when it is needed the most. I know this for a fact, and I have dealt with it a few times myself. I remember rebuilding a SQL server from scratch once with no previous training.......

 

The nature of the internet at this point is that there is no possibility of 100% uptime. I don't have 100% at my house, but I bet it is close to 99%, especially once I got my modem replaced. The CNI server is not always up, especially when I am monkeying around with the network configurations. There will be downtime, and the trick is to minimize it to the maximum extent possible.

 

Ironically, Facebook has a pretty good reputation in their uptime. sometimes things do not work right, but I cannot think of a time that it was completely down for any number of users. Netflix has been very stable for me. I've only had one instance (a couple months ago) where I was unable to stream. I have a lot of problems with Hulu plus during peak hours. It seems that Hulu running their own CDN is not as effective as Netflix using AWS to manage theirs.

 

Will downtime happen? Yes. Will it occur at the worst possible time? Most likely. Fortunately it was only a regional outage this time, and was back up and running within a day. I say that is pretty good from a network reliability standpoint.

 

100% availability is simply not an option for any network connected device or service.

 

Adam

Edited December 27, 2012 by ross549
clarification

[Guest LilBambi]

Of course it was, a business decision for the event in 2011- it was a bad business decision that has cost them quite a bit and only in the past 4-6 mos has it started to recover from that. However, please note I did not say it was a technical one. But that it compounded the more recent one. That's all. Sorry for the misunderstanding.

 

Some streaming numbers below, and some other info from the Wall Street Journal Online article Netflix Hit by Outage, Blames Amazon:

 

The glitch was at least the third major AWS outage this year for Amazon. Service was knocked out for prominent websites Pinterest, Instagram and Foursquare earlier this year frustrating customers andsending the companies scrambling for a fix.

 

Netflix reported 25.1 million streaming subscribers in the U.S. as of the end of its third quarter. In other regions it reported 4.31 million, but that figure includes the U.K., Ireland and other countries that weren't affected.

 

While Netflix competes with the Amazon Prime streaming video service it is also dependent on the Seattle-based online retailer's AWS unit. Netflix Chief Executive Reed Hastings said at an AWS conference in Las Vegas last month that 95% of the company's storage and computation needs are handled by Amazon.

 

BOLD emphasis in the quote mine.